Volatile logging for my infrastructure

One thing I always wanted to set up, but never found the time for, is volatile logging on remote hosts. No traces should be left on unencrypted servers (or encrypted servers, physically out of my control). Also, centrally storing, querying and visualizing the logs is neat.

Additionally, this can also be useful on machines with flash storage, to reduce disk writes.

For Linux with systemd, it is quite easy; just set journald to volatile storage and configure systemd-journal-remote. On Linux without systemd, syslog-ng, rsyslog, or any other syslog daemon capable of remote forwarding can be used. For OpenBSD, I had to dive deeper into the (awesome) man pages.

I’ll go a bit into the server side (receiving the logs) and some options at the end of this article.

Sending the Logs

OpenBSD

First (to prevent some issues later), set -h on OpenBSD’s syslogd¹ to correctly set the hostname when logging to remote hosts, and -s /var/run/syslogd.sock to allow querying with syslogc².

rcctl set syslogd flags -h -s /var/run/syslogd.sock

For volatile logging, we make use of syslogd’s memory buffer feature. With the following syntax, logs can be temporarily buffered in fixed-size memory buffers and then forwarded to a remote host.

For example, see the following snippet from my /etc/syslog.conf:

*.notice;auth,authpriv,cron,ftp,kern,lpr,mail,user.none :256:messages
kern.debug;syslog,user.info                             :256:messages2
auth.info,authpriv.debug                                :256:auth
cron.info                                               :256:cron
daemon.info                                             :256:daemon
mail.info                                               :256:mail

The memory buffer size and name are adjustable:

[!NOTE] syslog.conf(5)³ A colon, followed by a memory buffer size (in kilobytes), followed by another colon, followed by a buffer name. Selected messages are written to an in-memory buffer that may be read using syslogc(8). Memory buffered logging is useful to provide access to log data on devices that lack local storage (e.g. diskless workstations or routers). The largest allowed buffer size is 256kb.

• OpenBSD man pages

To send logs to a remote host, add following snippet to the end of syslog.conf:

# Remote log host
*.*        @tcp://192.168.0.69:514

[!WARNING] syslog.conf should be tab-separated, keep that in mind to prevent weird issues!

The logs in the memory buffers can then be queried using syslogc:

syslogc -f messages

Linux with journald

First of all, install the journald-remote-upload⁴ package for your distro (if not already packaged with systemd). For example on Debian:

apt-get install systemd-journal-remote

Put the following snippet in /etc/systemd/journald.conf.d/volatile.conf:

[Journal]
Storage=volatile
MaxRetentionSec=1h

Here, you may want to adjust MaxRetentionSec according to your needs. Also, if you want to keep logs locally, while still forwarding them to a remote host, adjust the Storage parameter.

Then restart journald:

systemctl restart systemd-journald

Next, configure systemd-journal-upload by putting the following in /etc/systemd/journal-upload.conf:

[Upload]
URL=http://logserver:port
#ServerKeyFile=/etc/ssl/private-journal-upload/journal-upload.pem
#ServerCertificateFile=/etc/ssl/certs/journal-upload.pem
#TrustedCertificateFile=/etc/ssl/ca/trusted.pem

Finally, enable and start systemd-journal-upload.service:

systemctl enable --now systemd-journal-upload.service

To make deployment easier I wrote an Ansible role. You can find it on git.drkhsh.at or Ansible Galaxy.

Receiving the Logs

On the receiving end, I currently settled on VictoriaLogs after trying out Grafana Loki briefly. As with VictoriaMetrics, it replaces a lot of software for me. For example, it can ingest from:

• Syslog (UDP, TCP) - BSD (RFC-3164) & IETF-syslog (RFC-5424)
• systemd-journal-upload (see above)
• Promtail (e.g., Kubernetes, Homeassistant via Logspout addon⁵)
• etc.

It also saves me quite some resources, as it is surprisingly light on memory and disk usage. Also, recently Denis Fondras ported it to OpenBSD.

Other options include, but are not limited to:

• Grafana Loki
  • Quite resource intensive
  • Installing and maintaining promtail agents on many machines sucks
• systemd-journal-remote⁶ or whatever syslog daemon floats your boat higher
  • NO syslog, promtail, etc.
  • Quite inefficient on disk usage and resource consumption

Conclusion

Volatile logging on remote hosts is a crucial security measure to ensure no traces remain behind when logs are stored on hosts physically out of control. Storing logs centrally opens quite interesting query and visualization possibilities.

Finally, I can also highly recommend looking into VictoriaLogs, as it not only replaces multiple tools but also efficiently saves resources without compromising performance - while also being portable across different systems like OpenBSD.