1

It has been established that journald logs are huge. On this specific system, the logs grow about 3GB per week. For audit purposes I would like to preserve system logs for a longer time than what can I comfortably store on that system; this would only be "just-in-case" storage that doesn't have to be quick to retrieve from. What would be the best way to archive these logs in a space-efficient way?

I am thinking of periodically doing some kind of journalctl | xz >>/var/log/old-logs.txt.xz, but that would be quite wasteful in terms of CPU usage, re-archiving the same parts of data again and again and might miss logs if they get deleted by journalctl, so maybe there's a better way?

liori
  • 737
  • 3
  • 14
  • Ship them off to another server where you already store logs? – Michael Hampton Oct 03 '20 at 22:22
  • @MichaelHampton, I'd probably do that if there was another server. No such thing here, though, it's a very small deployment, a single old HP Microserver gen7. With plaintext files I'd be able to set up `logrotate` and it would just work, but no idea how to approach this with `journald`. – liori Oct 03 '20 at 23:15

2 Answers2

3

journald rotates its own files, so it's safe to copy the old files (which always have an @ in the filename) elsewhere, compress them, or do whatever you want with them. If it's necessary to look into them, journalctl can be pointed at specific journal files with the --file command line option.

You should also see journald.conf for options to limit journald's disk usage.

Michael Hampton
  • 237,123
  • 42
  • 477
  • 940
  • 1
    This means I can just periodically run `xz /var/log/journal/**/*@*.journal`, will be enough for this server. Thank you! – liori Oct 04 '20 at 01:01
  • Also, from the systemd-journald man pages it mentions the following: "`If the daemon is stopped uncleanly, or if the files are found to be corrupted, they are renamed using the ".journal~" suffix, and systemd-journald starts writing to a new file.`" (Taken from [this answer here](https://askubuntu.com/a/864724/1355917) of an unrelated question that shows where the log files are stored). – Lennin Marte Sep 03 '21 at 21:23
1

Well it's true that xz compression might give you the best compression level.

Yet xz compressed logs can not be queried with journalctl.

The other approach is to use filesystem compression. I use btrfs for this with zstd compression and it gives me about 10x space usage reduction and logs are still available for interaction via journalctl. I do not recommend using btrfs volume for /var/log/journal as journalctl keeps marking its directories with No_COW attribute (see lsattr) which prevents compression. It's possible to overcome this like I've done in my setup, but it is more fragile and complicated. Another way is to move old log files to separate compressed dir. Something like this:

mkfs.btrfs "$BLOCK_DEVICE_PATH"
mkdir /var/log/journal_archive

# With my data zstd compression level gives best compression/speed ratio.
# Your mileage may vary.
#Default level 3 is quite good so you might just stick with the default
mount -o compress=zstd:8 "$BLOCK_DEVICE_PATH" /var/log/journal

# To archive old logs (older than a week):
find /var/log/journal/$MACHINEID/ -name '*@*.journal' -mtime +7 -exec mv -v {} /var/log/journal_archive \;

# To query old logs
journalctl --directory=/var/log/journal_archive

# Cleanup archive to keep 20GB of uncompressed files
journalctl --vacuum-size=20G --directory /var/log/journal_archive
Self-Perfection
  • 11
  • 1