Append to compressed log

Question

There is a couple of z-utilities (zless, zcat), which are able to read contents of zipped file in transparent manner.

Is it possible to append to compressed log file (in either format, not necessarily gzip)?

I want something like the following (ztee is imaginary utility with functionality of interest):

echo "[ $( date ) ] message" | ztee -a file.log.gz

Answer 1

Yes, but it's useless.

---

Some compressed file formats – including the ones used by gzip, bzip2, and xz – do support concatenation natively. (It may require explicit opt-in when using low-level APIs, but command-line decompressors accept it by default.)

echo "test 1" | gzip >> log.txt.gz
echo "test 2" | gzip >> log.txt.gz
echo "test 3" | tee >(gzip >> log.txt.gz)
echo "test 4" | (tee /dev/fd/3 | gzip >> log.txt.gz) 3>&1
zcat log.txt.gz

(Note that e.g. zcat is literally just a shellscript wrapper for gunzip...)

However, this means that each log message is compressed individually, without taking into account all the previous contents. As a result, if you try to chain many short log messages, your compression ratios will be very, very poor. (In my tests compressing a random log file this way, the resulting file actually grew to 120% its original size, because of all the repeated "header" overhead.)

$ dmesg > test.log
$ cat test.log | gzip > test-single.log.gz
$ cat test.log | while read -r line; do
                     echo "$line" | gzip
                 done > test-concat.log.gz
$ du -h test*
500K     test-concat.log.gz
416K     test.log
64K      test-single.log.gz

To my knowledge, there is no tool which would support loading the existing compressed file's headers and use it to compress new data. To achieve this, you would need to run a persistent gzip process and periodically feed it logs via stdin. For example:

#!/bin/bash

# open a subprocess (the bash equivalent of 'popen')
coproc LOG { gzip >> log.txt.gz; }

echo "doing stuff" >&${LOG[1]}
echo "doing more stuff" >&${LOG[1]}
echo "still doing stuff" >&${LOG[1]}

# close its stdin to finish compression
exec {LOG[1]}>&-

In "standard" shell, you could achieve the same using named pipes, or even more simply redirecting the whole script through gzip. (Just print the same message once to stdout and once to stderr, and you have your tee.)

---

Further information on compressed stream concatenation:

• https://stackoverflow.com/questions/8005114/fast-concatenation-of-multiple-gzip-files

• https://www.gnu.org/software/gzip/manual/html_node/Advanced-usage.html

  Multiple compressed files can be concatenated. In this case, gunzip will extract all members at once. If one member is damaged, other members might still be recovered after removal of the damaged member

• https://tukaani.org/xz/format.html

  Concatenation: Just like with .gz and .bz2 files, it is possible to concatenate .xz files as is. The decompressor can decompress a concatenated file as if it was a regular single-stream .xz file.