Which archiving method is better for compressing text files on Linux?

Question

In my application I need do compress of logs that are text files.

Seems that bzip2 and gzip have the same compression ratio.

Is that correct?

Answer 1

Normally, bz2 has a better compression ratio, combined with better recoverability features.

OTOH, gz is faster.

xz is said to be even better than bz2, but I don't know the timing behaviour.

Answer 2

Last update of maximumcompression.com is June-2011 _{(answer updated in Oct-2015)}
Therefore this website does not mention
the current champion text compressor worldwide:

      cmix

Competitions/Benchmarks:

• enwiki6
  18.2% compression of the 1MB text file enwik6
• Calgary
  17.6% compression of the 14 files of the Calgary corpus (3GB tar file)
• Hutter Prize
  15.7% compression of the 100MB text file enwik8
  (but cmix is not the winner because requires too much RAM, more than 20GB)
• Silesia Open Source Compression Benchmark
  15.7% compression of the 202MB Silesia corpus
• Large Text Compression Benchmark
  12.4% compression of the 1GB text file enwik9

Details:
Byron Knoll is actively developping cmix as libre software (GPL) since 2013 based on the book Data Compression Explained by Matt Mahoney. Matt Mahoney also maintains some of the above benchmarks and proposes ZPAQ (WP), a command line incremental archiver.

---

If you prefer a more standard tool (requiring less RAM) I recommend:

      lrzip

lrzip is an evolution of rzip by Con Kolivas.
lrzip stands for two names: Long Range ZIP and Lzma RZIP.
lrzip is often better than xz (another popular compression tool).
Alexander Riccio also recommends lrzip.

---

My favorite is:

      zpaq

The "archiver expert", Matt Mahoney, has intensively worked on PAQ algorithms for ten years and provide the best compromise between CPU/memory resources and compression level.

However, the last zpaq version is not often packaged/available on recent distro :-(
I always compile it from sources when I have a new machine and I need a very good compressor: https://github.com/zpaq/zpaq

clone https://github.com/zpaq/zpaq
cd zpaq
g++ -O3 -march=native -Dunix zpaq.cpp libzpaq.cpp -pthread -o zpaq

Answer 3

Maybe you could have a look to those benchmarks, especially the part testing the log files compression.

Answer 4

i have made a benchmark to test to compress the following:
204MB folder (with 1,600 html files)
results

7zip =>     2.38 MB
winrar =>   49.5 MB
zip =>      50.8 MB
gzip =>     51.9 MB

so the 7zip is the best among them you can get it from here
http://www.7-zip.org/

Answer 5

bz2 has tighter compression, the algorithm has more options to look for redundancy to compress away.

gzip is in much more tools, and is more cross platform. More Windows tools can deal with .gz files. It's part of http, so even web browsers can understand it.

On linux, there are tools that let you work on compressed files directly. zgrep and bzgrep can search in compressed files.

If just on Linux, I'd use bzip2, for the slightly better compression ratios.

Answer 6

xz compresses much better than bz2, but takes more time. So, if maximum compression is your goal and space on your hard drive is at a premium (which is my case with one drive at 98% full - while I reorganize my file systems), and you can fire off a script to do the work - take a break and come back in 5 minutes.

unxz is very fast to uncompress in my experience - which is a good thing for me on a daily basis.

bz2 is faster to compress than xz, but does not appear to achieve the compression results of xz.

The only way to make these assessments is to run benchmarks against a mix of common files you normally would compress/decompress, and vary the parameters to see which comes out on top.