Modern command line file compression utilities

I am frequently moving large files around from server to server.

UNIXes comes with standard gzip and bzip2 compressors. However, what open options there exist beyond these, especially ones utilizing multiple cores? I am aware of pzip2 which I am using currently.

Are there any up-to-date benchmarks and charts how fast/effective compressors are to see what's the tradeoff between time and the file size?

An open source alternative, which is distribution packaged and supported across Linux/OSX, preferred. Windows support unneeded.

Mikko Ohtamaa

Posted 2012-02-04T11:42:42.487

Reputation: 1 790

When you say pzip2, I assume you mean pbzip2? – Croad Langshan – 2012-02-05T16:24:06.613

Answers

You don't say much about your priorities, but it sounds like compression time for large files is your main priority?

This claims to be suited to that case (see the -l flag):

http://manpages.ubuntu.com/manpages/lucid/man1/lrzip.1.html

On the other end of the tradeoff (high compression ratio, long compression time), some notable software distributions have started using this recently:

https://en.wikipedia.org/wiki/Xz

http://tukaani.org/xz/

Re benchmarking: why not run some quick tests youself? Benchmarks are often misleading because results can vary a lot depending on the precise workload and test conditions. Still, here's a report in German (try google translate) about somebody else's benchmarking:

http://www.linux-community.de/Internal/Artikel/Print-Artikel/LinuxUser/2010/10/Aktuelle-Komprimierprogramme-in-der-Uebersicht/(printView)/true

Croad Langshan

Posted 2012-02-04T11:42:42.487

Reputation: 818

Also worth looking at are pigz for parallelism and 7zip if Windows support does rear its head.

user116587

Posted 2012-02-04T11:42:42.487

Reputation: 23