Converting gzip files to bzip2 efficiently

Question

I have a bunch of gzip files that I have to convert to bzip2 every now and then. Currently, I'm using a shell script that simply 'gunzip's each file and then 'bzip2's it. Though this works, it takes a lot of time to complete.

Is it possible to make this process more efficient? I'm ready to take a dive and look into gunzip and bzip2's source codes if necessary, but I just want to be sure of the payoff. Is there any hope of improving the efficiency of the process?

Answer 1

This question was asked a long time ago when pbzip2 either wasn't available or wasn't capable of compressing from stdin, but you can now parallelize both uncompressing and compressing steps using parallel and pbzip2 (instead of bzip2):

ls *.gz | parallel "gunzip -c {} | pbzip2 -c > {.}.bz2"

which is significantly faster than using bzip2.

Answer 2

Rather than gunzip in one step and bzip2 in another, I wonder if it would perhaps be more efficient to use pipes. Something like gunzip --to-stdout foo.gz | bzip2 > foo.bz2

I'm thinking with two or more CPUs, this would definitely be faster. But perhaps even with only a single core. I shamefully admit to not having tried this out, though.

Answer 3

GNU parallel (http://www.gnu.org/software/parallel) might be an option if you have multiple cores (or even multiple machines):

ls *.gz | parallel "gunzip -c {} | bzip2 > {.}.bz2"

Read the tutorial / man page for details and options.

Answer 4

What you're currently doing is your best bet. There is no conversion tool available, and attempting to bzip2 an already gzipped file is not really an option, as it frequently has undesired effects. Since the algorithm is different, converting would involve retrieving the original data regardless. Unless of course gzipping was a step in the bzip2 process, in which it isn't unfortunately.

Answer 5

Occasionally, I need to do the same thing with log files. I start with the smallest *.gz files first (ls -rS), gunzip and then and bzip2 them individually. I do not know if it is possible to direct the gunzip output directly to the bzip2 input. The bzip2 command is so much slower at compressing than gunzip is at decompression that it may consume the memory and swap space on the host.

Improvements or suggestions are welcome. Here is my one liner:

for i in $(ls -rS *.gz | sed 's/\.gz//'); do gunzip ${i}.gz; bzip2 -9 ${i}; done

Answer 6

Just had to do this a few minutes ago:

find . -name "*.gz" | perl -pi -e 's/\.gz$//g;' | xargs -n1 ./rezip

Where rezip would be defined as:

#!/bin/bash
gunzip -v $1.gz && bzip2 -9v $1

Optionally, you can also make it multi-threaded by using a -P option with xargs, but be careful with that one. (Start low!)

Answer 7

If you have more than a few, check out the LJ article with a nice shell script.

http://linuxgazette.net/123/bechtel.html

7zip gets better compression, and is multi threaded.