3
I am trying to split a number of huge gz file into N-line compressed gzipped chunks.
To demonstrate, let us consider the following:
seq 100 | gzip > big_file0.gz
I can split this into multiple 10-line compressed files as follows:
zcat big_file0.gz | split -l 10 --filter='gzip > $FILE.gz' - big_file0.
Let us assume we have a number of big files big_file0.gz
, big_file1.gz
...
I would now like to split each of these files using GNU parallel. Here's the command I come up with:
parallel "zcat {} | split -l 10 --filter='gzip > $FILE.gz' - {.}." ::: big_file0.gz big_file1.gz
However, the shell substitution for $FILE
does not work as expected. $FILE
is replaced with an empty string, so all the output is written to a file called .gz
.
How can I get the $FILE
substitution to work as expected in GNU parallel?