xargs --max-proc split output per proc?

Question

I recently discovered the xargs --max-procs feature.

How can split the output of the command by proc? Should I just create a mycommand --logfile $LOGFILE, or can I do it from xargs itself?

An example (for womble):

Suppose I have script myprocessor.sh, and a list of files. They can go in any order, but i want to keep the logging for each separate, then:

find $MY_FILE_TREE --print0 | xargs --null --max-procs 3 --max-args 1 --no-run-if-empty myprocess.sh

might be the parallel job I want to run. If myprocessor.sh is mouthy, then I'd like to be able to have each invocation print to a different log. Otherwise the stdout for each is the same, and the logs get jumbled.

Can you give an example of what you want to do? It's not clear to me what you're trying to achieve. — womble, Jan 20 '10 at 00:35
womble, I hope my added example is clear. Please request more information if not. — Gregg Lind, Jan 20 '10 at 00:50

score 1 · Answer 1 · answered Jan 20 '10 at 00:40

1

You could do this by running your xargs command through a shell - this will let you redirect the output - something like this:

find blah -type f | xargs -I{} -P 4 -n 1 sh -c 'yourcommand --input {} > {}.output'

...you'll probably have to tweak it a bit - xargs replaces {} with the item/file it's working on

answered Jan 20 '10 at 00:40

James

7,553
2
24
33

that replacement is a little hairy when the find output is rooted paths, but it's a good idea! – Gregg Lind Jan 20 '10 at 00:51

score 1 · Answer 2 · edited Sep 15 '11 at 21:40

GNU Parallel http://www.gnu.org/software/parallel/ seems to be made for you, because it automatically combines the standard output from the processes correctly.

find $MY_FILE_TREE --print0 | parallel --null --max-procs 3 --max-args 1 --no-run-if-empty myprocess.sh ">" {}.output

or shorter:

find $MY_FILE_TREE --print0 | parallel -0 -j3 -r myprocess.sh ">" {}.output

Watch the intro video: http://www.youtube.com/watch?v=OpaiGYxkSuQ

score 0 · Answer 3 · answered Jan 25 '10 at 16:15

0

You could change your script so that on startup it'll choose a random number/text, then prefix each line with this number? Then you can later split it using grep.

answered Jan 25 '10 at 16:15

Amandasaurus

30,211
62
184
246

xargs --max-proc split output per proc?

3 Answers3