2

I am trying to execute a long running process multiple times in parallel. The parameter for each execution of the process is stored in a space separated environment variable. Here is a contrived example of what I am trying to execute:

$ echo -e 1 2 3 4 | xargs --max-procs=3 --max-args=1 --replace=% echo % is the number being processed

Here is the output from that command:

1 2 3 4 is the number being processed

Why is max-args seemingly ignored? I then tried to explicitly set the delimiter which gives better results:

$ echo -e 1 2 3 4 | xargs -d " " --max-procs=3 --max-args=1 --replace=% echo % is the number being processed
1 is the number being processed
2 is the number being processed
3 is the number being processed
4
 is the number being processed

What is xargs doing when it processes the 4th argument?

After some searching, I did manage to almost get what I want. The arguments are processed correctly, but parallelism does not work (verified with another command not shown here):

$ echo -e 1 2 3 4 | xargs -n 1 | xargs --max-procs=3 --max-args=1 --replace=% echo % is the number being processed
1 is the number being processed
2 is the number being processed
3 is the number being processed
4 is the number being processed

What am I missing?

gordo911
  • 23
  • 1
  • 3

3 Answers3

2

Does

echo -e 1 2 3 4 | sed -e 's/\s\+/\n/g' | xargs --max-procs=3 --max-args=1 --replace=% echo % is the number being processed

accomplish the task? The output seems about right:

1 is the number being processed
2 is the number being processed
3 is the number being processed
4 is the number being processed

I also tried replacing echo with sleep to confirm it executes in parallel, and it does:

echo -e 1 2 3 4 5 6 7 8 9 9 9 9 9 9 9 9 9 9 9 9 | sed -e 's/\s\+/\n/g' | xargs --max-procs=20 --max-args=1 --replace=% sleep 1
dusty
  • 286
  • 1
  • 4
  • Thanks, that definitely works. For curiosity's sake, do you know why it is necessary to replace white space with a newline? I'm a xargs newbie. – gordo911 Jun 18 '13 at 14:22
  • 1
    Not quite sure about that, could be a bug/feature of xargs. The `man` page for xargs barely hints on this behaviour, suggesting usage of null character as a separator to solve this problem (`-0` option). – dusty Jun 18 '13 at 15:06
1

AFAIK the default input delimter of xargs is \r , so you have to change it to <space> and send your input ending accordingly, like so:

$ echo -n "1 2 3 4 " | xargs -d" " --max-procs=3 --max-args=1 --replace=% echo % is the number being processed
1 is the number being processed
2 is the number being processed
3 is the number being processed
4 is the number being processed
$

HTH

StefanKaerst
  • 69
  • 5
0

Some other ways, maybe simpler, also with -n, -max-args:

echo -n "foo bar baz" | tr ' ' '\n' | xargs -n 1 echo    # With \n, no need for -0
echo -n "foo bar baz" | tr ' ' '\0' | xargs -0 -n 1 echo # Using null character
echo -n "foo bar baz" | xargs -d ' ' -n 1 echo           # Telling to xargs the delimiter
  • Where of course echo could be any command you want.
  • See also -L parameter and -d '\n' (if lines with spaces).
  • Process win parallel with --max-procs or maybe better with parallel.
Pablo A
  • 169
  • 9