Solution for the local machine
# pssh -P --par 2 --hosts RemoteMachines /opt/RunJobs.sh
or:
# pssh -i --par 2 --hosts RemoteMachines /opt/RunJobs.sh
Explanation of the parameters:
-P
--print
Display output as it arrives. This option is of limited usefulness
because output from different hosts are interleaved.
-i
--inline
Display standard output and standard error as each host completes.
-p parallelism
--par parallelism
Use the given number as the maximum number of concurrent connections.
-h host_file
--hosts host_file
Read hosts from the given host_file.
# ansible --forks 2 -i RemoteMachines '*' -m command -a /opt/RunJobs.sh
Explanation of the parameters:
-f NUM, --forks=NUM
Level of parallelism. NUM is specified as an integer, the default is 5.
-i PATH, --inventory=PATH
The PATH to the inventory hosts file, which defaults to /etc/ansible/hosts.
-m NAME, --module-name=NAME
Execute the module called NAME.
-a 'ARGUMENTS', --args='ARGUMENTS'
The ARGUMENTS to pass to the module.
The command module takes the command name followed by a list of
space-delimited arguments. The given command will be executed on all
selected nodes. It will not be processed through the shell, so
variables like $HOME and operations like "<", ">", "|", and "&" will
not work.
You can read more in the Introduction To Ad-Hoc Commands.
N.B. ansible will not switch to the next group of hosts until all the current hosts ("forks") are done, so its parallelism is lower than of pssh (there might be a way to increase it, but I don't know it).
The RemoteMachines file looks something like this for both cases:
root@maria-clone1.skynet.tld
root@maria-clone2.skynet.tld
root@maria-clone3.skynet.tld
root@maria-clone4.skynet.tld
Solution for the remote machines
Rewrite RunJobs.sh into something like this:
find FileDirectory -name 'Input_*' -print0 | xargs -0 -P 2 -n 1 ./Executable
Explanation:
-0, --null
Input items are terminated by a null character instead of by
whitespace, and the quotes and backslash are not special (every
character is taken literally). Disables the end of file string,
which is treated like any other argument. Useful when input items
might contain white space, quote marks, or backslashes. The GNU find
-print0 option produces input suitable for this mode.
-P max-procs, --max-procs=max-procs
Run up to max-procs processes at a time; the default is 1. If
max-procs is 0, xargs will run as many processes as possible at a
time. Use the -n option or the -L option with -P; otherwise chances
are that only one exec will be done.
-n max-args, --max-args=max-args
Use at most max-args arguments per command line. Fewer than
max-args arguments will be used if the size (see the -s option) is
exceeded, unless the -x option is given, in which case xargs will
exit.
nitro2k01's solution based on GNU Parallel is more powerful, but as you can see, GNU xargs isn't too bad either.
maybe try sticking an ampersand somewhere.. http://superuser.com/questions/177218/how-to-start-gui-linux-programs-from-the-command-line-but-separate-from-the-com the term might be background process Some more ideas here that may be relevant http://stackoverflow.com/questions/29142/getting-ssh-to-execute-a-command-in-the-background-on-target-machine
– barlop – 2013-10-20T21:46:10.570I already have that with the actual job executions but you think maybe the './RunJobs.sh' lines needs to have an ampersand after it too? – JMzance – 2013-10-20T21:50:38.577
i'm very ignorant on this and I didn't notice you'd already done the ampersand. – barlop – 2013-10-20T22:07:04.993
for i in {1..9}; do nohup ssh RemoteMachine${i} ./RunJobs.sh& done
. It's not very robust, but it's a quick-and-dirty if that works for you. – Blacklight Shining – 2013-10-20T23:58:08.673