why is my master node idle when running gnu parallel remotely?

1

I'm using gnu-parallel to run a number of python scripts in parallel across a number of computers in my gigabit network.

The files being processed are located on a tmpfs (RAM) folder on my master node. When I start a parallel process I can see that all the cores in the remote node are working but my master node stays idle.

Here is what my parallel line looks like:

parallel --workdir ./netshare --filter-hosts -S :,192.168.88.222\
 'myscript.py {.}.shp output_{.}.shp input.shp' ::: tile_{1..445}.shp

I've tried reversing the order of the ip adresses 192.168.88.222,: with the same result.

I've tried running only on the local master node and other than the warning that no remote nodes are available all 8 local threads start working.

I thought it might be network latency but no python scripts start at all! Whereas on the remote host I see four running without problem.

As per Ole's comment I tested the issue with bzip2, here's the command:

 parallel -S $SERVER1,: 'cd /path/to/mounted/network/share && cp largefile.avi file{1}.avi && bzip2 -v file{1}.avi && rm file{1}.avi.bz2' ::: {1..100}

The test worked exactly as expected. All cores of all hosts fired up with cp then bzip. So guess the problem lies in my python script somehow.

Mr Purple

Posted 2015-08-08T02:45:50.743

Reputation: 305

1Can you reproduce this with other commands - such as bzip2? – Ole Tange – 2015-08-08T08:29:17.007

Turns out the answer was related to the working directory being correctly accessed. Careful remounting of each folder and checking the working dir flags fixed it. The solution is definitely not worth my answering. Thanks for the pointer though @Ole it gave me a simple means to bug fix. – Mr Purple – 2015-08-08T23:16:39.500

No answers