Retrying failed command that is run with xargs

I have a script that rsync's files over in parallel using xargs and rsync daemon mode. I occasionally get errors with rsync due to writes happening on the source. I want to be able to retry the rsync if there are any errors. I thought this would work, but having a semi-colon seems to break it. Is there another way to do this?

find ./path -mindepth 1 -maxdepth 1 -type d | xargs -r -t -n 1 -P 4 -I % until rsync -am %/ $host::%/; do echo "TRY AGAIN"; done ./rsync_retry.sh: line 21: syntax error near unexpected token `do'

minhaz1

Posted 2015-12-14T23:25:05.130

Reputation: 290

What's the test condition you're trying to check for? You want xargs to run an until loop on every file? Should until not be first line? – Xen2050 – 2015-12-15T06:59:29.760

So in ./path are a bunch of directories, and rsync will be syncing each of those directories over. Sometimes, a few of the files inside of those directories fails to copy and in those cases I want the rsync for that directory to retry. – minhaz1 – 2015-12-15T21:12:05.403

1xargs is a horrible broken program. Just say no. GNU parallel is a large improvement. For anything too complicated for that you should use a different language that actually supports concurrency. See UsingFind for how to not suck at find. – ormaaj – 2015-12-16T17:20:47.543

Answers

Just cut & pasting some stuff around, putting until at the start...

until find ./path -mindepth 1 -maxdepth 1 -type d | xargs -r -t -n 1 -P 4 -I % rsync -am %/ $host::%/
do
    echo "TRY AGAIN"
done

This runs the find...|xargs... until it returns true.

To just run the rsync on a single directory that fails, need an rsync for just one dir, no xargs, and try splitting the find and the test/loop (may fail if there's newlines or crazy characters in dir names):

for dir in $( find ./path -mindepth 1 -maxdepth 1 -type d )
do
    until rsync -am %/ $host::%/ $dir   # Or replace with whatever single dir rsync works best for you
    do
        echo "TRY AGAIN"
        sleep 2
     done
done

This next one is probably the answer you're waiting for, have xargs call a function a bunch of times, and the function does the "retry if failed" part:

doitright(){
until rsync -am "$@"/ $host::"$@"/ "$@"
    do
        echo "TRY AGAIN"
        sleep 2
     done
}

export -f doitright

find ./path -mindepth 1 -maxdepth 1 -type d | xargs -r -t -n 1 -P 4 -I{} bash -c "doitright {}"

Xen2050

Posted 2015-12-14T23:25:05.130

Reputation: 12 097

I don't want it to re-run all of it though. I only want the rsync's to repeat if they don't succeed. The process takes up to two hours, so redoing the whole thing isn't really an option. – minhaz1 – 2015-12-15T21:10:57.280

xargs is there for a reason, it's allowing me to run P number of those rsync's in parallel. This is a very important aspect as it makes a big difference to transfer times. – minhaz1 – 2015-12-16T05:46:53.160

edited in another solution. Wondering why not just run rsync on the main dir, and let it do all the work? Your network needs 4 transfers to get "up to speed", one wouldn't be 4 times faster than 4 separate ones? – Xen2050 – 2015-12-16T17:22:53.887

With GNU Parallel you would do:

find ./ -mindepth 1 -maxdepth 1 -type d | parallel --retries 3000  rsync -am {}/ $host::{}/

Ole Tange

Posted 2015-12-14T23:25:05.130

Reputation: 3 034