How many instances of FFmpeg commands can I run in parallel?

I tired running 8 commands in parallel to fully utilize the CPU and speed up Video conversions, something like this:

ffmpeg -i input.mp4 -t 60 -f mp4 /mnt/a.mp4 > /dev/null 2>&1 &
ffmpeg -i input.mp4 -t 60 -f mp4 /mnt/b.mp4 > /dev/null 2>&1 &
ffmpeg -i input.mp4 -t 60 -f mp4 /mnt/c.mp4 > /dev/null 2>&1  &
ffmpeg -i input.mp4 -t 60 -f mp4 /mnt/d.mp4 > /dev/null 2>&1 &
ffmpeg -i input.mp4 -t 60 -f mp4 /mnt/e.mp4 > /dev/null 2>&1 &
ffmpeg -i input.mp4 -t 60 -f mp4 /mnt/f.mp4 > /dev/null 2>&1 &
ffmpeg -i input.mp4 -t 60  -f mp4 /mnt/g.mp4 > /dev/null 2>&1 &
ffmpeg -i input.mp4 -t 60 -f mp4  /mnt/h.mp4 > /dev/null 2>&1 &

2 to 3 of them get stopped. Why does this happen? Is this a FFmpeg limitation? I have tried this on 16 core and 4 core machines, EC2 c1.xlarge and cc2.8xlarge. Same behavior. I have tried complicated commands and simple ones, still same, 2 or 3 get stopped.

d33pika

Posted 2013-01-18T10:52:34.283

Reputation: 1 391

Did you find any parameter for using multi processors?! – Dr.jacky – 2015-08-19T04:45:16.660

what do you mean by "get stopped" here? bash shows them as paused? – rogerdpack – 2016-01-31T04:33:10.413

3To my knowledge there shouldn't be anything in FFmpeg itself stopping you from doing so. Could be a shell issue? – slhck – 2013-01-18T11:28:11.517

What if I have different inputs for each command? How can I make them independent, because if one stops they all stop. – Samson – 2013-08-09T09:39:53.993

Answers

In answering the original question, as to why some of the jobs stop, ffmpeg on the command line is interactive. It's constantly reading input on the command line. In order for you to run all of them in the background you should change this:

ffmpeg -i input.mp4 -t 60 -f mp4 /mnt/a.mp4 > /dev/null 2>&1 &

to this:

ffmpeg -i input.mp4 -t 60 -f mp4 /mnt/a.mp4 </dev/null > /dev/null 2>&1 &

the addition of </dev/null tells ffmpeg to not look for input, and all of your jobs should run in the background.

DingoNV

Posted 2013-01-18T10:52:34.283

Reputation: 311

1What is this: 2>&1 – Dr.jacky – 2015-08-10T06:12:35.157

Thank you very much! I had the very same problem and this worked like a charm, please accept! – fr_andres SupportsMonicaCellio – 2017-06-21T23:23:47.770

3@Mr.Hyde Late to the party I know, but 2>&1 tells the error stream to go to the same place as the stout stream - so /dev/null. It's shorthand for > /dev/null 2>/dev/null. – berry120 – 2018-02-05T17:26:45.383

Just a thought on your command: a much easier way of hammering the machine with a single command is concatenating everything into a single line:

ffmpeg -i input.mp4 -t 60 -f mp4 /mnt/a.mp4 \
 -f mp4 /mnt/b.mp4 \
 -f mp4 /mnt/c.mp4 \
 -f mp4 /mnt/d.mp4 \
 -f mp4 /mnt/e.mp4 \
 -f mp4 /mnt/f.mp4 \
 -f mp4 /mnt/g.mp4 \
 -f mp4  /mnt/h.mp4 > /dev/null 2>&1

You might want to add (depending on your ffmpeg version) a flag to tell the server to use all available processors

-threads 0

For reference: http://ffmpeg.org/trac/ffmpeg/wiki/Creating%20multiple%20outputs

NublaII

Posted 2013-01-18T10:52:34.283

Reputation: 71

@NublaII What is > /dev/null 2>&1 !? – Dr.jacky – 2015-08-23T11:01:02.743

@Mr.Hyde It hides the standard output from you and shows you error outputs. Read this: http://www.xaprb.com/blog/2006/06/06/what-does-devnull-21-mean/

– Yuri Sucupira – 2016-09-07T22:52:47.293

I have tried this, it does not use all the processors efficiently. The load remains at 20-30%, the hyper threads don't get used at all. – d33pika – 2013-02-07T01:25:05.553

Maybe it has something to do with EC2 machines? I just run the above command on a 16 xeon core machine with 16 outputs and this is what I get: htop screenshot

– NublaII – 2013-02-08T16:09:21.587

Are you on the latest version of FFmpeg? – d33pika – 2013-04-03T02:20:11.580