FFMPEG Very Slow Yet Low CPU Utilisation

0

I am using FFMPEG to merge some videos into a single video. For some reason, it is running extremely slowly, barely using any computing power (see picture below). I am not recompiling the video. (See below for script that is running.) Normally it runs very quickly. Indeed, at first it was going at >1000 fps. Now it's down around 50.

Importantly, I am running three sessions simultaneously. That is, I have made three copies of the bin folder (which contains the executables) and am running each one separately. Note, though, that three instances totalling around 140fps is significantly worse than a single instance giving more like 1200!

Is there some way that I can get it to fully utilise the processing power? I'm using Windows 10.

FFMPEG_Slow

These three questions seem relevant.

Sam T

Posted 2020-02-27T09:16:12.943

Reputation: 177

Is mergeall a batch or script file? What is your actual ffmpeg command? – Mokubai – 2020-02-27T09:21:10.133

If I were running 3 different sessions of FFMPEG at the same time, on the same hard disk, I would expect about the same running frames per second also, or less. That is a lot of work for your system, especially your memory and harddisk. I bet the disk is hot when your done. – vssher – 2020-02-27T09:21:26.663

What is the bitrate of the files? Is your disk saturated by seeks or bandwidth? – Mokubai – 2020-02-27T09:22:40.310

1Are you running 3 concurrent conversions from a single rotating disk to that same disk? That seems quite suicidal! – Eugen Rieck – 2020-02-27T09:53:34.627

@vssher yes, I would expect them to be all the same as each other (up to some small variation). It just seems odd that the sum is ~140fps when an individual one running at >1000fps, almost 10x time speed – Sam T – 2020-02-27T10:04:10.730

@Mokubai They're HD videos. But the point not the absolute bitrate, but the relative between running a single process vs three concurrent. I don't know what those other words mean, sorry – Sam T – 2020-02-27T10:04:53.537

@EugenRieck Perhaps you could elaborate on why doing that should be 10x as slow as doing them one at a time? Ideally in an answer? (In the help, "Comments are used to ask for clarification or to point out problems in the post.") – Sam T – 2020-02-27T10:05:29.683

In task manager click on the "performance" tab. In that screen click on the disk (C:, D:, etc) where the ffmpeg command is running. Share that page as a screenshot here. It would be good to know the same for 2 encodes and a single encode. – Mokubai – 2020-02-27T10:11:33.213

@Mokubai I'm at work now, but I shall do so this evening – Sam T – 2020-02-27T11:14:28.477

@SamT Eugens answer was pretty much what I was pressing towards, but it would be good to verify that this is the case. As a demonstration you can look at HDD benchmarks sequential read/writes get speeds of >100MB/s, random read/writes drops down to 1MB/s under heavy load. Chances are you are hitting that wall to some extent.

– Mokubai – 2020-02-27T11:19:56.573

Yeah, Eugen explained well. I shall try to rewrite my batch file so that I can set three items going, but have them run in series rather than in parallel! – Sam T – 2020-02-27T11:22:27.863

I've moved to the SSD, and three simultaneous are stable at around 320fps. On the other hand, a single version is stable at around 1000. Although, it spent a while around 1200, then 1100, then 1000 but ended up around 950 (about 3mins to run the script). So a big improvement over the HDD! – Sam T – 2020-02-27T22:51:40.627

Answers

3

Assuming that you run those conversions from and to a rotating disk, you are very likely to make a CPU-bound job into a disk-bound one.

A conversion process consist of three tasks:

  1. Read the original file from disk
  2. Do some calculation on the data
  3. Write the result file to the disk

A rotating disk is good at sequential reads or writes, but extremly bad at random IO - so even a single conversion can be hampered by the concurrency between 1. and 3. This implies that a conversion from one physical disk to another is likely to be faster than a conversion from a disk to itself.

If you now multiply this concurrency by three, you are very likely to end up in a scenario, where the seek and wait-on-rotation times of the disk by far outweigh the actual read times - this can easily lead to throughput going down by orders of magnitude: It is not uncommon for a disk, that can reach over 100MB/s sequential read, to reach less than a single MB/s random reads.

A usually seen pattern is very fast inital performance, while the writes are buffered in RAM, but dropping of a cliff, when the cache is full and the writes really need to hit the disk.

Recommendations: - First of all get rid of spinning rust - it is 2020. - If that is no option, then try to limit IO concurrency by using different disks for read and write. The best way might be to create a RAM disk as a target device (as usual in the broadcast industry). In fact since RAM is so cheap it might be a good idea to convert from a RAM disk into a RAM disk. - Carefully chose the number of concurrent conversion jobs to find the sweet spot between IO saturation and CPU/GPU saturation.

Eugen Rieck

Posted 2020-02-27T09:16:12.943

Reputation: 15 128

Thanks for this description; it is very clear! I do relatively little video stuff, so don't want to splash out big £££ on a large SSD to be used infrequently. I think I shall try to set it up to read from one disk and write to another. In fact, my videos are rarely >10GBs, so I can read it from my large HDD and write it to my smaller SSD! – Sam T – 2020-02-27T11:21:19.817

Neat benchmark to prove the point: https://hdd.userbenchmark.com/Seagate-Barracuda-1TB-2016/Rating/3896 sequential reads/writes = fast, small (4k) reads/writes = painfully slow. Once the head starts seeking across the drive the performance of the disk is dominated by that time and read/write speeds can fall down to less than 1% of the peak performance of a spinning rust disk.

– Mokubai – 2020-02-27T11:23:14.767

Here's a benchmark of my machine (same site) from a few weeks ago. Hard drives are pretty bad in it, for some reason. (It was a gaming machine back in 2010 when I got it! Look at the poor thing now...) – Sam T – 2020-02-27T22:45:05.737

Asked: 2020-02-27T09:16:12.943

Viewed: 44 times

Active: 2020-02-27T10:30:55.790