Assuming that you run those conversions from and to a rotating disk, you are very likely to make a CPU-bound job into a disk-bound one.
A conversion process consist of three tasks:
- Read the original file from disk
- Do some calculation on the data
- Write the result file to the disk
A rotating disk is good at sequential reads or writes, but extremly bad at random IO - so even a single conversion can be hampered by the concurrency between 1. and 3. This implies that a conversion from one physical disk to another is likely to be faster than a conversion from a disk to itself.
If you now multiply this concurrency by three, you are very likely to end up in a scenario, where the seek and wait-on-rotation times of the disk by far outweigh the actual read times - this can easily lead to throughput going down by orders of magnitude: It is not uncommon for a disk, that can reach over 100MB/s sequential read, to reach less than a single MB/s random reads.
A usually seen pattern is very fast inital performance, while the writes are buffered in RAM, but dropping of a cliff, when the cache is full and the writes really need to hit the disk.
Recommendations:
- First of all get rid of spinning rust - it is 2020.
- If that is no option, then try to limit IO concurrency by using different disks for read and write. The best way might be to create a RAM disk as a target device (as usual in the broadcast industry). In fact since RAM is so cheap it might be a good idea to convert from a RAM disk into a RAM disk.
- Carefully chose the number of concurrent conversion jobs to find the sweet spot between IO saturation and CPU/GPU saturation.
Is
mergeall
a batch or script file? What is your actual ffmpeg command? – Mokubai – 2020-02-27T09:21:10.133If I were running 3 different sessions of FFMPEG at the same time, on the same hard disk, I would expect about the same running frames per second also, or less. That is a lot of work for your system, especially your memory and harddisk. I bet the disk is hot when your done. – vssher – 2020-02-27T09:21:26.663
What is the bitrate of the files? Is your disk saturated by seeks or bandwidth? – Mokubai – 2020-02-27T09:22:40.310
1Are you running 3 concurrent conversions from a single rotating disk to that same disk? That seems quite suicidal! – Eugen Rieck – 2020-02-27T09:53:34.627
@vssher yes, I would expect them to be all the same as each other (up to some small variation). It just seems odd that the sum is ~140fps when an individual one running at >1000fps, almost 10x time speed – Sam T – 2020-02-27T10:04:10.730
@Mokubai They're HD videos. But the point not the absolute bitrate, but the relative between running a single process vs three concurrent. I don't know what those other words mean, sorry – Sam T – 2020-02-27T10:04:53.537
@EugenRieck Perhaps you could elaborate on why doing that should be 10x as slow as doing them one at a time? Ideally in an answer? (In the help, "Comments are used to ask for clarification or to point out problems in the post.") – Sam T – 2020-02-27T10:05:29.683
In task manager click on the "performance" tab. In that screen click on the disk (C:, D:, etc) where the ffmpeg command is running. Share that page as a screenshot here. It would be good to know the same for 2 encodes and a single encode. – Mokubai – 2020-02-27T10:11:33.213
@Mokubai I'm at work now, but I shall do so this evening – Sam T – 2020-02-27T11:14:28.477
@SamT Eugens answer was pretty much what I was pressing towards, but it would be good to verify that this is the case. As a demonstration you can look at HDD benchmarks sequential read/writes get speeds of >100MB/s, random read/writes drops down to 1MB/s under heavy load. Chances are you are hitting that wall to some extent.
– Mokubai – 2020-02-27T11:19:56.573Yeah, Eugen explained well. I shall try to rewrite my batch file so that I can set three items going, but have them run in series rather than in parallel! – Sam T – 2020-02-27T11:22:27.863
I've moved to the SSD, and three simultaneous are stable at around 320fps. On the other hand, a single version is stable at around 1000. Although, it spent a while around 1200, then 1100, then 1000 but ended up around 950 (about 3mins to run the script). So a big improvement over the HDD! – Sam T – 2020-02-27T22:51:40.627