Recording with high bitrates to a HDD instead of SSD leads to severe lags

0

I'm trying to make offline recordings with OBS, but when I have it write to a classic hard drive, it lags heavily, even though the HDD has fast enough writing speed (+40MB/s). I tried to increase the buffer in the same settings, but they don't seem to relate to output IO. So far, I couldn't find any OBS/ffmpeg settings that could help. How can this be fixed?

Log: https://paste.fedoraproject.org/paste/L9LoUWC0VRGdxGSYESD2kw

System info dump: https://paste.fedoraproject.org/paste/~OrQZas9rYhaHTtEgJpVfg

The log shows a simple speed test, using dd (via WSL) and the OBS log showing two recordings, with the only difference being the save location. As seen, the recording to the HDD (D:) lags / loses frames considerably. In comparison, the first recording to the SSD works as expected. Note that OBS didn't log the skipped frames due to encoding lag for the first recording, because it worked without issue.

I did:

  • test it several times, so the log shows a representative example.
  • test it with different buffer sizes for encoding, with no effect.
  • test it with CRF instead if CBR.
  • tried it with lower bitrates. Less severe lags can already be seen on 10k bitrate.

To be clear:

  • there is no CPU bottleneck.
  • there is no HDD overall writing speed bottleneck (60k bitrate is about 8 MB/s, which is fine).

I think the issue is: too high write latency and too small output buffer, which could either be a bug in OBS or ffmpeg, or just a missing setting I don't know of.

I asked this question on r/obs before.

Update: added system info dump & additional recording log info.

Vektorweg

Posted 2018-05-25T16:13:50.930

Reputation: 111

Please provide details of your system specs. Just telling us there is sufficient bandwidth or throughput is not sufficient. We need the hard drive model, the mainboard model or system brand model name. Based on what you've told us the issue is obvious: Your spinny-disk HDD isn't up to the job, telling us it is isn't a very convincing argument. – music2myear – 2018-05-25T16:19:38.237

@music2myear How can the HDD be not up to the job, when speed tests show, that it can easily handle the bandwidth? – Vektorweg – 2018-05-25T16:28:10.853

If I were to guess? Because it's doing more than simply accepting the bit-for-bit filestream of the output file. Evidence? It works fine on an SSD. Please edit your question to provide the specific details of the system components as requested so that we can either confirm or correct this initial observation and point you towards a good solution. – music2myear – 2018-05-25T16:33:01.393

@music2myear Yes, its just a simple bit-for-bit filestream, with regards to latency. I really don't see the point, but I added a system info dump anyway. – Vektorweg – 2018-05-25T16:55:47.767

A simple list of the hard drives, their connection methods, and your CPU and memory would have been sufficient, but given that dump can you clarify which drive you are writing to and how it is connected to your computer? – music2myear – 2018-05-25T17:01:55.970

Am I correct in assuming you have a 750GB WD HDD connected to a USB2 port and this is what you're writing to and experiencing the lag on? – music2myear – 2018-05-25T17:04:02.610

@music2myear Yes. Are you trying to tell me, that this is an issue, even though i can write at 40 MB/s anyway? – Vektorweg – 2018-05-25T17:09:12.867

Maintain the bitrate but drop output framerate to 30 and check, – Gyan – 2018-05-25T18:49:37.007

I wonder if defragging the disk will help. If the disk is highly fragmented it will not be able to write blocks sequentially and will perform worse. – davidgo – 2018-05-25T20:12:15.067

@davidgo the partition is barely used, includes only a few writes and deletions and is mostly empty. so, probably not. – Vektorweg – 2018-05-26T01:13:34.140

Answers

1

If it walks like a duck and quacks like a duck...

The issue is the USB2 connection and the drive at the end of it.

While USB2 is faster than some things it is slower than most things, and is generally insufficient for any real-time media operations.

Your metrics may indicate sufficient bandwidth, the likely culprit is that the pure datastream does not account for all communication going back and forth across that link, and tests are often optimized and do not represent real-world data types and situations.

A USB3 drive would probably quite capable of what you require.

music2myear

Posted 2018-05-25T16:13:50.930

Reputation: 34 957

That's a wild guess at best. USB2 may increase latency and cap the bandwidth, but bandwidth is still not the issue and lags from latency can practically always be fixed with sufficiently big buffers. The used bandwidth even scales with the bitrate, as I can see in the Taskmanager. But it lags nonetheless. So the issue is clearly not the bandwidth and assuming that better hardware would fix it, is missing the point here. – Vektorweg – 2018-05-25T17:56:58.640

1

Thanks to Suchiman for this hint.

The write-cache is disabled by default for external disks on Windows. Enabling it did fix the issue.

Image of a dialog window in the Windows device manager

It is still not the best solution, as OBS/ffmpeg should have their own write-cache in this case.

Vektorweg

Posted 2018-05-25T16:13:50.930

Reputation: 111