How do I optimize Windows disk I/O for high-latency throughput?

Question

Let's say there are 25 processes each which want to read and index a large (say, 1GB) file. The hard drive may be capable of ~100MB/sec throughput, but the head movement and rotational latency involved in moving between such processes is significant so may reduce throughput to 5MB/sec, or just 5% of the conventional drive's throughput capability.

Were the hard drive to read a large amount of data (say, 32MB) or read whatever it can for a long time (say, 200ms) before moving onto the next process, file IO could approach the full throughput of the drive at the expense of latency.

Does Windows have such a facility for accomplishing this goal in cases where latency takes a back seat to throughput?

Answer 1

This is what the NCQ protocol is for. A drive that supports NCQ looks at all of the incoming commands and reorders them to service them more efficiently (and essentially, reordering the commands is what you're asking about).

That said, I see in your example you have a disk queue depth of 27, so we're well past the help of even NCQ. This is a problem for RAID (possibly coupled with faster physical storage).

There are storage parameters that can be adjusted in Windows, particularly those related to caching behavior, but that's not going to help enough in your case.