Does 'Write cliff' phenomenon only happen to SSD disks?

Question

First of all, I am new to disks, I just know the basics. I am trying to improve performance of a software with heavy I/O and I need to understand how my SAS disks work in terms of read and write speed.

I have done very basic read/write tests and I have noticed what seems to be a 'Write cliff'. Disks have several TB of storage capacity, and the machine they are connected to has 630GB of RAM. My tests show that after ~200GB of intensive writing, the writing speed falls from ~2000MBps to ~300MBps.

I guess there is some kind of big buffer or caché that fills up at some point, and that causes the decrease in writing speed. How does that really work? How can I calculate the size of it? Is it configurable?

Answer 1

300M/s is a perfectly normal write speed for a direct attached SAS array. Everything above that is an artifact of caching by the operating system.

It's hard to tell without knowing what your application does, but in many case using the O_DIRECT flag when opening a file will give you a more realistic behaviour: It makes the system bypass the cache and go directly to the disk controller, thus removes the cache artifacts and the write cliff.

Ofcourse this comes at the cost of reducing your write performance to the real level of your I/O hardware - but that's exactly the point: You can't write faster than what your disk subsystem allows in a sustainable way: Cache will help a lot with bursty write loads but can't overcome the limitations of the underlying hardware forever.