Windows WriteFile performance (SSD vs HDD)?

Question

This question is tied to the following observation:

https://social.technet.microsoft.com/Forums/exchange/en-US/09dd046e-8127-4550-8e26-5fba7a5a0743/performance-flaw-in-file-system-decreases-writefile-throughput-towards-0?forum=winserverfiles

It is related to writing a single big file on the disk with fast rate.

I try to do something similar by creating a benchmark on how writing a big file (minimum 30 GB) behaves on an SSD (OS is Windows 7). The basis of the test code can be found to the link above (shortcut here: http://www.quicklz.com/write.c).

Results for creating a single 30 GB file can be seen in the following table. Chunk size (written to disk) is 256 KiB aligned (using _aligned_malloc at 4096 KiB physical page size on NTFS). Also tried larger Chunk Size (e.g. 1 GB) but results were similar. First column is file size created in each iteration (close to 1 GB appended each time) and columns 2-5 are the Write Speeds observed for HDD / SSD using "caching / buffering" (default option in WriteFile) compared to the "no buffering / caching" case (See CreateFile flag FILE_FLAG_NO_BUFFERING). In the Windows caching case the OS decides when to FlushFileBuffers (not called explicitly).

+-----------+---------------+--------------+---------------+--------------+
| File Size | Win cache HDD | No cache HDD | Win cache SSD | No cache SSD |
+-----------+---------------+--------------+---------------+--------------+
| 1024 MiB  | 3282 MiB/s    | 205 MiB/s    | 2626 MiB/s    | 187 MiB/s    |
| 2048 MiB  | 2853 MiB/s    | 203 MiB/s    | 2626 MiB/s    | 383 MiB/s    |
| 3072 MiB  | 2861 MiB/s    | 207 MiB/s    | 3132 MiB/s    | 388 MiB/s    |
| 4097 MiB  | 2977 MiB/s    | 198 MiB/s    | 2626 MiB/s    | 393 MiB/s    |
| 5121 MiB  | 2986 MiB/s    | 206 MiB/s    | 2731 MiB/s    | 388 MiB/s    |
| 6145 MiB  | 3282 MiB/s    | 200 MiB/s    | 2626 MiB/s    | 199 MiB/s    |
| 7169 MiB  | 33 MiB/s      | 201 MiB/s    | 215 MiB/s     | 377 MiB/s    |
| 8194 MiB  | 2626 MiB/s    | 201 MiB/s    | 287 MiB/s     | 197 MiB/s    |
| 9218 MiB  | 2626 MiB/s    | 207 MiB/s    | 317 MiB/s     | 345 MiB/s    |
| 10242 MiB | 2853 MiB/s    | 197 MiB/s    | 215 MiB/s     | 181 MiB/s    |
| 11266 MiB | 3122 MiB/s    | 202 MiB/s    | 375 MiB/s     | 343 MiB/s    |
| 12291 MiB | 3282 MiB/s    | 202 MiB/s    | 368 MiB/s     | 209 MiB/s    |
| 13315 MiB | 187 MiB/s     | 199 MiB/s    | 194 MiB/s     | 362 MiB/s    |
| 14339 MiB | 145 MiB/s     | 201 MiB/s    | 329 MiB/s     | 356 MiB/s    |
| 15363 MiB | 140 MiB/s     | 205 MiB/s    | 125 MiB/s     | 195 MiB/s    |
| 16388 MiB | 135 MiB/s     | 195 MiB/s    | 220 MiB/s     | 360 MiB/s    |
| 17412 MiB | 126 MiB/s     | 204 MiB/s    | 279 MiB/s     | 196 MiB/s    |
| 18436 MiB | 124 MiB/s     | 195 MiB/s    | 269 MiB/s     | 103 MiB/s    |
| 19460 MiB | 129 MiB/s     | 199 MiB/s    | 242 MiB/s     | 104 MiB/s    |
| 20485 MiB | 137 MiB/s     | 203 MiB/s    | 189 MiB/s     | 107 MiB/s    |
| 21509 MiB | 128 MiB/s     | 198 MiB/s    | 120 MiB/s     | 116 MiB/s    |
| 22533 MiB | 134 MiB/s     | 202 MiB/s    | 109 MiB/s     | 107 MiB/s    |
| 23557 MiB | 137 MiB/s     | 194 MiB/s    | 112 MiB/s     | 113 MiB/s    |
| 24582 MiB | 135 MiB/s     | 204 MiB/s    | 107 MiB/s     | 102 MiB/s    |
| 25606 MiB | 131 MiB/s     | 194 MiB/s    | 122 MiB/s     | 101 MiB/s    |
| 26630 MiB | 130 MiB/s     | 200 MiB/s    | 151 MiB/s     | 108 MiB/s    |
| 27654 MiB | 139 MiB/s     | 199 MiB/s    | 108 MiB/s     | 117 MiB/s    |
| 28679 MiB | 130 MiB/s     | 206 MiB/s    | 104 MiB/s     | 103 MiB/s    |
| 29703 MiB | 121 MiB/s     | 195 MiB/s    | 105 MiB/s     | 107 MiB/s    |
| 30727 MiB | 132 MiB/s     | 203 MiB/s    | 102 MiB/s     | 104 MiB/s    |
+-----------+---------------+--------------+---------------+--------------+

Some observations here:

Both Win Cache cases (SSD and HDD) have very fast speed at first due to windows caching. Then rate goes lower.

On the HDD case without caching we can see stable rate (and ok rate for my HDD since it is about 207 MB/s).

On the SSD case without caching we can actually see getting even worst rate compared to the HDD case.

So finally to my question:

Why?

Why I am getting worst rates when I use no buffering on an SSD? Where is the bottleneck? Windows OS handling? SSD native behavior? Or both Software and Hardware?

Can I achieve constant rate of transfer as that of the HDD case (but higher)? Assuming my SSD has a capability of Write Speed 340 MB/s can I achieve this in a constant rate?

Please let me understand what I am ignoring here.

Answer 1

Basically most modern cheap SSDs use TLC NAND flash which is actually quite slow at being written to compared to MLC or SLC NAND flash memory.

It provides a much higher capacity though, and the read speed is comparable to the other types which, for the "normal" system random read/write profile can be quite acceptable.

In general they augment this slow writing speed by adding a write buffer of faster NAND that can, until the buffer is filled, bring performance up to nearly normal levels.

Once the buffer is filled though all writes to the drive become limited by how fast data can be written to the slower NAND.

For more info read TLC NAND SSDs: The crippling problem storage makers don't advertise

The only thing you can do is look carefully at the specifications for the drives you buy and make sure to get one that doesn't use TLC NAND. This information can be difficult to find and the "consistently fast" drives will probably be significantly more expensive.