Weird behaviour on ext4 dd performance test

0

I'm testing the performance of ext3/ext4 on a raid1 system (but the same goes for a non raid system). The results from bonnie++ give a distorted image because they circumvent the delayed allocation on ext4. So i wanted to test the performance with dd. 

dd bs=20M count=1024 if=/dev/zero of=/mountpoint/test conv=fdatasync

These are the results from a bunch of dd runs on ext4:

21474836480 bytes (21 GB) copied, 177.162 s, 121 MB/s
21474836480 bytes (21 GB) copied, 181.148 s, 119 MB/s
21474836480 bytes (21 GB) copied, 181.873 s, 118 MB/s
21474836480 bytes (21 GB) copied, 183.865 s, 117 MB/s
21474836480 bytes (21 GB) copied, 189.333 s, 113 MB/s
21474836480 bytes (21 GB) copied, 190.711 s, 113 MB/s
21474836480 bytes (21 GB) copied, 195.004 s, 110 MB/s
21474836480 bytes (21 GB) copied, 197.961 s, 108 MB/s
21474836480 bytes (21 GB) copied, 202.645 s, 106 MB/s
21474836480 bytes (21 GB) copied, 204.505 s, 105 MB/s
21474836480 bytes (21 GB) copied, 210.022 s, 102 MB/s
21474836480 bytes (21 GB) copied, 215.073 s, 99.8 MB/s
21474836480 bytes (21 GB) copied, 221.405 s, 97.0 MB/s
21474836480 bytes (21 GB) copied, 226.552 s, 94.8 MB/s
21474836480 bytes (21 GB) copied, 236.038 s, 91.0 MB/s
21474836480 bytes (21 GB) copied, 242.735 s, 88.5 MB/s
21474836480 bytes (21 GB) copied, 252.866 s, 84.9 MB/s
21474836480 bytes (21 GB) copied, 263.934 s, 81.4 MB/s
21474836480 bytes (21 GB) copied, 277.89 s, 77.3 MB/s
21474836480 bytes (21 GB) copied, 295.957 s, 72.6 MB/s
21474836480 bytes (21 GB) copied, 320.406 s, 67.0 MB/s
21474836480 bytes (21 GB) copied, 310.082 s, 69.3 MB/s
21474836480 bytes (21 GB) copied, 180.104 s, 119 MB/s
21474836480 bytes (21 GB) copied, 178.806 s, 120 MB/s
21474836480 bytes (21 GB) copied, 184.888 s, 116 MB/s
21474836480 bytes (21 GB) copied, 184.223 s, 117 MB/s
21474836480 bytes (21 GB) copied, 189.156 s, 114 MB/s
21474836480 bytes (21 GB) copied, 191.854 s, 112 MB/s
21474836480 bytes (21 GB) copied, 195.005 s, 110 MB/s
21474836480 bytes (21 GB) copied, 199.11 s, 108 MB/s
21474836480 bytes (21 GB) copied, 203.337 s, 106 MB/s
21474836480 bytes (21 GB) copied, 206.47 s, 104 MB/s
21474836480 bytes (21 GB) copied, 211.136 s, 102 MB/s
21474836480 bytes (21 GB) copied, 216.129 s, 99.4 MB/s
21474836480 bytes (21 GB) copied, 221.425 s, 97.0 MB/s
21474836480 bytes (21 GB) copied, 228.962 s, 93.8 MB/s
21474836480 bytes (21 GB) copied, 238.214 s, 90.1 MB/s
21474836480 bytes (21 GB) copied, 245.252 s, 87.6 MB/s
21474836480 bytes (21 GB) copied, 254.058 s, 84.5 MB/s
21474836480 bytes (21 GB) copied, 267.616 s, 80.2 MB/s
21474836480 bytes (21 GB) copied, 282.603 s, 76.0 MB/s
21474836480 bytes (21 GB) copied, 298.554 s, 71.9 MB/s
21474836480 bytes (21 GB) copied, 322.213 s, 66.6 MB/s
21474836480 bytes (21 GB) copied, 282.227 s, 76.1 MB/s
21474836480 bytes (21 GB) copied, 179.186 s, 120 MB/s
21474836480 bytes (21 GB) copied, 180.16 s, 119 MB/s
21474836480 bytes (21 GB) copied, 185.174 s, 116 MB/s
21474836480 bytes (21 GB) copied, 184.693 s, 116 MB/s
21474836480 bytes (21 GB) copied, 188.318 s, 114 MB/s
21474836480 bytes (21 GB) copied, 192.305 s, 112 MB/s
21474836480 bytes (21 GB) copied, 195.372 s, 110 MB/s
21474836480 bytes (21 GB) copied, 200.711 s, 107 MB/s
21474836480 bytes (21 GB) copied, 204.003 s, 105 MB/s
21474836480 bytes (21 GB) copied, 206.468 s, 104 MB/s
21474836480 bytes (21 GB) copied, 213.033 s, 101 MB/s
21474836480 bytes (21 GB) copied, 217.02 s, 99.0 MB/s
21474836480 bytes (21 GB) copied, 222.62 s, 96.5 MB/s
21474836480 bytes (21 GB) copied, 227.523 s, 94.4 MB/s

As you can see, the performance starts out as expected, around 120MB/s but drops to around 65MB/s, than jumps back to 120MB/s, and so on...

At the moment, I have no clue whatsoever about the cause of these weird results. The results from my ext3 test where as expected (all around 100MB/s). Would someone be able to enlighten me ?

Here are some more system specs:

mdadm controlled raid1
ext4 with default options (mkfs.ext4 /dev/sdb1)
mounted with default options (mount /dev/sdb1 /mountpoint)
disks: WD5003ABYX
os: custom debian kernel 2.6.36 x86

Minion91

Posted 2013-03-14T08:38:18.380

Reputation: 121

Answers

0

Add oflag=direct or oflag=direct,sync to see if you get more consistent results.

My guess is you are slightly exceeding the average sustained write rate, and then it starts to slow down. After some time you may then be experiencing some memory starvation, which could explain why your effective bandwidth halves. See these for some considerations: https://unix.stackexchange.com/questions/30286/can-i-configure-my-linux-system-for-more-aggressive-file-system-caching and https://stackoverflow.com/questions/3019748/how-to-reliably-measure-available-memory-in-linux/3031924#3031924

Make sure you are working on an idle system (check with top and iotop), you should see high iowait while writing.

If you can, boosting the I/O priority can help too: ionice -c1 dd [...]

Check you don't have any power saving enabled. This is a useful resource: http://www.noah.org/wiki/Benchmarks

mr.spuratic

Posted 2013-03-14T08:38:18.380

Reputation: 2 163

using oflag=direct/sync both also circumvent the delayed allocation on ext4, do they not ? And the tests are done on an idle system. I'm gonne check iowait in a second, as that could explain something. How can powersave influence this test ? The disks are constantly being written, so they can't go to power save. And the os isn't heavily stressed, so even if it would go to power save, it still should perform just fine. And lastly the jumps back to 120MB/s are still very strange. – Minion91 – 2013-03-14T14:43:42.393

direct should enable O_DIRECT and hence mballoc. Not 100% sure about delayed allocation, repeat the tests with and without the mount option nodelalloc to confirm. Powersave is something of a guess, there are different levels of powersave (possibly eco and low-noise modes). – mr.spuratic – 2013-03-14T15:20:58.500

Asked: 2013-03-14T08:38:18.380

Viewed: 293 times

Active: 2013-03-14T13:07:23.427