1
0
I am observing something that I cannot explain. I am trying to come up with a good filesystem configuration using HP P420i controller and SSDs. Out of curiosity I have tried BTRFS (still unstable so I can't really expect to be able to use it) and noticed that the read speed is about 150% of ext4 - while write speed is comparable.
To be clear, I am using RAID0 with two SSDs with strip size of 256Kb. I have 6 disks so I have created 3 logical disks, 2 SSDs each - just for testing. And then I have formatted them with ext4, XFS and BTRFS.
When I write (something like dd if=/dev/zero of=test2 bs=512k count=20000 conv=fdatasync,fsync) and watch the system using iostats, I see that both BTRFS and EXT4 are writing at approximately the same rate with similar number of write requests:
(ext4 - writing) Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sdb 0.00 0.00 0.00 1791.00 0.00 895.00 1023.43 141.73 78.97 0.56 100.00 (btrfs - writing) Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sdc 0.00 0.00 0.00 1786.00 0.00 893.00 1024.00 137.87 77.21 0.56 100.10
When I read, I observe different picture:
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util (ext4 - reading) sdb 0.00 0.00 4782.00 0.00 597.75 0.00 256.00 1.57 0.33 0.18 84.10 (btrfs - reading) Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sdc 207.00 0.00 1794.00 0.00 886.40 0.00 1011.90 10.59 5.90 0.56 100.00 (xfs - reading) Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sdd 0.00 0.00 4623.00 0.00 577.88 0.00 256.00 1.71 0.37 0.21 97.00
And this is what I see if I just try to read the block device with dd:
(reading block device) Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sdb 132055.00 0.00 4259.00 0.00 532.38 0.00 256.00 1.61 0.38 0.23 99.80 sdc 131750.00 0.00 4250.00 0.00 531.25 0.00 256.00 1.58 0.37 0.24 100.00 sdc 142476.00 0.00 4596.00 0.00 574.50 0.00 256.00 1.61 0.35 0.20 92.40
All settings seem to be identical (I/O scheduler, readahead...) for all 3 logical volumes.
So, everything else being equal, I clearly see that btrfs does much fewer reads per second and clearly reads more bytes per second. And that number of rrqm/s - this is the number of merged requests. I can only see it on the device that is formatted with btrfs.
Kernel: 3.8.13-35.3.5.el6uek.x86_64 #2 SMP Fri Aug 8 21:58:11 PDT 2014 x86_64 x86_64 x86_64 GNU/Linux
Puzzled. What is so special btrfs can do (and ext4 and xfs do not do) to make such a difference?
P.S. No, compression is disabled with mount option compress=no
1Don't use
/dev/zero
. The FS might optimize for that somehow. Use/dev/urandom
(unlocked random number generator). – allquixotic – 2014-08-27T00:32:38.057I have tried with a random file (took a while to generate enough data for sizable test) - same results. btrfs is way faster to read and iostat shows the same request pattern. I start suspecting that btrfs's readahead feature may have something to do with this mystery. If I can find how to turn it off, I could prove it. – Nikolai Grigoriev – 2014-08-27T01:18:02.587
And yes - I can confirm it is btrfs readahead that does the trick. It can be controlled via /sys/class/bdi/btrfs-XX/read_ahead_kb. Default is 4096 (4Mb). If I disable it or set it much lower value I see the performance decrease. At 128Kb I see 2600 reads/s and only ~40 merged requests. And the total speed drops to 380Mb/s. – Nikolai Grigoriev – 2014-08-27T01:40:46.903
1Trying bonnie++ for disk benchmarking in linux – cybernard – 2014-08-30T00:57:05.867