How come my RAID-6 outperforms my RAID-10?
Docker container on Synology with 8x4TB RAID-10:
[root@centos Files]# bonnie++ -d /Files/tmp -r 16G -s 32G -n 1 -f -b -u root
Using uid:0, gid:0.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
centos 32G 454106 19 177752 12 336051 11 174.9 9
Latency 637ms 437ms 271ms 1199ms
Version 1.97 ------Sequential Create------ --------Random Create--------
centos -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
1 19 1 +++++ +++ 24 2 19 1 +++++ +++ 23 3
Latency 401ms 63us 177ms 177ms 19us 122ms
1.97,1.97,centos,1,1503010478,32G,,,,454106,19,177752,12,,,336051,11,174.9,9,1,,,,,19,1,+++++,+++,24,2,19,1,+++++,+++,23,3,,637ms,437ms,,271ms,1199ms,401ms,63us,177ms,177ms,19us,122ms
FreeNAS 6x4TB RAID-Z2, RAID-6 from a jail:
root@testjail:/ # bonnie++ -d /Files/tmp -r 16G -s 32G -n 1 -f -b -u root
Using uid:0, gid:0.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
testjail 32G 1158187 63 850172 70 3130981 97 7888 90
Latency 173ms 79669us 13285us 197ms
Version 1.97 ------Sequential Create------ --------Random Create--------
testjail -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
1 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 19728us 41us 2506us 102us 20us 154us
1.97,1.97,testjail,1,1503021586,32G,,,,1158187,63,850172,70,,,3130981,97,7888,90,1,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,,173ms,79669us,,13285us,197ms,19728us,41us,2506us,102us,20us,154us
root@testjail:/ #
Specs: 16GB DDR4 (2133 vs 2400)
- Synology RS3617RPxs - Intel Xeon D-1521 - 8 x WDD Red 4TB in Raid 10 - Quad Core 2,2 (base) / 2,7 (turbo) GHz - 2 x 8 GB DDR4 ECC UDIMM
- FreeNAS VMware VM - 1 x 16GB allocated RAM (out of 2x16GB) - 16GB NVMe bootdisk only - 6x4TB desktop drives - Was difficult to get working.
My expectations were ~400MB/sec write on a good day for the RAIDZ2 as some drives are really old and ~750MB/sec for the RAID10 array.
But the raidz2 seems impossibly fast and the raid10 seems very slow.