I have ZFS pool RAIDZ-1 on 4 x 3.84TB SATA3 'enterprise' SSDs.
zfs create rpool/fio
zfs set primarycache=none rpool/fio
fio --ioengine=sync --filename=[X] --direct=1 --rw=read --bs=4K --numjobs=1 --iodepth=1 --runtime=10 --time_based --name=fio
Results:
--filename=/dev/sdg:
read: IOPS=25.1k, BW=98.2MiB/s (103MB/s)
--filename=/rpool/fio/testx
read: IOPS=1408, BW=5634KiB/s (5769kB/s)
zpool configuration:
NAME PROPERTY VALUE SOURCE
rpool/fio type filesystem -
rpool/fio creation Thu Nov 18 11:04 2021 -
rpool/fio used 30.5G -
rpool/fio available 9.69T -
rpool/fio referenced 30.5G -
rpool/fio compressratio 1.05x -
rpool/fio mounted yes -
rpool/fio quota none default
rpool/fio reservation none default
rpool/fio recordsize 128K default
rpool/fio mountpoint /rpool/fio default
rpool/fio sharenfs off default
rpool/fio checksum on default
rpool/fio compression lz4 local
rpool/fio atime off inherited from rpool
rpool/fio devices on default
rpool/fio exec on default
rpool/fio setuid on default
rpool/fio readonly off default
rpool/fio zoned off default
rpool/fio snapdir hidden default
rpool/fio aclmode discard default
rpool/fio aclinherit restricted default
rpool/fio createtxg 11836 -
rpool/fio canmount on default
rpool/fio xattr on default
rpool/fio copies 1 default
rpool/fio version 5 -
rpool/fio utf8only off -
rpool/fio normalization none -
rpool/fio casesensitivity sensitive -
rpool/fio vscan off default
rpool/fio nbmand off default
rpool/fio sharesmb off default
rpool/fio refquota none default
rpool/fio refreservation none default
rpool/fio guid 11473612800347485365 -
rpool/fio primarycache none local
rpool/fio secondarycache all default
rpool/fio usedbysnapshots 0B -
rpool/fio usedbydataset 30.5G -
rpool/fio usedbychildren 0B -
rpool/fio usedbyrefreservation 0B -
rpool/fio logbias latency default
rpool/fio objsetid 9843 -
rpool/fio dedup off default
rpool/fio mlslabel none default
rpool/fio sync standard inherited from rpool
rpool/fio dnodesize legacy default
rpool/fio refcompressratio 1.05x -
rpool/fio written 30.5G -
rpool/fio logicalused 32.0G -
rpool/fio logicalreferenced 32.0G -
rpool/fio volmode default default
rpool/fio filesystem_limit none default
rpool/fio snapshot_limit none default
rpool/fio filesystem_count none default
rpool/fio snapshot_count none default
rpool/fio snapdev hidden default
rpool/fio acltype off default
rpool/fio context none default
rpool/fio fscontext none default
rpool/fio defcontext none default
rpool/fio rootcontext none default
rpool/fio relatime off default
rpool/fio redundant_metadata all default
rpool/fio overlay on default
rpool/fio encryption off default
rpool/fio keylocation none default
rpool/fio keyformat none default
rpool/fio pbkdf2iters 0 default
rpool/fio special_small_blocks 0 default
Is there something wrong with the test or the ZFS? Any advice/insights appreciated! Thanks.
UPDATE 1:
I've update the initial test to --bs=4K and the story broadly stays the same... Is testing on the raw device valid?
UPDATE 2:
It seems I get a different picture from zpool iostat to fio output. I've adjust the fio test to randion r/w with 64 jobs, which should reveal SSD benchmark more 'realistically'. NB the system has no other workload.
On SSD:
fio --ioengine=libaio --filename=/rpool/fio/testx --size=4G --time_based --name=fio --group_reporting --runtime=10 --direct=1 --sync=1 --iodepth=1 --rw=randrw --bs=4K --numjobs=64
read: IOPS=4022, BW=15.7MiB/s (16.5MB/s)
write: IOPS=4042, BW=15.8MiB/s (16.6MB/s)
# zpool iostat -vy rpool 5 1
capacity operations bandwidth
pool alloc free read write read write
---------------------------------------------------- ----- ----- ----- ----- ----- -----
rpool 216G 27.7T 28.1K 14.5K 1.17G 706M
raidz1 195G 13.8T 13.9K 7.26K 595M 358M
ata-INTEL_SSDSC2KB038TZ_BTYI13730BAV3P8EGN-part3 - - 3.60K 1.73K 159M 90.3M
ata-INTEL_SSDSC2KB038TZ_BTYI13730B9Q3P8EGN-part3 - - 3.65K 1.82K 150M 89.0M
ata-INTEL_SSDSC2KB038TZ_BTYI13730B9G3P8EGN-part3 - - 3.35K 1.83K 147M 90.0M
ata-INTEL_SSDSC2KB038TZ_BTYI13730BAT3P8EGN-part3 - - 3.34K 1.89K 139M 88.4M
raidz1 21.3G 13.9T 14.2K 7.21K 604M 348M
sde - - 3.39K 1.81K 149M 87.5M
sdf - - 3.35K 1.90K 139M 86.3M
sdg - - 3.71K 1.70K 163M 87.8M
sdh - - 3.69K 1.81K 152M 86.4M
---------------------------------------------------- ----- ----- ----- ----- ----- -----
On HDD:
read: IOPS=1382, BW=5531KiB/s
write: IOPS=1385, BW=5542KiB/s
$ zpool iostat -vy rpool 5 1
capacity operations bandwidth
pool alloc free read write read write
-------------------------------- ----- ----- ----- ----- ----- -----
rpool 160G 18.0T 3.07K 2.71K 393M 228M
mirror 32.2G 3.59T 624 589 78.0M 40.2M
scsi-35000c500de5c67f7-part3 - - 321 295 40.1M 20.4M
scsi-35000c500de75a863-part3 - - 303 293 37.9M 19.7M
mirror 31.9G 3.59T 625 551 78.2M 49.9M
scsi-35000c500de2bd6bb-part3 - - 313 274 39.1M 24.2M
scsi-35000c500de5ae5a7-part3 - - 312 277 39.0M 25.7M
mirror 32.2G 3.59T 648 548 81.1M 45.9M
scsi-35000c500de5ae667-part3 - - 320 279 40.1M 23.0M
scsi-35000c500de2bd2d3-part3 - - 328 268 41.0M 23.0M
mirror 31.6G 3.59T 612 536 76.5M 45.5M
scsi-35000c500de5ef20f-part3 - - 301 266 37.7M 22.7M
scsi-35000c500de5edbfb-part3 - - 310 269 38.9M 22.8M
mirror 32.0G 3.59T 629 555 78.7M 46.5M
scsi-35000c500de5c6f7f-part3 - - 318 283 39.8M 23.1M
scsi-35000c500de5c6c5f-part3 - - 311 272 38.9M 23.4M
-------------------------------- ----- ----- ----- ----- ----- -----
Per device performace looks fine on the HDDs, and the SSDs look to be roughtly 5x faster than HDDs in random r/w. Is that to be expected? I was expecting 10-100x faster...
Thanks for your attention!
