0

I have ZFS pool RAIDZ-1 on 4 x 3.84TB SATA3 'enterprise' SSDs.

zfs create rpool/fio
zfs set primarycache=none rpool/fio

fio --ioengine=sync --filename=[X] --direct=1 --rw=read --bs=4K --numjobs=1 --iodepth=1 --runtime=10 --time_based --name=fio

Results:

--filename=/dev/sdg:        
  read: IOPS=25.1k, BW=98.2MiB/s (103MB/s)

--filename=/rpool/fio/testx
  read: IOPS=1408, BW=5634KiB/s (5769kB/s)

zpool configuration:

NAME       PROPERTY              VALUE                  SOURCE
rpool/fio  type                  filesystem             -
rpool/fio  creation              Thu Nov 18 11:04 2021  -
rpool/fio  used                  30.5G                  -
rpool/fio  available             9.69T                  -
rpool/fio  referenced            30.5G                  -
rpool/fio  compressratio         1.05x                  -
rpool/fio  mounted               yes                    -
rpool/fio  quota                 none                   default
rpool/fio  reservation           none                   default
rpool/fio  recordsize            128K                   default
rpool/fio  mountpoint            /rpool/fio             default
rpool/fio  sharenfs              off                    default
rpool/fio  checksum              on                     default
rpool/fio  compression           lz4                    local
rpool/fio  atime                 off                    inherited from rpool
rpool/fio  devices               on                     default
rpool/fio  exec                  on                     default
rpool/fio  setuid                on                     default
rpool/fio  readonly              off                    default
rpool/fio  zoned                 off                    default
rpool/fio  snapdir               hidden                 default
rpool/fio  aclmode               discard                default
rpool/fio  aclinherit            restricted             default
rpool/fio  createtxg             11836                  -
rpool/fio  canmount              on                     default
rpool/fio  xattr                 on                     default
rpool/fio  copies                1                      default
rpool/fio  version               5                      -
rpool/fio  utf8only              off                    -
rpool/fio  normalization         none                   -
rpool/fio  casesensitivity       sensitive              -
rpool/fio  vscan                 off                    default
rpool/fio  nbmand                off                    default
rpool/fio  sharesmb              off                    default
rpool/fio  refquota              none                   default
rpool/fio  refreservation        none                   default
rpool/fio  guid                  11473612800347485365   -
rpool/fio  primarycache          none                   local
rpool/fio  secondarycache        all                    default
rpool/fio  usedbysnapshots       0B                     -
rpool/fio  usedbydataset         30.5G                  -
rpool/fio  usedbychildren        0B                     -
rpool/fio  usedbyrefreservation  0B                     -
rpool/fio  logbias               latency                default
rpool/fio  objsetid              9843                   -
rpool/fio  dedup                 off                    default
rpool/fio  mlslabel              none                   default
rpool/fio  sync                  standard               inherited from rpool
rpool/fio  dnodesize             legacy                 default
rpool/fio  refcompressratio      1.05x                  -
rpool/fio  written               30.5G                  -
rpool/fio  logicalused           32.0G                  -
rpool/fio  logicalreferenced     32.0G                  -
rpool/fio  volmode               default                default
rpool/fio  filesystem_limit      none                   default
rpool/fio  snapshot_limit        none                   default
rpool/fio  filesystem_count      none                   default
rpool/fio  snapshot_count        none                   default
rpool/fio  snapdev               hidden                 default
rpool/fio  acltype               off                    default
rpool/fio  context               none                   default
rpool/fio  fscontext             none                   default
rpool/fio  defcontext            none                   default
rpool/fio  rootcontext           none                   default
rpool/fio  relatime              off                    default
rpool/fio  redundant_metadata    all                    default
rpool/fio  overlay               on                     default
rpool/fio  encryption            off                    default
rpool/fio  keylocation           none                   default
rpool/fio  keyformat             none                   default
rpool/fio  pbkdf2iters           0                      default
rpool/fio  special_small_blocks  0                      default

Is there something wrong with the test or the ZFS? Any advice/insights appreciated! Thanks.

UPDATE 1:

I've update the initial test to --bs=4K and the story broadly stays the same... Is testing on the raw device valid?

UPDATE 2:

It seems I get a different picture from zpool iostat to fio output. I've adjust the fio test to randion r/w with 64 jobs, which should reveal SSD benchmark more 'realistically'. NB the system has no other workload.

On SSD:

fio --ioengine=libaio --filename=/rpool/fio/testx --size=4G --time_based --name=fio --group_reporting --runtime=10 --direct=1 --sync=1 --iodepth=1 --rw=randrw  --bs=4K --numjobs=64

read: IOPS=4022, BW=15.7MiB/s (16.5MB/s)
write: IOPS=4042, BW=15.8MiB/s (16.6MB/s)


# zpool iostat -vy rpool 5 1
                                                        capacity     operations     bandwidth 
pool                                                  alloc   free   read  write   read  write
----------------------------------------------------  -----  -----  -----  -----  -----  -----
rpool                                                  216G  27.7T  28.1K  14.5K  1.17G   706M
  raidz1                                               195G  13.8T  13.9K  7.26K   595M   358M
    ata-INTEL_SSDSC2KB038TZ_BTYI13730BAV3P8EGN-part3      -      -  3.60K  1.73K   159M  90.3M
    ata-INTEL_SSDSC2KB038TZ_BTYI13730B9Q3P8EGN-part3      -      -  3.65K  1.82K   150M  89.0M
    ata-INTEL_SSDSC2KB038TZ_BTYI13730B9G3P8EGN-part3      -      -  3.35K  1.83K   147M  90.0M
    ata-INTEL_SSDSC2KB038TZ_BTYI13730BAT3P8EGN-part3      -      -  3.34K  1.89K   139M  88.4M
  raidz1                                              21.3G  13.9T  14.2K  7.21K   604M   348M
    sde                                                   -      -  3.39K  1.81K   149M  87.5M
    sdf                                                   -      -  3.35K  1.90K   139M  86.3M
    sdg                                                   -      -  3.71K  1.70K   163M  87.8M
    sdh                                                   -      -  3.69K  1.81K   152M  86.4M
----------------------------------------------------  -----  -----  -----  -----  -----  -----

On HDD:

read: IOPS=1382, BW=5531KiB/s
write: IOPS=1385, BW=5542KiB/s

$ zpool iostat -vy rpool 5 1
                                    capacity     operations     bandwidth 
pool                              alloc   free   read  write   read  write
--------------------------------  -----  -----  -----  -----  -----  -----
rpool                              160G  18.0T  3.07K  2.71K   393M   228M
  mirror                          32.2G  3.59T    624    589  78.0M  40.2M
    scsi-35000c500de5c67f7-part3      -      -    321    295  40.1M  20.4M
    scsi-35000c500de75a863-part3      -      -    303    293  37.9M  19.7M
  mirror                          31.9G  3.59T    625    551  78.2M  49.9M
    scsi-35000c500de2bd6bb-part3      -      -    313    274  39.1M  24.2M
    scsi-35000c500de5ae5a7-part3      -      -    312    277  39.0M  25.7M
  mirror                          32.2G  3.59T    648    548  81.1M  45.9M
    scsi-35000c500de5ae667-part3      -      -    320    279  40.1M  23.0M
    scsi-35000c500de2bd2d3-part3      -      -    328    268  41.0M  23.0M
  mirror                          31.6G  3.59T    612    536  76.5M  45.5M
    scsi-35000c500de5ef20f-part3      -      -    301    266  37.7M  22.7M
    scsi-35000c500de5edbfb-part3      -      -    310    269  38.9M  22.8M
  mirror                          32.0G  3.59T    629    555  78.7M  46.5M
    scsi-35000c500de5c6f7f-part3      -      -    318    283  39.8M  23.1M
    scsi-35000c500de5c6c5f-part3      -      -    311    272  38.9M  23.4M
--------------------------------  -----  -----  -----  -----  -----  -----

Per device performace looks fine on the HDDs, and the SSDs look to be roughtly 5x faster than HDDs in random r/w. Is that to be expected? I was expecting 10-100x faster...

Thanks for your attention!

simonw
  • 13
  • 4

0 Answers0