Similar ZFS performance on HDD and SSD

Question

I've created two zfs pools (one with ssds, the other rust) and run bonnie against each to try to understand their performance but I find the output somewhat confusing as the performance seems practically identical. I was wondering if I'm misinterpreting the output or if someone could offer advice on what might be going on?

Both pools have a single mirror vdev containing two disks.

SSD Pool - 2 x 500GB Samsung 850 Pro, ashift=13

bonnie -u root -d /ssdpool/images/

Version      1.98   ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
is7734 126G::8192:5  177k  99  475m  97  438m  98  376k  99  1.4g  99 +++++ +++
Latency             69204us     100ms     104ms   33890us     925us    6948us
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
      files:max:min  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
is77346          16 16845  98 +++++ +++  2463  10 +++++ +++ +++++ +++ 22164  98
Latency              6213us     708us    5089ms    2565us      42us     910us

bonnie++ -d /ssdpool/images/ -s 128G:4k -u root

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
is77346     128G:4k  180k  99  343m  98  305m  99  396k  99  989m  99 +++++ +++
Latency             31352us     102ms     101ms   16948us     952us   12772us
Version  1.98       ------Sequential Create------ --------Random Create--------
is77346             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16     0  98 +++++ +++ 724249376  10 +++++ +++ +++++ +++ 724249376  99
Latency              2523us     488us    5379ms    2562us      92us     926us

fio --name=random-write --ioengine=sync --iodepth=4 --rw=randwrite --bs=4k --direct=0 --size=256m --numjobs=16 --end_fsync=1
[...]
Run status group 0 (all jobs):
  WRITE: bw=212MiB/s (222MB/s), 13.2MiB/s-15.5MiB/s (13.9MB/s-16.2MB/s), io=4096MiB (4295MB), run=16565-19359msec

HDD Pool - 2 x 4TB WD Green, ashift=12

bonnie -u root -d /hddpool/images/

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
is77346        126G  166k  94  391m  86  345m  82  387k  99  1.4g  99 11676 399
Latency             97954us     101ms     105ms   39728us     939us   10904us
Version  1.98       ------Sequential Create------ --------Random Create--------
is77346             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16     0  73 +++++ +++ 724249376   7     0  78 +++++ +++ 724249376  16
Latency               100ms     667us    6853ms    3384us      20us     563ms

bonnie++ -d /hddpool/images/ -s 128G:4k -u root

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
is77346     128G:4k  166k  99  315m  96  268m  96  355k  99  979m  99  6563 259
Latency             32958us     101ms     103ms   26880us    1019us   11142us
Version  1.98       ------Sequential Create------ --------Random Create--------
is77346             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16     0  92 +++++ +++ 724249376   7     0  81 +++++ +++ 724249376  18
Latency              2562us     949us    6462ms    3451us      31us     626ms

fio --name=random-write --ioengine=sync --iodepth=4 --rw=randwrite --bs=4k --direct=0 --size=256m --numjobs=16 --end_fsync=1
[...]
Run status group 0 (all jobs):
  WRITE: bw=33.0MiB/s (35.6MB/s), 2175KiB/s-2623KiB/s (2227kB/s-2686kB/s), io=4096MiB (4295MB), run=99950-120548msec

In case it's also worth mentioning the other config in the system:

• OS is Ubuntu 20.04 running on SSD
• 64GB RAM
• both pools use: atime=off, recordsize=64k, compression=lz4
• I haven't explicitly configured zil or l2arc so unless they use the OS disk by default then they won't be configured.

On @francisaugusto suggestion in the comments I ran fio (see results above) and it shows a significant difference between the two pools. I'm still unsure why bonnie gives such similar results but I feel reassured that things are working properly. If anyone is reading this in the future, maybe give bonnie a miss and start out with fio.