So recently upon testing a ZoL system we found out poor performance of random and sequential reads and poor performance of random writes on our SSDs.
Our system is a stripe of 2x Samsung 1TB 850Evo SSDs to test ZFS performance and it was abysmal compared to LVM: reads are slower than HDDs and writes are not on par of expected 1.7GBs we get on LVMs. It's weird because our FreeBSD backup server has slow HDDs and older type of SSD and performs better at the same test.
The system is somewhat deprived of RAM though (zfs gets 4gb for arc and everything else is taken by VMs) however with no cache and no sync the performance is still not even close to anything.
So we are looking into buying newer systems based on AMD Epyc and setting up either full NVMe or NVMe with SSDs with disabling cache to free up ram from ZFS at least a bit (we want it to use 10GB max for everything). We don't really need all of security features of ZFS apart checksumming (but with SSDs it seems to be redundant as SSD run an inside checksumming system) so SSDs will be a stripe of vdevs.
We prefer ZFS for zle on thin-provisioned zvols and the ease of snapshotting and incremental backups to remote system (which runs ZFS as well).
However the struggle for performance is hard...
Would greatly appreciate any advice