Why is ZFS gzip-9 compression so much worse than actually using zip files?

Question

I am working with some very large packet capture files (~150GB across 30 files) but because of how highly redundant packet captures are, these compress to ~7% (14.29x) of their originally file size using .zip files.

I was wondering if I could work with these files "natively" by creating a ZFS partation with the "compression=gzip-9" setting. I created the partition, copied the packet captures over, and the end result was a compression ratio of 20.83% (4.80x).

I am wondering why there is such a large discrepancy.

Answer 1

This is because each record (file block) is compressed independently (to allow reading from the middle of the file for large files). Set large record size:

zfs set recordsize=128k zpool/fsname

or create new filesystem with large record size:

zfs create -o recordsize=128k zpool/fsname

This setting affects only new files created in the filesystem fsname.