FreeNAS 11.2
I have a USB boot device and 4 hard disks in a ZFS pool. I'd like those 4 disks to spin down when not being accessed, i.e. most of the time.
In Storage -> Disks I set the "HDD Standby" to 20 minutes and the "Adv. Power Management" level to 127.
The disks now spin down, but nearly immediately spin back up. What else do I need to do to have the disks spin down and stay spun down until a user operation needs the pool?
There are a handful of reasons disks might be accessed and spun up by the system. The following changes let my disks stay spun down.
By default S.M.A.R.T. checks drives every 30 seconds, waking them up.
Services -> S.M.A.R.T. -> Configure
Set "Power Mode" to "Standby". This will prevent checking disks that are in standby mode (spun down).
I've noticed that the Disk IO plots on the Reporting page show writes to the pool even though no clients are connected. This is writes to the System Dataset, I believe mostly collectd writing logs and metrics.
By default the System Dataset is placed on the main pool. It can be placed elsewhere.
Use System -> System Dataset to move this to a flash device.
If you boot from an SSD, use freenas-boot. Else use another SSD.
You can use a USB flash device, but like Richard says in the comments this dataset receives a lot of writes and will likely quickly wear out a USB stick. SATA SSD prices have plummeted to around $30 now, so it's hard to justify the extra downtime and hassle to literally save a handful of dollars. A USB stick I used failed in months and the NAS was not bootable at all.
Apart from the above, of course make sure any jails or VM storage you create live on a flash device. When creating your first jail the system will suggest you place jail storage on your non-system pool, so likely your disks. This can be changed via Jails -> Activated Pool.