Are ZFS's ARC counters persistent?

Question

Suppose I have some data that is accessed frequently and some less-frequently accessed data. I understand that the ARC will eventually learn to stop evicting the frequently-accessed data to make room for other data just because the other data was recently accessed. Suppose I've been running a ZFS system for quite some time and it has picked up on this access pattern.

If I reboot my box and frequently accessed data is just read and thus in the ARC, will new reads of other data cause its eviction? Or does ZFS remember that that data is frequently used?

If it's implementation-dependent, then this question specifically pertains to Solaris.

Answer 1

The ARC doesn’t have any persistent counters, so it will have to see your access pattern again to determine that something is used frequently. However, it won’t evict anything until it’s full / there’s memory pressure from something else on the system, so at first boot everything that’s read or written will end up in the cache. As long as your frequently accessed data gets read twice before then, it should make it into the “MFU” list.

You can read more about the ARC algorithm here. In a nutshell, it is actually made from two LRU lists, one for stuff that has been acccessed once (the “MRU list”), and one for stuff that has been accessed twice or more (the “MFU list” — yes, the name is incorrect, it really uses LRU to evict because LRU is faster and simpler to implement than MFU). There are also “ghost lists” that keep track of recently evicted keys (but not data) from each list, which help it determine how big the two caches should be relative to each other.

The L2ARC is persistent (usually stored on SSDs) but does not use the ARC algorithm (yet another suboptimal name). I believe it simply round-robins the data in the cache. Also, I believe you can’t reuse it after a reboot unless you’re using Nexenta’s fork of OpenZFS (I don’t think they’ve landed it upstream yet).

Except for Nexenta’s “Persistent L2ARC” feature, nothing in this answer is platform-specific.