1

We've got Debian Wheezy host with ZFS on Linux with in. It is used only for serving large files over http. User activity is mostly characterized by a lot of random read.

We've noticed that ~95% ARC hit rate gives us ~80 MByte/s hdd reading activity with the overall ~160 MByte/s server output.

I've already checked ARC and L2 prefetch options are disabled. Linux read ahead for this array is set at default 512 and changing this value doesn't give any sensible effect.

What is the reason of such difference and is there any way to get proper statistic for real ARC usage?

Alexander Klimenko
  • 151
  • 5
  • What tools are you using to observe this? – ewwhite Mar 03 '16 at 06:43
  • @ewwhite atop, iostat and zpool iostat shows same results for hdd and network, arcstat.py and /proc/spl/kstat/zfs/arcstats for ARC performance. – Alexander Klimenko Mar 03 '16 at 06:57
  • What difference are you referring to? If all the data is being served over your network, the network is likely limiting performance too. What is the network hardware? – Andrew Henle Mar 03 '16 at 12:00
  • @AndrewHenle We've got Intel Corporation 82599EB 10-Gigabit SFI/SFP+ (interrupts are spread between cores - 1 irq per 1 core). If network is limited the reading speed should decrease when it meets buffer size, isn't it? – Alexander Klimenko Mar 04 '16 at 05:33
  • @AlexanderKlimenko Only if the filesystem and hardware is unable to keep up with the network transfer. File read-ahead may allow slower hardware to mask delays by simply adding a bit of latency that you won't notice when sending a large file over the network. The HTTP transfer over the network also may have latencies that allow your hardware to keep up. How fast can you transfer data that's known to already be entirely in memory? – Andrew Henle Mar 05 '16 at 15:29
  • @AndrewHenle We got some servers with 40gbit/s networks that serve files only from memory (with 99-100 arc git ratio) we are getting up to 16gbit/s and speed/connections ratio shows that it is not the limit. One of the ideas is that arc stores not only data but metadata too. May be metadata takes other part of those hits? – Alexander Klimenko Mar 06 '16 at 11:31

0 Answers0