5

I was reading up on L2ARC for my ZFS. I currently have two Pools, One for Data, and the other for Linux Containers and Virtual Machines. I would like some help with L2ARC, I know how to add it but was wonder if it would benefit me.

I'm running Ubuntu 16.04 with 6-8 LXC containers and sometimes a virtual machine or two in headless-virtualbox, I was wondering if adding a L2ARC will help with performance of my Virtual Machines and my LXC Containers.

Current Setup

pool: black
state: ONLINE
scan: scrub repaired 0 in 3h57m with 0 errors on Mon Jul 11 04:12:35 2016
config:

    NAME                                          STATE     READ WRITE CKSUM
    black                                         ONLINE       0     0     0
      mirror-0                                    ONLINE       0     0     0
        ata-WDC_WD10JFCX-68N6GN0_WD-WX91A65P0T3Z  ONLINE       0     0     0
        ata-WDC_WD10JFCX-68N6GN0_WD-WXA1EA4CSNJF  ONLINE       0     0     0

errors: No known data errors

pool: raid
state: ONLINE
scan: scrub repaired 0 in 26h32m with 0 errors on Tue Jul 12 04:04:11 2016
config:

    NAME                                          STATE     READ WRITE CKSUM
    raid                                          ONLINE       0     0     0
      raidz1-0                                    ONLINE       0     0     0
        ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N6LTTK5S  ONLINE       0     0     0
        ata-WDC_WD30EFRX-68EUZN0_WD-WCC4NJTT02XC  ONLINE       0     0     0
        ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N6LLTRUY  ONLINE       0     0     0
      raidz1-1                                    ONLINE       0     0     0
        ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0UHPYUR  ONLINE       0     0     0
        ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N2ET6YU3  ONLINE       0     0     0
        ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N6TZJ7ZF  ONLINE       0     0     0

 errors: No known data errors

As you can see all my drives are WD Reds. I would add the L2ARC to Pool black, which is my LXC and Virtual Machine storage. I just did a write test to the pool and got 70-80MBPS of a 20GB file(video file), Read 80-90MBPS. I would add a 120GB or 240GB ssd to it as L2ARC.

Would this help the performance of the Pool for what I do?

Linux Containers; Emby Media Server, Apache, Web Dev Apache,Gitlab,Torrent,OpenWrt Dev,VPN Server,Mysql

Virtual Machines; Windows mainly for testing, odd Linux Server/Desktop for testing.

Specs AMD Phenom™ II X6 1090T 32GB DDR3 Ram

Jonny
  • 53
  • 6

2 Answers2

3

The general advice for this is to maximize your system RAM (ARC), and then add L2ARC if the need is there.

This can be measured using the arcstat command in your server.

Something like:

arcstat.py -f "time,read,hit%,hits,miss%,miss,arcsz,c" 1

Where the output shows the time number of reads/second, the hit ratio, the number of hits/misses, and the present and target ARC size:

    time  read  hit%  hits  miss%  miss  arcsz     c
15:15:21  1.5K    99  1.5K      0     2    55G   55G
15:15:22  221K    99  220K      0   584    55G   55G
15:15:23  245K    99  245K      0   544    55G   55G
15:15:24  262K    99  262K      0   551    55G   55G
15:15:25  321K    99  321K      0   489    55G   55G
15:15:26  262K    99  261K      0   479    55G   55G
15:15:27  203K    99  203K      0   585    55G   55G
15:15:28  207K    97  202K      2  5.3K    55G   55G
user
  • 4,267
  • 4
  • 32
  • 70
ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • I get the following from the output http://pastebin.com/9hBeMrcD Do you think its worth adding, no VMs are running currently just LXC – Jonny Jul 15 '16 at 22:25
2

Your pool black currently consists of two mirrored WD10JFCX drives, which according to Western Digital's spec sheet are:

  • 5400 rpm (rotational latency about 5 ms on average, 11 ms worst case)
  • 16 MB cache
  • 1 TB
  • 2.5" form factor
  • SATA 6 Gb/s (SATA 3)

Neither the 5400 rpm rotation speed nor the puny 16 MB on-drive cache is conducive to highest performance, and the 1 TB is pretty small by today's standards as SATA drives go. 5400 rpm gives you, in theory, about 90 IOPS to the platters, and assuming that on average the platter will be half a revolution away from the data, its 11 ms rotation time gives you an average about 5 ms rotational latency.

Compare to this that you can get the Intel 540S series for about $308 apiece from Amazon (in the US), for which Intel gives the specs as:

  • Up to 78,000 IOPS doing random reads, 85,000 IOPS doing random writes (latency 50 µs)
  • Up to 560 MB/s sequential read, 480 MB/s sequential write
  • 1 TB
  • 2.5" form factor
  • SATA 6 Gb/s (SATA 3)

Even if we play nice, you are looking at a 1,000-fold reduction in latency and a 1,000-fold increase in IOPS for about $600 and change. On a performance per dollar ratio, I'm pretty sure there's no beating that by adding L2ARC to your pool, and it's a drop in replacement. If you are cash-strapped, you can even replace one drive at a time with a corresponding SSD and still start reaping the benefits immediately.

Replacing those 5400 rpm disks with SSDs will do more to your black pool performance than any L2ARC you can realistically throw at it, almost regardless of your workload. What's more, while the Intel 540S series may or may not be a good fit for your requirements (see also here), even such a cheap SSD is almost certain to provide a better return on investment as compared to simply adding a L2ARC. If you want something higher spec'd, looking just at Intel, a DC S3500 1.2 TB SSD will set you back about $900 apiece, and a DC S3710 1.2 TB SSD will set you back about $1,000 apiece, both of which have similar performance but are likely to hold up better to use over time.

No endorsement of any specific products or resellers mentioned is either expressed or implied.

user
  • 4,267
  • 4
  • 32
  • 70
  • I'll think about moving the black pool to two SSDs, would adding a ssd for l2arc be worth it at all? I would be using Samsung 850s SSD either way – Jonny Jul 17 '16 at 19:25
  • Good infomation though, Thank you. – Jonny Jul 17 '16 at 19:25
  • 1
    @Jonny To provide any substantial benefit, the L2ARC must be significantly faster (in terms of IOPS and/or throughput) than the main storage media, and it must be populated with your working data set. (Last I looked, L2ARC in ZoL was non-persistent.) Using a SSD as a L2ARC for a SSD-backed pool seems unlikely to provide any significant performance benefit. – user Jul 17 '16 at 19:33
  • Thanks for your help. I think i might add a SSD to the l2arc – Jonny Jul 17 '16 at 20:42
  • This advice has the added benefit of eliminating the need for a separate device for the ZIL, as your main pool disks are already fast SSDs. Of course, write endurance becomes a consideration then, but it depends on your workload. – user121391 Jul 18 '16 at 10:05
  • @user121391 [Write endurance being a problem is a scam.](https://serverfault.com/q/507521/58408) (Can you tell I'm getting a tad bit tired of write endurance constantly being brought up as a reason against SSDs?) Pick an *appropriately spec'd* SSD and write endurance will be the least of your concerns. – user Jul 18 '16 at 10:51
  • 1
    @MichaelKjörling Yes, that's what I meant to say - maybe my comment was not precise enough. Consideration means to choose the right SSD for the job depending on the workload, not to not choose an SSD at all. Or if the choice is fixed, consider mitigations like manual over-provisioning where relatively cheap disks can have a longer live than advertised (at the cost of storage capacity of course). – user121391 Jul 18 '16 at 11:14