22

LSI offers their CacheCade storage tiering technology, which allows SSD devices to be used as read and write caches to augment traditional RAID arrays.

Other vendors have adopted similar technologies; HP SmartArray controllers have their SmartCache. Adaptec has MaxCache... Not to mention a number of software-based acceleration tools (sTec EnhanceIO, Velobit, FusionIO ioTurbine, Intel CAS, Facebook flashcache?).

Coming from a ZFS background, I make use of different types of SSDs to handle read caching (L2ARC) and write caching (ZIL) duties. Different traits are needed for their respective workloads; Low-latency and endurance for write caching. High capacity for read.

  • Since CacheCade SSDs can be used for write and read cache, what purpose does the RAID controller's onboard NVRAM play?
  • When used as a write cache, what danger is there to the CacheCade SSDs in terms of write endurance? Using consumer SSDs seems to be encouraged.
  • Do writes go straight to SSD or do they hit the controller's cache first?
  • How intelligent is the read caching algorithm? I understand how the ZFS ARC and L2ARC functions. Is there any insight into the CacheCade tiering process?
  • What metrics exist to monitor the effectiveness of the CacheCade setup? Is there a method to observe a cache hit ratio or percentage? How can you tell if it's really working?

I'm interested in opinions and feedback on the LSI solution. Any caveats? Tips?

Community
  • 1
ewwhite
  • 194,921
  • 91
  • 434
  • 799

4 Answers4

9

Since CacheCade SSDs can be used for write and read cache, what purpose does the RAID controller's onboard NVRAM play?

If you leave the write caching feature of the controller enabled, the NVRAM will still be used primarily. The SSD write cache will typically only be used for larger quantities of write data, where the NVRAM alone is not enough to keep up.

When used as a write cache, what danger is there to the CacheCade SSDs in terms of write endurance? Using consumer SSDs seems to be encouraged.

This depends on how often your writes are actually causing the SSD write cache to become necessary... whether or not your drives are able to handle the write load quickly enough that the NVRAM doesn't fill up. In most scenarios I've seen, the write cache gets little to no action most of the time, so I wouldn't expect this to have a big impact on write endurance - most writes to the SSDs are likely to be part of your read caching.

Do writes go straight to SSD or do they hit the controller's cache first?

Answered above... Controller cache is hit first, SSD cache is more of a 2nd line of defense.

How intelligent is the read caching algorithm? I understand how the ZFS ARC and L2ARC functions. Is there any insight into the CacheCade tiering process?

Sorry... no knowledge to contribute on that - hopefully someone else will have some insight?

What metrics exist to monitor the effectiveness of the CacheCade setup? Is there a method to observe a cache hit ratio or percentage? How can you tell if it's working?

It doesn't look like any monitoring tools are available for this as there are with other SAN implementations of this feature set... And since the CacheCade virtual disk doesn't get presented to the OS, you may not have any way to manually monitor activity either. This may just require further testing to verify effectiveness...

Opinion/observation: In a lot of cases (when used correctly, read cache appropriately sized for the working data set) this feature makes things FLY. But in the end, it can be hit-and-miss.

JimNim
  • 2,736
  • 12
  • 23
  • 1
    Hey, just wanted to add that there may be some metrics that are available in OS, from the manual page 220, talking about the MegaCLI options (which can be installed in Linux, Windows, Unix, Bsd, etc). It says `Convention MegaCli -perfmon -start -interval -aN` `Description -perfmon: Specifies collection of performance data.` `The possible parameters are:` `-start: Starts the performance data collection.` `-interval: the time interval (in minutes) that you specified for which you want the performance data collection. The default value is 5 minutes` – NickW Jun 10 '13 at 12:21
  • Link to the pdf: http://www.lsi.com/downloads/Public/Advanced%20Software/LSI%20MegaRAID%20CacheCade%20Software/megaraid-cachecade-user-guide.pdf – NickW Jun 10 '13 at 12:23
  • Hmmm, not too verbose. I'd have to see what the collection format looks like. – ewwhite Jun 10 '13 at 17:22
  • I'd be interested to see what data is actually collected/reported by that if anyone tests that out... – JimNim Jun 18 '13 at 20:21
1

Speaking about hardware solutions I found no way to know exact hit ratio or something. I believe there are 2 reasons for that: the volume behind controller appears as a single drive (and so it should "just work"), and it is hard to count "hits" which will be not for files but rather for HDD sectors so there'll may be some hit rate even on empty HDD which may be confusing. Moreover algorithms behind "hybridisation" are non-public so knowing hitrate won't help much. You just buy it and put it to work - low spendings (compared to pure SSD solution), nice speed impact.

"Buy it and use it" approach is pretty good thing to consider, but the fact is noone knows for sure how to build the fastest combination: should we use several big HDD and several big cache SSD, or should we use many small HDD and several big SSD etc., and what's the difference between 100 or, say, 500 Gb or 2000Gb of SSD cache (even 500 looks overkill if volume hot data are small-sized), and should it be like 2x64Gb or 8x8Gb to have data transfer paralleled. Again, each vendor uses its own algorithm and may change it on next firmware update.

I write this mostly to say that my findings given me strange answer: if you use some general-purpose and general-load-profiled server, then h/w hybrid controller is fine even with relatively small SSDs, but if your tasks used to be specific you'd better go for s/w solution (which you'll be able to choose since you're the only one who knows the load profile) or for some high-priced PCI-card storages.

Alexander
  • 724
  • 2
  • 11
  • 19
1

Here are some claims from Dell:

Storage performance—I/Os per second (IOPS):

  • CacheCade provided an increase of 3x more IOPS than hard drive storage alone
  • SAS SSD storage achieved 8x more IOPS than HDD storage
  • One Express Flash drive resulted in 10.5x more IOPS than 16 HDDs

In terms of IOPS, their numbers seem to be on par. I tend to keep DiskSpd reports from various servers, and if I scale one of the reports I have to the spindle count I have in the cachecade server, I should only get about 750k IOPS. However, in testing this server i was getting over 2M. It was really the only time I've seen cpu load as a factor by the diskspd threads start to be a factor. Usually cpu is still minimal when the disk starts capping out but that was not the case here. I am kicking myself for not running diskspd with and without but oh well.

The other factor here is it is totally transparent...spend a few hundred bucks on a smaller enterprise class ssd, add it as a cachecade volume and you're done. If you've got money to burn on all-ssd storage then it doesn't matter, but for breathing life into physical spindles, I do consider it worth it.

b_levitt
  • 143
  • 6
0

I have tried it on Dell R515 with Dell PERC H700 Raid w 1GB memory with couple 500MB/s SSDs

I did my bench marking few hours after installation, and after 48 hours

I didn't see much improvement for write speed, a little improvement for Read, I did the test a white ago and I don't have numbers now.

But it wasn't significant and I ended up using the storage box without this feature.

From my experience, most of these software are just a joke! if you need storage tiering, then build your own, get reliable hardware from Dell & fill the box with SSDs

At my work place storage tiering work really well HP 3Par & adaptive optimization add-on, it works as advertised but this solution is around 100K :)

user1007727
  • 421
  • 5
  • 20