2

I've bought a used HP MDS 600 to use as JBOD to a ZFS filesystem.

Originally I just assumed that it was a pretty 'dumb' box and I'd be able to use a SAS card like the 16-port LSI one, and most any SATA drives, in particular I was thinking of the 3TB WD Caviar Reds.

Any contrary experience? I have 25 days or so to 'return' this thing. I'd be willing to spend a little more $$ on a genuine HP SAS card, but I definitely am not going to be able to afford to stock this thing with supremely overpriced HP 'approved' drives.

Update: I'm planning on using a Dell PowerEdge 1950 which has two empty PCIe 8x slots.

ewwhite
  • 194,921
  • 91
  • 434
  • 799
Cyclone
  • 557
  • 4
  • 15

1 Answers1

6

The HP Storageworks MDS600 is ZFS-compatible. I can't speak to the zoning capabilities and control from within an OS, but it sounds like you're going to be using this with a single server head.

I would recommend using LSI controllers, but possibly with two smaller controllers, like the LSI 9205e instead of a single 16-port model. Assuming your MDS has multiple controller interfaces, you may want to leverage SAS multipath.

That brings up the matter of disk selection...

Please don't use SATA drives in your production ZFS deployment.

No good can come from it... Nexenta barely supports SATA. The failures on SATA ZFS arrays can be spectacular. There's also a performance penalty...

if you're seeking capacity, go a little further and spec midline or nearline SAS disks. They are mechanically-equivalent to the large SATA disks, but feature a SAS interface, which has performance, error-correction and reliability implications. Plus, they are dual-ported and support multipath. If capacity isn't an issue and performance is the focus, I'd go with high-RPM enterprise disks...

A few notes...

  • HP-labeled SAS controllers are of no benefit here. They're rebadged LSI controllers and not on the same tier as the 9205e model I suggested above.
  • The MDS600 is a 3 Gb/s (3G) SAS enclosure. Keep that in mind. 6G disks will run at 3G. If you use current SATA disks (3G), they will run at 1.5 Gb/s.
  • You don't need HP disks... You do need HP drive caddies/carriers. Those can run up to $30/disk. Did you plan for that?
  • Which ZFS implementation are you using? My experience is centered around ZFS on NexentaStor. Device support may be different if you're using something like ZFS on Ubuntu...
  • I'm not sure if you're connecting this to an HP ProLiant server. You may want to keep an HP Smart Array controller handy in order to perform firmware updates on the enclosure.
Community
  • 1
ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • Cool, very helpful answers. I'm going to be using FreeBSD for ZFS. I think I've found some third party caddies that will work, closer to $10-15. Some clarifications: Why exactly would 3GB/s SATA drive run at 1.5 in this enclosure? (Not that I think that would be a bottleneck). From what I've read SATA drives work fine for a lot of people. We have some Dell PV1000s with SATA drives that have been no problem for ~5years. Is it something in particular with MDS600+ZFS+SATA? So I should I have an HP controller around for FW updates, even if I use the LSI board normally? – Cyclone Jul 21 '12 at 20:05
  • 2
    The speeds are lower because the enclosure uses an older implementation of the SAS protocol in its expander. SATA and ZFS and expanders isn't a good combination. See [this for a general explanation](http://serverfault.com/questions/331499/how-can-a-single-disk-in-a-hardware-sata-raid-10-array-bring-the-entire-array-to/331504#331504). When I refer to an HP controller, I'm talking about an [HP Smart Array RAID controller](http://h18004.www1.hp.com/products/servers/proliantstorage/arraycontrollers/index.html) (not SAS HBA). – ewwhite Jul 21 '12 at 20:22
  • 1
    +1 Agree with everything in this answer. If you're going to go "cheap", this is the way to do it. – Chris S Jul 21 '12 at 20:47
  • I think I'll heed all the advice except the SATA bit, as that no longer makes any of this a 'cheap' solution. The cheapest 3TB SAS drive on newegg is ~$400, the WD Caviar Red, which has TLER, is ~$200. At that difference in cost I can afford to have a duplicate for offsite backup... – Cyclone Jul 21 '12 at 21:00
  • Depends on your budget. I probably wouldn't use 3TB disks at this point. The sweet spot for large-capacity SAS drives is probably 2TB. Your MDS enclosure holds 70 disks. The [SAS versus SATA discussion](http://serverfault.com/questions/301567/sas-or-sata-for-3-tb-drives) has been had here, though. Things will get wonky under load with SATA disks and that enclosure setup. – ewwhite Jul 21 '12 at 22:47
  • @ewwhite Cool, I'll think about that hard. Would this not be a problem for a different enclosure, or would you have the same recommendation for SATA drives in any large SAS expander box? Also, I'm a bit confused as to the IO modules. Each IO module has two SF8088 connectors. Am I right in assuming I need to connect both to access all 35 drives in the tray? What happens if only one is connected? I was planning on buying a SAS HBA with 4 SF 8088 ports, or as you suggested, 2 card with 2 ports each. – Cyclone Jul 23 '12 at 22:53
  • SATA is a no-go for most enterprise ZFS solutions. For the MDS600, see the [cabling guide for more detail](http://h10032.www1.hp.com/ctg/Manual/c02571448.pdf). The I/O modules are basically in-out for multipath or cascading. You only need one cable per module. – ewwhite Jul 23 '12 at 23:21
  • @ewwhite Thanks. From looking at that PDF it looks like the second port on each module isn't even supported in a direct attached configuration. I'm guessing the second port is for more specialized higher bandwidth stuff then, like a SAS switch? If I were to just run this thing as a JBOD for a couple SATA drives would you foresee the same problems? Part of the reason for getting this is we already have 20 or so SATA drives with archived data we want to bring back online. – Cyclone Jul 23 '12 at 23:36
  • @Cyclone Did you end up doing this? – ewwhite Dec 30 '13 at 01:38
  • 1
    @ewwhite I did get the MDS600 and for the main pools (2) I used SAS 3TB drives. I also have some 3-4TB SATA drives that are also in the box (but not part of the pools) and they haven't seem to have caused any problems. – Cyclone Jan 07 '14 at 18:29
  • @Cyclone Thank you. Did you do any dual-pathing/multipath to the enclosure, or are you just using single connections? – ewwhite Jan 07 '14 at 18:50
  • @ewwhite single connections. the other IO modules are currently just sitting there as backups if I need to switch to them. – Cyclone Jan 07 '14 at 18:57