17

I'm planning to use ZFS on my system (HP ML370 G5, Smart Array P400, 8 SAS disk). I want ZFS to manage all disks individually, so it can utilize better scheduling (i.e. I want to use software RAID feature in ZFS).

The problem is, I can't find a way to disable RAID feature on the RAID controller. Right now, the controller aggregates all of the disks into one big RAID-5 volume. So ZFS can't see individual disk.

Is there any way to acomplish this setup?

ewwhite
  • 194,921
  • 91
  • 434
  • 799
Arie K
  • 1,583
  • 5
  • 18
  • 27
  • 3
    Disable hardware RAID in favour of software RAID? That just doesn't make sense to me, especially as you have such a well proven controller. – John Gardeniers Jul 17 '09 at 04:49
  • 6
    Well, this is IMHO debatable. But I think ZFS has clear advantages over *pure* hardware RAID. – Arie K Jul 17 '09 at 06:35
  • 5
    @JohnGardeniers You obviously knew nothing of ZFS when you wrote that comment. If your lack of understanding has now been corrected, you may want to consider tidying up (deleting) your comment. – Mark Booth Mar 12 '14 at 11:02
  • @JohnGardeniers P400/P410i doesn't support device passthrough, i.e. TRIM/discard with RAID arrays will be unsupported, so, if you have SSD's, they might and die very soon in intensive applications. If application is not that intensive - don't use SSD's. – GioMac Jul 07 '16 at 19:20

7 Answers7

25

It is not possible to disable the RAID functionality of HP Smart Array controllers older than the P420i

A common "solution" to this problem is to create single-disk RAID-0 volumes at the controller level. This is not a good solution and definitely is not equivalent to a JBOD arrangement. There's RAID metadata on the disks, and the failures will produce unexpected results.

In this case, be careful with the Smart Array P400 single-disk RAID 0 setup. If you have a hotplug event (disk failure/drive removal), ZFS won't recognize the new disk without a reboot. There's no true JBOD setting on these cards.

Using a dedicated SAS HBA is the right path.

ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • What is the difference between JBOD and creating RAID0 volumes for each disk? Are there performance penalties or certain risks? – Tommiie Jan 22 '19 at 15:37
  • Less perfomance. More of a management thing. Failure of a disk == failure of the LUN or logical drive... so you may have to reconfigure or add a new logical drive to handle a disk replacement. – ewwhite Jan 22 '19 at 19:10
11

It should also be noted, just because I just killed 7TB of data by following LapTop006's, well, lets call it "personal opinion", that a P400 Controller would expose unassigned disks as JBOD, that this is nothing but a guess, and it is false, at least for my P400. There may be other controllers behaving like LapTop006 said, the P400 does not, at least not with the original firmware (V2.75).

I learned this the hard way today when trying to bring over a 6 disc software RAID-5 from a machine with a faulty 6 channel SATA RAID controller. They had always been part of a software RAID, the RAID functionality of the ICH9 "Fake" RAID controller had never been used anyway.

The target machine didn't have enough SATA ports, so I thought, well, no problem, it is a SOFTWARE RAID anyway, why not attach the disks to a P400, the disks would - if the controller behaved like stated - appear as JBOD, and the OS would - like it had done many times before when I moved software RAIDs from one machine to another - recognize the RAID.

In my case, however, the P400 did recognize the disks as new and - without seeking my confirmation - it did auto-create a RAID-5 array at the controller level as soon as I powered up the computer. Bye-Bye software RAID.

I brought the disks back to the original machine, but the RAID had already been corrupted, the OS saw 6 empty disks now.

Bye, 7TB of data.

Damage already done, I played with the disks a little bit. Back at the P400 equipped machine, I deleted the unwanted RAID-5, the disks didn't appear at the OS level. I had to create 6 RAID-0 disks, and they appeared - all empty, however.

Conclusions:

  • The P400 does not pass unassigned disks to the OS.
  • You need to create RAID-0 configs to get the disks through to the OS.
  • Saving the RAID-0 config (or any other P400 config) will empty the disk(s).
  • The P400 auto-config may have killed anything on the disks anyway, by creating a RAID-5 without asking permission.
mc0e
  • 5,786
  • 17
  • 31
Nimral
  • 213
  • 2
  • 6
4

Here is my personal experience with a HP Smart Array P410 RAID Controller:

I have a P410 controller bought off ebay. It came with 512MB of RAM, "Flash" backed, with "SuperCapacitor". I also found it came loaded with a "Feature License", which is called "Advanced Pack 01". This "Advanced Pack" license enables Raid 6 and 60, and other "Advanced Settings". Right now, I am not sure if this "Feature" is what get enabled when you add the RAM and Battery/SuperCapacitor to the card - I am thinking it is... please chime in, anybody, if you have some additional info.

I took a working 320GB SATA disk which I had loaded files onto, and plugged it into a port on the HP P410 controller. I then configured it with the HP "Smart Storage Administrator" (A Windows Utility) as a "Single Disk Raid 0" as others have commented.

So, after doing this, Windows immediately pops up the new drive. Lo and behold, my data is all there, nice!

You will notice, if you check the "Physical Device" info in Smart Storage Administrator there is an item in the list which reads: "Exposed To OS: No". Seems the P410 knows we want JBOD but refuses to give it to us! That's rather frustrating.....

Now, here's what I found. The Smart Array controller overwrites some data at the beginning of the disk and... if you try to take this drive out of the Smart Array and go back into a regular motherboard port, Windows Disk Management shows it as "Unallocated".

The solution to this..... Use the excellent "TestDisk" utility, you can select the physical disk, tell it you had an Intel partition, and after a quick search, it should find your missing information. You can use the "P: List Files" to se ethat those files are indeed there... Then you tell TestDisk to "Write" the info back to the disk, reboot the computer, and voila! Back to happy-land. So, there seems to be a way to get the job done here, you just have to be aware that the Smart Array (and many other controllers from what I've read) will over-write a bit of data on any disk you decide to set up as a RAID array of any sort. However, if you're careful and you don't initialize or erase, you can often get your drive back to working well by using a nice little tool such as "TestDisk".

Note: I have also had good success running a Raid 5 on the P410, Originally, I created it on a P400, then decided to swap it out for a P410. No going into the configuration menu or other intervention needed, the P410 recognized the Raid 5 array and immediately made it available, upon boot-up! I though that was quite nice. So, some more evidence that the HP Raid cards store all the array info on the drives themselves, not the card. Nice!

One other thing, I found no way to enable a true "Pass-Through" on this Smart Array. Leaving disks "Unconfigured" or otherwise not a member of an array prevented them from being shown to the O/S. The Smart Array will only play nice if you let it administer the raw disks. That having been said.... what I learned tells me the level of "interference" on the part of the Smart Array is minimal... to me it seems like not a big deal.

BTW I am screwing around with all this on a SuperMicro 745 Server/Workstation. The 743, 745, 747, and 748 are amazing chassis/systems! You can pick up a 743 or 745 fairly cheaply on ebay. And if you love MEGA storage, you can pick up an 846 (24 bays) cheaply on ebay (I got one).

Jeff

Atari Jeff
  • 41
  • 3
1

The other option is to create each disk as a single-disk RAID-0 array (yes, it should let you do this). Then ZFS will see all the indidual disks.

The 'technical' name for the option you're looking for is called JBOD - Just a Bunch of Disks.

Mark Henderson
  • 68,316
  • 31
  • 175
  • 255
1

I simply moved the internal SCSI cable from the smart array card to the on board SCSI Controller located on the system board. This gave me access to the raw disks.

Jarrett
  • 31
  • 1
  • 3
    That may apply to older pre-G5 ProLiant servers. It's not valid on modern systems because they no longer have onboard SCSI. – ewwhite Mar 06 '12 at 19:01
1

To anyone viewing this older thread and using HP smartarray kit - don't do it! - here's why:

ZFS is best used where the controller won't disconnect the disks if it sees a problem. Let ZFS manage that itself. Smartarray will disconnect the disk if it sees a read(or any other issue) - this is not ideal for ZFS at all, and can cause you a lot of grief!

This means, avoid smartarray stuff P420, P8*, etc at all costs for ZFS until HP allow direct passthrough without the smartarray disconnecting the disk even if there is an issue. As what will happen when a disk starts to fail (but isn't completely borked) is that the smartarray controller will simply disconnect the whole disk. You might think this is the correct thing to do, however if the disk is still functional (on other parts of its surface) then it's still useful to have, especially when re-silvering to a new disk. ZFS can still use a failing disk. It has CRC/etc and can manage this for you. The smart array simply disconnects the whole disk and if you're not on something like raidz3 (or don't scrub regularly) then you could be in for a bad experience when resilvering and you find another disk has an issue in a different area (such is the life using HP SAS)

tl;dr HP Smartarray should not be used for ZFS until it can do passthrough without disconnecting disks that are having problems. Use a different HBA card that won't interfere if a disk has a problem (let ZFS manage it, it's very good at this)

  • 1
    This is incorrect. The Gen8 controllers and newer (P420i, for example) have HBA mode and allow raw disk passthrough to the operating system. They are perfectly fine to use for the purpose of ZFS and any other software-defined-storage solution (Storage Spaces, VMware vSAN, etc.) – ewwhite Mar 31 '19 at 18:38
-1

If you simply delete the volume in the RAID BIOS that exposes the disks directly in some versions of the SmartArray controllers. We always use the controller even with things like ZFS.

LapTop006
  • 6,466
  • 19
  • 26
  • This is the right answer - just delete the array in the ACU, that way the BIOS just sees the disk. That said are you SURE you don't want it managed by the P400? they're a damn good card. – Chopper3 Jun 21 '09 at 14:46
  • Aha, so this bring back ol' software vs hardware RAID war :) Should I combine both setup: ZFS' pool on top of single RAID-5 (or even RAID 1+0) volume exposed by P400? – Arie K Jun 21 '09 at 16:18
  • 12
    There are advantages to letting ZFS have the raw disks over a hardware raid controller. For one if you have a disk go bad ZFS will only need to rebuild the actual data rather than the entire block device speeding up rebuild and reducing the possibility of running into another problem while rebuilding. By giving ZFS the entire raw disk ZFS also manages the drive cache which improves efficiency. Third ZFS will properly detect and correct IO errors due to its exhaustive data checksumming. Because of this I wouldn't put a hardware raid inbetween ZFS and the Disks. – 3dinfluence Jun 22 '09 at 03:41
  • 2
    Adding my thoughts here, since this came up in my search results... In my case, I'm using an HP Smart Array P212. It looks like this controller can not present the raw disks (no jbod mode). If you delete all raid volumes, it just doesn't present anything to the OS. I want to use linux software raid (md raid) because: 1) no raid 6 support on this controller (may be able to purchase a license to add this) which I want since I'm attaching 24 drives, and 2) I'm using desktop drives, which I hear md raid handles desktop drives better than enterprisey controllers that expect enterprise drives. – James Jan 03 '12 at 14:32
  • 4
    This is wrong. There's no way to present RAW drives from a Smart Array controller. – ewwhite Nov 04 '12 at 07:45
  • 2
    Not only is this answer wrong, it's harmful. Scroll down and notice some poor guy who lost data due to this bad answer. – Mikey T.K. Oct 14 '16 at 13:53