I'm planning to use ZFS on my system (HP ML370 G5, Smart Array P400, 8 SAS disk). I want ZFS to manage all disks individually, so it can utilize better scheduling (i.e. I want to use software RAID feature in ZFS).
The problem is, I can't find a way to disable RAID feature on the RAID controller. Right now, the controller aggregates all of the disks into one big RAID-5 volume. So ZFS can't see individual disk.
Is there any way to acomplish this setup?
It is not possible to disable the RAID functionality of HP Smart Array controllers older than the P420i
A common "solution" to this problem is to create single-disk RAID-0 volumes at the controller level. This is not a good solution and definitely is not equivalent to a JBOD arrangement. There's RAID metadata on the disks, and the failures will produce unexpected results.
In this case, be careful with the Smart Array P400 single-disk RAID 0 setup. If you have a hotplug event (disk failure/drive removal), ZFS won't recognize the new disk without a reboot. There's no true JBOD setting on these cards.
Using a dedicated SAS HBA is the right path.
It should also be noted, just because I just killed 7TB of data by following LapTop006's, well, lets call it "personal opinion", that a P400 Controller would expose unassigned disks as JBOD, that this is nothing but a guess, and it is false, at least for my P400. There may be other controllers behaving like LapTop006 said, the P400 does not, at least not with the original firmware (V2.75).
I learned this the hard way today when trying to bring over a 6 disc software RAID-5 from a machine with a faulty 6 channel SATA RAID controller. They had always been part of a software RAID, the RAID functionality of the ICH9 "Fake" RAID controller had never been used anyway.
The target machine didn't have enough SATA ports, so I thought, well, no problem, it is a SOFTWARE RAID anyway, why not attach the disks to a P400, the disks would - if the controller behaved like stated - appear as JBOD, and the OS would - like it had done many times before when I moved software RAIDs from one machine to another - recognize the RAID.
In my case, however, the P400 did recognize the disks as new and - without seeking my confirmation - it did auto-create a RAID-5 array at the controller level as soon as I powered up the computer. Bye-Bye software RAID.
I brought the disks back to the original machine, but the RAID had already been corrupted, the OS saw 6 empty disks now.
Bye, 7TB of data.
Damage already done, I played with the disks a little bit. Back at the P400 equipped machine, I deleted the unwanted RAID-5, the disks didn't appear at the OS level. I had to create 6 RAID-0 disks, and they appeared - all empty, however.
Conclusions:
Here is my personal experience with a HP Smart Array P410 RAID Controller:
I have a P410 controller bought off ebay. It came with 512MB of RAM, "Flash" backed, with "SuperCapacitor". I also found it came loaded with a "Feature License", which is called "Advanced Pack 01". This "Advanced Pack" license enables Raid 6 and 60, and other "Advanced Settings". Right now, I am not sure if this "Feature" is what get enabled when you add the RAM and Battery/SuperCapacitor to the card - I am thinking it is... please chime in, anybody, if you have some additional info.
I took a working 320GB SATA disk which I had loaded files onto, and plugged it into a port on the HP P410 controller. I then configured it with the HP "Smart Storage Administrator" (A Windows Utility) as a "Single Disk Raid 0" as others have commented.
So, after doing this, Windows immediately pops up the new drive. Lo and behold, my data is all there, nice!
You will notice, if you check the "Physical Device" info in Smart Storage Administrator there is an item in the list which reads: "Exposed To OS: No". Seems the P410 knows we want JBOD but refuses to give it to us! That's rather frustrating.....
Now, here's what I found. The Smart Array controller overwrites some data at the beginning of the disk and... if you try to take this drive out of the Smart Array and go back into a regular motherboard port, Windows Disk Management shows it as "Unallocated".
The solution to this..... Use the excellent "TestDisk" utility, you can select the physical disk, tell it you had an Intel partition, and after a quick search, it should find your missing information. You can use the "P: List Files" to se ethat those files are indeed there... Then you tell TestDisk to "Write" the info back to the disk, reboot the computer, and voila! Back to happy-land. So, there seems to be a way to get the job done here, you just have to be aware that the Smart Array (and many other controllers from what I've read) will over-write a bit of data on any disk you decide to set up as a RAID array of any sort. However, if you're careful and you don't initialize or erase, you can often get your drive back to working well by using a nice little tool such as "TestDisk".
Note: I have also had good success running a Raid 5 on the P410, Originally, I created it on a P400, then decided to swap it out for a P410. No going into the configuration menu or other intervention needed, the P410 recognized the Raid 5 array and immediately made it available, upon boot-up! I though that was quite nice. So, some more evidence that the HP Raid cards store all the array info on the drives themselves, not the card. Nice!
One other thing, I found no way to enable a true "Pass-Through" on this Smart Array. Leaving disks "Unconfigured" or otherwise not a member of an array prevented them from being shown to the O/S. The Smart Array will only play nice if you let it administer the raw disks. That having been said.... what I learned tells me the level of "interference" on the part of the Smart Array is minimal... to me it seems like not a big deal.
BTW I am screwing around with all this on a SuperMicro 745 Server/Workstation. The 743, 745, 747, and 748 are amazing chassis/systems! You can pick up a 743 or 745 fairly cheaply on ebay. And if you love MEGA storage, you can pick up an 846 (24 bays) cheaply on ebay (I got one).
Jeff
The other option is to create each disk as a single-disk RAID-0 array (yes, it should let you do this). Then ZFS will see all the indidual disks.
The 'technical' name for the option you're looking for is called JBOD - Just a Bunch of Disks.
I simply moved the internal SCSI cable from the smart array card to the on board SCSI Controller located on the system board. This gave me access to the raw disks.
To anyone viewing this older thread and using HP smartarray kit - don't do it! - here's why:
ZFS is best used where the controller won't disconnect the disks if it sees a problem. Let ZFS manage that itself. Smartarray will disconnect the disk if it sees a read(or any other issue) - this is not ideal for ZFS at all, and can cause you a lot of grief!
This means, avoid smartarray stuff P420, P8*, etc at all costs for ZFS until HP allow direct passthrough without the smartarray disconnecting the disk even if there is an issue. As what will happen when a disk starts to fail (but isn't completely borked) is that the smartarray controller will simply disconnect the whole disk. You might think this is the correct thing to do, however if the disk is still functional (on other parts of its surface) then it's still useful to have, especially when re-silvering to a new disk. ZFS can still use a failing disk. It has CRC/etc and can manage this for you. The smart array simply disconnects the whole disk and if you're not on something like raidz3 (or don't scrub regularly) then you could be in for a bad experience when resilvering and you find another disk has an issue in a different area (such is the life using HP SAS)
tl;dr HP Smartarray should not be used for ZFS until it can do passthrough without disconnecting disks that are having problems. Use a different HBA card that won't interfere if a disk has a problem (let ZFS manage it, it's very good at this)
If you simply delete the volume in the RAID BIOS that exposes the disks directly in some versions of the SmartArray controllers. We always use the controller even with things like ZFS.
