8

I recently purchased a Dell R730xd that came with an H730 RAID card. I would like to bring an existing 6 disk ZFS pool to this server as well as create an additional 6 disk hardware RAID.

Is it possible to configure an H730 RAID card to operate in both HBA and RAID mode simultaneously? And then only manage 6 of the disks in a hardware RAID and allow the OS to manage the remaining 6 disks in a software RAID?

If not, can a Dell R730xd use both an H730 RAID card and an LSI 9207-8i HBA card simultaneously? Would you simply plug one SAS cable from each card into each part of the Backplane? Will I need to purchase any new cables to connect the LSI 9207-8i to the backplane in my R730xd?

Is there a better card to use for ZFS in an R730xd using spinning disks than the LSI 9207-8i?

TJ Zimmerman
  • 241
  • 5
  • 17
  • Can you describe your use case in more detail? – ewwhite Apr 19 '20 at 21:01
  • What exactly do you want me to elaborate further on? – TJ Zimmerman Apr 19 '20 at 21:05
  • i solved that on a other kind of server with creating 6x single raids and 1x with raid 10 with 8 drives as requested by customer so it might be possible but as correctly asked by others for whatever reason you want this? – djdomi Apr 19 '20 at 21:07
  • Unfortunately that will not work for me since I am bringing an existing ZFS array to the server. Not creating a new one from empty RAID 0 disks. – TJ Zimmerman Apr 19 '20 at 21:08
  • Also I have read anecdotally that there are some pretty big issues around using ZFS with Dell PERC cards in HBA. If you have experience with this maybe you could answer another related question I just posted separately? https://serverfault.com/q/1013096/423284 – TJ Zimmerman Apr 19 '20 at 21:14

3 Answers3

6

Regarding your question about splitting the backplane across controllers, yes, it's possible. I've done this with 8-bay enclosures where two 4-lane SAS SFF-8087 connectors were present. 4 disks to one controller, 4 disks to the other. I don't know the composition or expander situation on the Dell server you're describing, though.

The only controllers that would allow this are those that have Mixed/Hybrid mode ports. HPE Smart Array controllers on Gen10 systems are a good example. They allow hardware RAID or HBA mode on a per-port/per drive basis.

I also have this running on an Avago MR9363-4i. /dev/sdy at the bottom is a RAID1 Hardware RAID1 for the OS. The remaining disks are split across two normal ZFS pools.

[0:0:4:0]    enclosu LSI      SAS3x40          0601  -
[0:0:5:0]    disk    SEAGATE  ST1800MM0128     E002  /dev/sda
[0:0:6:0]    disk    SEAGATE  ST1800MM0128     E002  /dev/sdb
[0:0:7:0]    disk    SEAGATE  ST1800MM0128     E002  /dev/sdc
[0:0:8:0]    disk    SEAGATE  ST1800MM0128     E002  /dev/sdd
[0:0:9:0]    disk    SEAGATE  ST1800MM0128     E002  /dev/sde
[0:0:10:0]   disk    SEAGATE  ST1800MM0128     E002  /dev/sdf
[0:0:11:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdg
[0:0:12:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdh
[0:0:13:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdi
[0:0:14:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdj
[0:0:15:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdk
[0:0:16:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdl
[0:0:17:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdm
[0:0:18:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdn
[0:0:19:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdo
[0:0:20:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdp
[0:0:21:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdq
[0:0:23:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdr
[0:0:24:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sds
[0:0:25:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdt
[0:0:26:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdu
[0:0:27:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdv
[0:0:28:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdw
[0:0:29:0]   disk    ATA      SAMSUNG MZ7LM1T9 204Q  /dev/sdx
[0:2:0:0]    disk    AVAGO    MR9363-4i        4.68  /dev/sdy

enter image description here

ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • @shodanshok implied in his answer that the H730 card is capable of operating in RAID mode and assigning specific disks to Non-RAID mode which are then managed by the OS. There are some drawbacks to this approach though. Notably that SMART data may not be available for the disks in the OS and ZFS may not like not having physical access to the disks. Would you recommend operating my storage in this dual RAID/Non-RAID configuration with the H730 or buying a new Controller/Backplane with support for dual RAID/HBA modes? – TJ Zimmerman Apr 19 '20 at 21:49
  • @ewwhite I am curious, does the HP controller expose SMART data for disks in HBA mixed mode? What about the Avago controller? Thanks. – shodanshok Apr 20 '20 at 07:40
  • @shodanshok On the Avago, S.M.A.R.T. seems to be accessible. `smartctl -i` returns info for the disks in the HBA mode. For the proper HPE Smart Array controllers, you can see `hpssacli` output as well as S.M.A.R.T. info. – ewwhite Apr 20 '20 at 13:04
  • It seems as if sticking with the H730 and using RAID Mode / Non-RAID Configuration is probably the best bet at the moment. – TJ Zimmerman Apr 20 '20 at 15:57
  • @ewwhite just to be sure: are SMART data accessible via `smartctl` on *both* the Avago controller and the HBA-mode connected to the HP controller? Or do you need to use `hpssacli` to get SMART data from the HBA-mode disks attached to the HP RAID? Thanks. – shodanshok Apr 20 '20 at 16:28
  • Another person has just told me that they personally have used an R730xd with multiple storage controllers and the same Backplane. They told me that it is a supported configuration for the R730xd but NOT the R730. – TJ Zimmerman Apr 20 '20 at 17:49
  • 1
    @shodanshok `hpssacli` doesn't show SMART info. But any drive on a mixed-mode controller (or HPE HBA mode) will be passed to the OS with SMART attributes and all. See the image I uploaded. – ewwhite Apr 20 '20 at 19:16
4

Replying to each questions:

  1. while you can install another PCI-E RAID card, the backplane can be connected to a single card. So you can't concurrently use the native RAID card and the additional one to access the disks on the same internal backplane [edit: @ewwhite suggested splitting disks between controller should be possible, so maybe I am wrong here. However the only officially supported dual-controller setup on the R730xd requires the use of flex-zoning backplane option, so I am not sure if/how two controllers can be put into operation in that specific case];

  2. probably not, but see #1

  3. PERC H730 supports two operating modes: RAID mode and HBA mode

    • the standard RAID mode, which supports RAID and non-RAID disks. In RAID mode, the controller expect the disks to be part of a RAID array and will not expose the raw disk to the operating system. In non-RAID (or passthrough) mode, the disks declared as non-RAID are exposed to the operating system (see note #1). In this mode, SMART reporting is firmware dependent and, if not working, you can try using the dedicated megaraid driver - ie: issuing something as smartctl -a -d megaraid,0 /dev/sda;

    • in HBA mode, the card works as a normal LSI-based SAS controller. This means any RAID capabiilty is disabled and disks are directly managed by the operating system. In such a mode SMART monitoring by the card is disabled and SMART data should be directly accessible (and monitored) by the operationg system. Additionally, you lose the card LED management and you may lose the added benefit offered by the controller writeback cache (I don't have direct confirmation), meaning you get much lower performance for random writes. However, please note that in ZFS you should use a SLOG to get high randon write performance rather than relying on the controller cache anyway.

In the end, as PERC 9/10 series cards are quite flexible you have no reason to use an additional RAID card.

Note #1, from the card manual:

By default, all the disks are in RAID capable unconfigured state. The user can also convert the RAID capable disks to non-RAID disks using either the BIOS configuration utility or the UEFI/HII RAID configuration utility

shodanshok
  • 44,038
  • 6
  • 98
  • 162
  • Thank you for your response. I didn't know the Backplane could only work with a single card. After cracking open the case and taking a look it makes more sense. Another distinction you made that helped me understand was clarifying the difference between HBA and Non-RAID modes. – TJ Zimmerman Apr 19 '20 at 21:34
  • Am I correct in asserting from you answer that I can configure the H730 to be in RAID mode. And then configure 6 disks to be in Non-RAID RAID Mode so that they are managed by the OS. While the H730 manages thee other 6 disks in a hardware RAID simultaneously? Are there performance drawbacks to overloading like this? Would you recommend doing the above, using HBA mode and expanding my ZFS array to 12 disks, or using HBA mode and managing two separate ZFS pools of 6 disks? – TJ Zimmerman Apr 19 '20 at 21:35
  • Also, how would you respond to this answer suggesting there are misunderstandings about the Pass-Through and that using it is still a dangerous gamble with ZFS? https://serverfault.com/a/735748/423284 – TJ Zimmerman Apr 20 '20 at 01:09
  • @TJZimmerman yes, in RAID mode you can mix a RAID array with non-RAID disks. However in this mode the controller write back cache remains active and a controller failure can potentially destroy the zpool due to the reordered writes done via the cache. This is an unlikely event though. In HBA mode the card should totally disable any write back cache, becoming no different then a "normal" LSI expansion card. – shodanshok Apr 20 '20 at 07:20
  • Hm, The hardware cache is a very good point. It's impossible to disable this in RAID mode? Even if it would worsen performance to not have one? Do you have any advice or a primary source for how I could gain a better mental model of why using Cache with ZFS can cause corruption? – TJ Zimmerman Apr 20 '20 at 16:03
  • 1
    @TJZimmerman if I remember correctly, the writeback cache can be put in writethrough mode (ie: read-only mode) to avoid caching any write. The problem with an on-board writeback cache is that if the controller fails with the to-be-written data in cache (ie: losing a "dirty cache"), the (unlikely) possibility exists that the entire zpool can be trashed. I would not mind the cache (you *need* a proper backup anyway) but, if you want to totally disable any caching, you are better off using the card in HBA mode and using linux MDRAID to manage the non-ZFS disks. – shodanshok Apr 20 '20 at 16:33
  • Another person has just told me that they personally have used an R730xd with multiple storage controllers and the same Backplane. They told me that it is a supported configuration for the R730xd but NOT the R730. – TJ Zimmerman Apr 20 '20 at 17:50
  • Actually, I see now you added an edit linking to the Dell Support thread asking about a second controller. And a comment in that thread linked to a PDF with demonstrated the various configurations. I guess in the event you're using two controllers the second controller can't manage any of the disks in the backplane. Just the two 2.5" disks present in the Flexbay? If so, it seems misconstrued to imply that the Backplane can be used with two controllers. Rather, the R730xd can be. Where each controller targets an entirely separate storage configuration. – TJ Zimmerman Apr 20 '20 at 17:55
0

I got same thoughs about mixing Perc hardware RAID and ZFS, I was thinking about splitting backplane or doing other nasty procedures. From one side I needed hardware raid volumes, from other side I needed to import existing ZFS pool.

I have readed that H730 can be set up for either HBA or RAID mode. However that's not true, at least for latest firmware. I have tested on R730, H730p mini mono and ESXI 7- everything works perfect. No need to create virtual disk. Passthrough works pretty cool!

Also, on ESXi 7 no more RDM passthrough needed!

Step 1. Upgrade PERC firmare to latest 25.5.7.005. Set it to RAID mode.

Step 2. Create Virtual Disk for needed hardware raid layout.

Step 3. Mark needed ZFS disks as Non-Raid ones.

Step 4. Im not sure if You can disable write cache for Non-Raid disks via BIOS, but I have suceeed disabling non-raid cache through OpenManage (of course cache still enabled for HW RAID):

H730p openmanage settings

Step 5. Non-raid disks and RAID disks will be normally visible on ESXi. Of course no problems with hotplugging.

Detected devices by vSphere

Step 6. Create/import FreeNAS VM. Edit VM settings, click Add Hard Disk -> New Raw Disk. Non-VMFS raw devices will be shown, and You can add them one-by one as passthrough:

New one-click RDM passthrough

RDM passthrough

Remember to leave Disk Compatibility: Physical

RDM passthrough

Step 7. Boot FreeNas, all passthrough disks will be visible as directly attached, including disks serial numbers, APM managment, etc:

PERC h730p nonraid disks in FreeNAS

Step 8. Use shell and test smartctl:

Smart results

Disks are directly visible by FreeNas. Also, I recommend upgrading to TrueNas 12.0 - iSCSI speed and stability improvments are really big step forward. No more ctl_datamove aborted errors.

You can also hotplug any other disk, set it to nonraid via iDrac and then just on-live Edit VM options and add new RAW device, it will be detected and attached to FreeNAS immediatly without rebooting of course.

I have readed too much on forums that's not possible... not sure if new firmware added such capability, but actually I am more than happy. System is very flexible that way.

From one side iDrac is fully helath-monitoring created RAID Virtual disk (incl. patrol read and all), and from other side non-perc RAID disks are monitored via FreeNAS. Also, because thats 1:1 passthrough, of course disks temperatures are monitored too.

Small tip: Don't forget to disable "Delayed ACK" when using ESXi software iscsi client. Feed FreeNas with decent amount of RAM. After disabling Delayed Ack, and upgrading NAS to v12.0 (From 11.3 Freenas to TrueNAS) actual iSCSI backup times dropped from 11+ hours to 3.5 hrs.

It will look like this from iDrac perspective:

However, remember that Write Cache can't be disabled for Non-Raid disks via iDrac, there is no such setting in controller settings. So OpenManage (confirmed), or bios maybe.

iDrac perspective

user600606
  • 1