9

I'm new to SAS expanders and I'm trying to get things right. I understand that SAS expanders, when used with SAS controllers which supports expanders, allows you to utilise the full capacity of supported drives on a controller.

However, say I've got a SAS controller with 4 x SSF-8087 ports, which by itself would support 16 disks. I could then, as far as I understand, instead connect each of these ports to an expander to increase disk capacity.

However, will the SAS controller "see" all disks as if they were connected directly to the controller and not through an expander. Could I look at controllers as switches in a network? Would I be able to create a RAID from a disk on expander 1 and expander 3?

UPDATE:

I don't really have a specific project where I need to use SAS expanders. I'm just curious about how they work, and was thinking about how it would work if I used one or more SAS expanders with a LSI MegaRAID controller (9260-16i) which is running in one of our servers. However, I haven't really put any thought into the distinction between "local" disks and disks in another enclosure. The LSI MegaRAID controller has 4 SFF-8087 ports, which makes it capable of accomodating 16 drives.

sbrattla
  • 1,456
  • 3
  • 26
  • 48

2 Answers2

11

First, stop and think about your realistic goals...

The short answer to your question is that SAS is a switching protocol. The drives connected via expanders are all accessible and can be addressed transparently by a single controller.

However, the rest of your questions are going to be a function of your particular controller, your storage architecture and the backplanes in the storage enclosure(s) you end up using.

It's not possible to give a specific answer without more information.

  • How many disks do you need?
  • What RAID controller or SAS HBA are you using?
  • Will these disks be internal or external?
  • What specific SAS expander card are you looking to use (it's not like there are many on the market)?
  • What server and enclosure hardware are you considering?
  • What is the application/reason for this setup? (ZFS? Storage Spaces?)

Let's use your example of a controller with 4 x SAS SFF-8087 ports. These are meant for communication inside of a chassis. Each SFF-8087 port is a 4-lane connector that can accommodate 4 drives. 16 drives total, in your case... Servers typically only have 4, 8, 12, 16, or 24/25 internal drive bay capacities.

  • At the 4 and 8-drive level, the internal server drive backplane will have a 1:1 lane-to-drive ratio and typically have one or two SFF-8087 ports. No oversubscription!

  • For 12-port servers, the expander is usually built into the backplane, and the input will be a single SFF-8087 port. Oversubscription!

  • For servers with 16 internal bays, you may see four separate SFF-8087 ports intended to be served by a single or multiple SAS/RAID controllers. But at 16-ports, you may have an expander on the backplane instead.

  • Almost every 24 or 25-bay server I've encountered has expanders built onto the drive backplane and accommodate one or two SFF-8087 ports. The exception is something like the HP ProLiant DL370 G6, which could accept three 8-bay drive cages, each requiring 2 SFF-8087 ports.

So when do people use PCIe card SAS expanders?

Note: HP makes a great PCIe SAS expander. There's an entire community built around its use for storage projects.

The most common case I see is a motherboard or PCIe controller with TWO SFF-8087 ports and a need to connect more than 8 drives split across multiple drive cages. I can't envision something like your scenario where you have 16 ports available on a controller and would want to multiply that via expanders to accommodate more internal drives. If you're working with a server with more than 16 disks, chances are that you'll be encountering a backplane expander somewhere. This would make the PCIe expander unnecessary.

This is all different when you look at external storage, though. The transport there would be a SAS SFF-8088 external cable. Most external drive shelves/JBODs have expanders built-in.

Example RAID configuration output from an HP Smart Array P410i controller connected to two drive cages inside of an HP ProLiant DL370 G6 server through an HP SAS expander card. Note the card is represented as an enclosure (Vendor ID HP, Model HP SAS EXP Card) and has its own SAS WWN.

Smart Array P410i in Slot 0 (Embedded)    (sn: 5001438006503640)

   array A (Solid State SATA, Unused Space: 0  MB)

      logicaldrive 1 (223.5 GB, RAID 0, OK)

      physicaldrive 5C:2:8 (port 5C:box 2:bay 8, Solid State SATA, 240.0 GB, OK)

   array B (SAS, Unused Space: 0  MB)

      logicaldrive 2 (68.3 GB, RAID 1, OK)

      physicaldrive 7C:2:1 (port 7C:box 2:bay 1, SAS, 72 GB, OK)
      physicaldrive 7C:2:2 (port 7C:box 2:bay 2, SAS, 72 GB, OK)

   array C (SATA, Unused Space: 2651025  MB)

      logicaldrive 3 (1000.0 GB, RAID 1+0, OK)
      logicaldrive 4 (500.0 GB, RAID 1+0, OK)

      physicaldrive 2C:1:1 (port 2C:box 1:bay 1, SATA, 1 TB, OK)
      physicaldrive 2C:1:2 (port 2C:box 1:bay 2, SATA, 1 TB, OK)
      physicaldrive 2C:1:3 (port 2C:box 1:bay 3, SATA, 1 TB, OK)
      physicaldrive 2C:1:4 (port 2C:box 1:bay 4, SATA, 1 TB, OK)
      physicaldrive 3C:1:5 (port 3C:box 1:bay 5, SATA, 1 TB, OK)
      physicaldrive 3C:1:6 (port 3C:box 1:bay 6, SATA, 1 TB, OK)

   Expander 250 (WWID: 50014380133153E6, Box: 1)
   Enclosure SEP (Vendor ID HP, Model HP SAS EXP Card) 248 (WWID: 50014380133153E5, Box: 2)
   SEP (Vendor ID PMCSIERA, Model  SRC 8x6G) 249 (WWID: 500143800650364F)

enter image description here

ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • Good answer. I was going to write something similar, but you beat me to it. Additional word of warning: Putting an expander behind another expander is usually a bad idea. Both from a performance and a stability perspective: There is a huge chance for subtle compatibility issues. – Tonny May 10 '13 at 09:11
  • 1
    @Tonny The point here is that an expander may be redundant to wholly unnecessary in the OP's situation. But without specifics... we don't know. – ewwhite May 10 '13 at 09:16
  • You are right. His intend is not clear from the question. And most server already have internal expanders. That's why I warned putting one after the other. From experience I know that most people don't realize that these things are not really plug&play. You need equipment that is matched to each other or things can go very wrong. – Tonny May 10 '13 at 09:25
  • @ewwhite thanks for your reply. I've updated my question to answer some of your questions. – sbrattla May 10 '13 at 10:17
  • @sbrattla For the controller you've listed, you have plenty of internal ports. What type of server is it? – ewwhite May 10 '13 at 10:20
  • 1
    @ewwhite : the server is a Norco RPC-3216. It has 16 bays, so I'm using all of them. Just trying to figure out what I could do to extend storage capacity if I needed to expand the array to more disks. Obviously, I can't add more disks. – sbrattla May 10 '13 at 11:02
  • @sbrattla Then your expansion would need to be external. Something like the [LSI 9280 controller](http://www.lsi.com/products/storagecomponents/Pages/MegaRAIDSAS9280-16i4e.aspx) would keep the internal ports but also allow you to expand into an external JBOD. But at that point, do you really need more than 16 disks in a single array? Are *larger* drives an option? – ewwhite May 10 '13 at 11:06
  • I guess larger drives could be an option, as there's probably also a limit on the max. number of drives in an array. But I've also got another server with a PERC H700 controller w/2 SFF-8087 connectors. I might get the need to increase the disks for that on too, and an expander might be helpful in that case I guess? – sbrattla May 10 '13 at 11:10
  • 1
    I actually have one of those HP SAS Expanders and can vouch for both it's price and capabilities. Works great in a Duct-tape and WD-40 rig. – Chris S Feb 10 '14 at 05:25
0

I can agree with the thing that a expander enclosure should not put behind another expander enclosure, from my experiance it looks like to work but hsving lot of very weird behaviours like seeing the drives and initialising them.. now having both expander enclosures connected seperately to the SAS raid card from main server. it seems to work lot better but still have some issues but that can be related to the drives themself. The expanders are not the same type but the first one i run for years without issue on that raidcontroller, now the second expander is different, so have to find out the troubles i have, it can be related to anything software or hardware. here the specs i use at the moment;

-Highpoint 2722 SAS raidcard with 2 external SAS 8088 ports - expander enclosure 1 with 36 port 24 bays with Chenbro expander(not on backplane) - expander enclosure 2 with 24 bay and Gooxi jbod expander backplane

The raidcard sees both expanders , the Chenbro as a 24 4U enclosure and the Gooxi one even with the hardware monitoring like fans and temps.

Till sofar i have some troubles installing drives on the Gooxi and that takes loads of time-)

Highflow
  • 1