1

Having a RAID controller that has eight internal SATA3 lanes, you can get 6 Gb/s on all eight drives. What if I connect a 24 port SAS expander to an eight port RAID controller, do I still get max throughput of 8 x 6 Gb/s, or am I able to get 24 x 6 Gb/s, assuming the expander is rated for 6 Gb/s on all ports?

Of course the PCIe bandwidth is going to limit it, as well as the RAID controller but is this right theoritically speaking? PCIe 2.0 x8 has bandwidth of 4000 MB/s and PCIe 3.0 x8 has 7880 MB/s.

As an example, I was thinking of buying LSI MegaRAID 9271-8i for my home server. It has eight internal SATA 6 Gb/s lanes. With that one I am able to connect eight hard drives and they can work on their limits in terms of transfer rates because there is one 6 Gb/s lane for each drive available. But in the future the storage capacity might be too low. I thought I could just add a SAS expander, like Intel RES2SV240. It is a 24 port expander rated for 6 Gb/s per port. So do I get the full potential out of the expander, to have 6 Gb/s connection for all the possible 24 drives? If so, could I buy the 9271-4i (has only four internal SATA ports) and the Intel SAS expander to be able to connect up to 24 hard drives and have them work at their full speed?

MikkoP
  • 167
  • 1
  • 2
  • 11

2 Answers2

2

Yes and no...

Think about this: Your disks will not perform at 6Gbps (unless they're SSDs). So some level of oversubscription is okay when you go to using a SAS expander.

A more common scenario is the use of an external JBOD storage enclosure. Those usually have 1 or 2 x 4-lane SAS connectors linking them to the main server. Let's assume 4 x 6Gbps, so 24Gbps total bandwidth. Things are definitely oversubscribed there, as you may have 24 disks linked at 6Gbps... but recall that most disks won't be able to achieve more than 1.5 or 2Gbps in practice, so that level of oversubscription is okay.

Remember, 6Gbps is just a link speed. You will not be able to achieve that through an expander to all connected disks, because the expander has upstream connections to a RAID controller. The RAID controller is the limiting factor here.

See:
Do SAS expanders work transparently with SAS controllers?
How exactly does a SAS SFF-8087 breakout cable work? + RAID/connection questions

Community
  • 1
ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • So you are saying that adding a SAS expander won't increase the bandwidth beyond the bandwidth of the controller's internal SATA lanes? So, with a 4 lane SATA 6 Gb/s controller the maximum bandwidth is 24 Gb/s? I am going to use Western Digital Red drives in my RAID array. I read some benchmarks and they seem to go up to 1160 Mb/s reads and writes. So I could run up to 20 drives at their max speed before the bandwidth limit (24000 / 1160 is approximately 20,7). And because eight lines of PCIe 2.0 is 32000 Mb/s, it wont limit it either, meaning I shouldn't have a bottleneck with 20 drives. – MikkoP Aug 11 '14 at 15:20
  • Yes. Think of it as a water pipe which you fork into multiple, equally big pipes. If only one pipe is in use that it will get the full bandwidth back to the source (the RAID card). If you are using multiple pipes then they will have to share the bandwidth of the original pipe. – Hennes Aug 11 '14 at 15:22
  • Okay, thanks! So, to connect the RES2SV240 expander to the 9266-4i controller I need to use one of the cables from the expander to connect it to the controller? Meaning I have 23 ports free to connect to the hard drives? It has six 4-lane SAS ports. – MikkoP Aug 11 '14 at 15:28
  • @MikkoP Your Western Digital Red drives are not capable of those speeds. They'll likely max out at 200Mb/s. – ewwhite Aug 11 '14 at 15:28
  • According to multiple benchmarks I found online, they seem to go that far. Notice I was speaking of megabits, not bytes. That's 145 MB/s. To list a few: http://goo.gl/XTfoVv http://goo.gl/TNeRTc http://goo.gl/d1O8BW http://goo.gl/CZS77k – MikkoP Aug 11 '14 at 15:35
  • Hey, how would that expander connect to the controller? Somewhere I read that I would use one lane of the connector, some say I could plug in the whole 4-lane SAS cable between the controller and the expander. I think I should be connecting all four or I wouldn't get the full 24 Gb/s of the 9266-4i. – MikkoP Aug 12 '14 at 14:34
1

Real world experience (and a lot of money)... I built two machines each with (16) 512GB Samsung 850 Pros on Adaptec 81605Zs. Direct port connections with no expander. I got 6.2GB/s when the drives were fresh with WINSAT. Dropped to 5.8GB/s after a couple of days. Main problem is making sure motherboard slot was not shared with anything else. My personal machine has (8) SSDs via a LSI 9265-8i and the RES2SV240 expander mentioned above - 3.2GB/s. The other port on the apdater handles another expander with (13) drives - mechanical - nothing to write about. When the (8) SSD were split between a single port and sharing the second expander - performance was 2.7GB/s. Once again, bad or shared slot placement in a Rampage IV Extreme can drop these to 1.4GB/s. All RAID 0.

Guest
  • 11
  • 1
  • An addendum to Googlers: Most RQAID cards do not support TRIM on SSD's when in RAID mode, and Many do not support it even in HBA / pass-through mode. Unless you can be sure TRIM is being used, that's going to be the source of slowdown after a few days. – 1n5aN1aC Mar 19 '18 at 22:58