6

I'm trying to get my head around how a multi lane x4 SAS connector works in a DAS system such as the Dell MD1000. I understand that each lane is 3Gbps, and that multiple hard drives can share a lane. What I'm having trouble finding information on is how are drives assigned to a lane, and how does that change when you daisy chain two additional md1000's to another md1000?

That's 15 disks per shelf and a total of 45 disks in a three shelf configuration. This would be in a single path configuration BTW, meaning one x4 SAS cable.

Edit 1:

All, first thanks for all the help, but I think you're all heading down a slightly different path than what I'm asking. I get the whole thorughput saturation, that wasn't my question. I know what the theoretical max is simply based on the fact that the server has a single 12g conection per array of shelves. Meaning Server >>>>12g>>>>MD10001>>>>>MD10002>>>>>MD10003 is going to be 12g as it all depends on the servers single connection and each md1000 is daisy chained with a 12Gbps link.

My question is simply if there are 15 drives per shelf and 3 shelves, how do i know which drives go on which lane of a given 4x connector? While it's likely not to matter in the end, i was merely curious.

Also FYI, the enclosure is SAS, but the drives are SATA.

Eric C. Singer
  • 2,319
  • 15
  • 17
  • For whatever it's worth, this seems like an awful lot of disks for a single path configuration. How many heads would roll if there was a SAS controller failure or a loss of connectivity? – Skyhawk May 08 '12 at 18:05
  • 45 disks on a single path.. do you care about accessing the data? – pauska May 08 '12 at 18:06
  • First, I just joined the company, this wasn't my design. Second, this is for disk to disk to tape. While it would suck if an entire array of shelves went offline, it's not the end of the world. . – Eric C. Singer May 08 '12 at 18:13
  • BTW, this is a Dell limitation TMK. Dual paths would only provide redudancy, not throughput. – Eric C. Singer May 08 '12 at 18:15
  • For what it's worth, Dell/Compellent (for their Compellent SANs) recommends a maximum of 48 drives on a 3Gb SAS chain, max of 96 drives on a 6Gb SAS chain. So this design might not be too bad. I don't have any experience regarding MD1000 SAS, specifically. – Jeremy May 08 '12 at 18:16
  • @EricC.Singer hey, relax, I wasn't criticizing you. Just wanted to make sure that this isn't any data that you can't live without. Single path to a B2D backend is fine :) – pauska May 08 '12 at 18:30
  • it was a little harsh, i removed the last part, apperently not quick enough though. – Eric C. Singer May 08 '12 at 18:42
  • 3
    @Eric C. Singer Don't forget you can attach each shelf independantly to a card rather than daisy chain them, which would increase throughput. Striping at the server level if neccesary has very little overhead. – JamesRyan May 08 '12 at 18:42

3 Answers3

5

You'll be massively oversubscribed. A single 4-lane SAS link at 3Gbps == 12Gbps total throughput. There's an expander on each MD1000 enclosure, so your 45 (SAS or SATA) disks will easily saturate that link. That's a theoretical max throughput of 1.5 Gigabytes/second over that connection - 12 Gigabits/sec == 1.5 Gigabytes/sec.

ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • I am confused where the 1.5Gb/sec throughput number comes from - is that a limit of the expander? – Jeremy May 08 '12 at 18:17
  • 6
    12 Gigabits/sec == 1.5 Gigabytes/sec. – ewwhite May 08 '12 at 18:18
  • Ok, so help me through this a little more. I get that 12Gbps is the theoretical "aggregate" maximum. Is that measurment reporting a duplexed speed? Meaning if I'm reading data, and lucky enough that its all spread evenly across all lanes, would i then read at 12Gbps or 6Gbps? Secondly, do you know which disks go on which lane? Is that documented anywhere? – Eric C. Singer May 08 '12 at 18:19
  • Sorry, I mis-read your statement (you converted to giga*bytes*). Anyway, I'm still curious. Is that 12Gbps each way (meaning 24Gbps) or one way? – Eric C. Singer May 08 '12 at 18:22
  • I think it's clear that you will be massively oversubscribed from a SAS perspective, but what is the connection between the server and the MD1000's? Faster than 12Gbit/sec? If not, you will never achieve your SAS bottleneck, and will be bottlenecked on the server instead. – Jeremy May 08 '12 at 18:25
  • That's a full-duplex link. If using high-RPM SAS disks, you could probably saturate that connection with one shelf... But @Jeremy is correct that the inter-enclosure links are a bottleneck. – ewwhite May 08 '12 at 18:26
  • 2
    Ok, so I just need you guys to understand that I was just trying to see how Dell scaled the disks across the various lanes. Anyway, 12Gbps as a whole is not slow at all. Think about it this way, three to 4 years ago an average ESX server had what, two 1Gbs links for iSCSI? Fast forward to now, usually two 10g links. So... a single 10g link isn't that bad if you really think about it. 1.5GBps isn't what I would call slow – Eric C. Singer May 08 '12 at 18:35
  • oh and look at a SAN like the dell EQL 6510e, 48 disks coming out of two 10g connections. Not that much different. Anyway, so when you say duplex, that means i can do 3Gbps per channel each way, correct? – Eric C. Singer May 08 '12 at 18:38
3

I'm going to argue that while the theoretical sequential max throughput of all your disks is greater than the SAS chain can handle, with a backup server you may very well never reach that limit or come close to it.

Let's look at some limits in your system.

Theoretical max throughput of a 7.2k SAS drive ~ 1.2Gbit/sec (150 megabytes per second) Theoretical max throughput of 45 of your SAS drives= (45*1200) = 54Gbit/sec Theoretical max throughput of your SAS chain = 12Gbit/sec

So we're down to 12Gbit/sec so far.

How is your server connected to the DAS? 3Gb SAS? Ok, you've got 12Gbit again.

Your application is a backup server. Does it really have 12Gbit/sec connectivity to all of its backup clients? If so, can each client saturate the backup network (reading from their own disks) to the point where you would actually get 12Gbit/sec coming into the backup server? Probably not. That is a LOT of throughput! Your network would have to support that traffic. The backup server would have to have enough CPU to process all that traffic. Etc. etc.

My point is simply, if you have a couple of 1Gb NICs on this box and are using it for backup, you very well may never need to worry about the bandwdith of the SAS chain, because you'll never hit that limit before you max out your network or the throughput capabilities of your backup clients.

That said, if I could design the system myself, I would give you more SAS bandwidth, but my take away here is it may not be a problem in the real world at all.

Jeremy
  • 938
  • 2
  • 7
  • 18
  • Don't forget maybe 4x write amplification for RAID when thats on the card not in the enclosure. A valid point about saturation though. – JamesRyan May 08 '12 at 18:39
  • 1
    @JamesRyan I'm not familiar with this concept, could you please elaborate? – Jeremy May 08 '12 at 18:43
  • To be precise on a RAID 5 for example for each write it is 2 reads (data, parity) then 2 writes (new data, new parity), RAID 6 - 3 reads + 3 writes, RAID 1 - 2 writes. This occurs between RAID controller and drives so in a proper SAN or NAS internally, but in a dumb box like the MD1000 it is across the main link. (Also why software RAID can quickly saturate your bus) – JamesRyan May 08 '12 at 18:57
1

All,

So here is the answer, the SAS back plane is basically just a huge bus architecture. It works very much like a hub (not a switch). The md1200's to some extent have no bearing on this. From a generic raid controllers perspective it just see's 45 drives. The lanes aren't being broken apart or anything like that. At least not between the chassis.

SAS, by in large is just a big single cable, the easiest way to think about it is Christmas lights. The cable is the path, and the lights are the HDD's.

Eric C. Singer
  • 2,319
  • 15
  • 17
  • 2
    That's how table-table routing in SAS 2+ works, more ore less. Multiple links between initiator (SAS HBA) and target (storage drive) can be aggregated. Also, there can be multiple paths to the same target (say you had a HBA with 2 miniPorts, and two enclosures; you link from the HBA to each enclose, and a "cross-over" between enclosures; you now have a multi-path layout). It should be noted, this is not how SAS 1 works. – Chris S May 07 '13 at 13:31