7

We want to implement an iSCSI SAN and all of our testing has shown that we can implement this quite cheaply (Starwind target, refurbished HP Storage array).

What concerns me is the throughput/latency of the switch itself, so open questions:

  1. Which brand/model of switch (1Gbps) would you recommend for an iSCSI rollout and why?
  2. What bad experiences have you had with switches in an iSCSI environment?

Thanks,
Rick.

Rick
  • 245
  • 3
  • 8

3 Answers3

8

I agree completely with ynguldyn's answer - for the most part any modern switch intended for use in a server-room\data center should be more than good enough for your needs and keeping things consistent in your environment is likely to be more important to you from a support\manageability perspective.

That said if you really want to get the most out of your iSCSI set up use switches that have:

Sufficient per port buffer memory. Ideally something >512k per port but there is a trade-off here. Some switches use larger buffers to mask poor switching speed so you need to look for more than this. Too little buffer memory will lead to packet loss under heavy load and the TCP layer will have to resend packets which will slow everything down dramatically.

Sufficient per port processing capability. This can be hard to establish - the best metric to look for is switching speed. A switch with 100microsec switching speed can only handle 10k packets/sec which would not be able to switch GigE at line speed, one with a 3microsec switching speed can (in theory) handle up to 300k packets/sec, which is fine. Anything below 12microsec is likely to be good enough. Faster is better but prices go up quite dramatically as that number heads for low single digits.

Support for hardware flow control (802.3x). This will be useless if your server NICs and array don't also support this but if they do it allows your iSCSI network to handle flow control much more efficiently at layer 2 rather than rely on higher level congestion control such as TCP's congestion avoidance algorithms which will be significantly less efficient. That said it's hard to find a proper switch that doesn't support it today.

Support for Jumbo frames. Again this is only going to be beneficial if your iSCSI array, server hardware, and OS also support jumbo frames. At the most basic level Jumbo frames decreases protocol overhead and can push throughput up by 10-20% but those gains depend highly on traffic patterns. For extended high bandwidth data transfers 9k Jumbo frames will reduce the CPU overhead on your array, servers (and switch) by up to 80% as well. This may or may not be significant in your environment as the initial CPU overhead may be relatively low. Low end switches sometimes claim Jumbo frame support but don't support 9k Jumbo frames which is the generally agreed optimal size for GigE so check that first. If your array doesn't support Jumbo frames there's no need to worry about this, obviously.

High bandwidth switching and stacking capability. For GigE you should be aiming to be >1Gbps per port, ideally 2Gbps to handle full duplex traffic at line speed across all ports. For a 24 port switch you want it to be able to switch 48Gbps internally and be able to be stacked\uplinked at a significant percentage of that if you are using multiple switches. For some iSCSI architectures (e.g. HP Lefthand and Dell Equallogic) you need to support very high bandwidth traffic between all ports on all arrays and the aggregate switching speed becomes very important. For switches that support mixed 1GigE and 10GigE adjust accordingly the total switching bandwidth should cover all ports running at full speed in full duplex mode.

Spannning Tree. You want to be able to disable it completely if your iSCSI environment is simple enough and isolated from everything else or have it support Rapid Spanning Tree\Port Fast\Edge Ports where you can selectively disable full spanning tree behaviour on specific ports.

Helvick
  • 19,579
  • 4
  • 37
  • 55
  • So, bottom line is that with 1 target and 4 initiators almost any 16-port "datacenter" GigE switch should suffice IF it has 9k Jumbo frames. Thanks for the other tips. – Rick Dec 20 '09 at 00:35
  • Yes - you're not going to push the limits on a switch in this scenario unless it's very low end - you only need to support 10Gbps total switching bandwidth (at most) and your major concern should be 802.3x and 9k Jumbo frame support. One other thing is to keep this network isolated from your production LAN - iSCSI and "normal" traffic should be kept isolated from each other, but as I read your question I was assuming you were doing this. – Helvick Dec 20 '09 at 01:46
4

GigE is an old and stable technology, and the processing power of modern switches is sufficient to handle it very easily, especially when it's just one target and a handful of initiators. You should expect any decent switch ($20 little-boxes-that-developers-hide-under-their-desks-to-annoy-sysadmins of course excluded) to have no timing or performance issues in the SAN environment. Relevant feature sets are also pretty much the same across all of them, with jumbo frames, flow control, VLANs, and everything else you might need.

Instead, you should focus on the budget, existing vendor relationships, installed hardware and in-house expertise: get what you can afford and what you know best, and stick to the same brand you already use (two reasons: fewer manuals to read means deeper knowledge of what you have, and you'll avoid interoperability issues). Cisco, ProCurve, Nortel, high-end Netgears should all be good.

Max Alginin
  • 3,284
  • 14
  • 11
  • Everything apart from the Netgears. Even the high-end ones can't compare with the quality of Cisco or Procurve. Force10 are also worth a look, not cheap but very very fast. – Tom O'Connor Dec 19 '09 at 12:40
  • Reliability is a relative thing. There's plenty of people out there who're ok with losing one switch out of five every year, if those switches cost them half of what they'd pay for a Procurve (which, in turn, is 60-70% of comparable Catalysts). And when they work, they work - all frames get where they need to go. – Max Alginin Dec 19 '09 at 15:38
-1

iSCSI is just a protocol like any other; the main issue to look for is that you probably want a switch that supports Jumbo Frames correctly.

pjz
  • 10,497
  • 1
  • 31
  • 40