6

I'm a bit confused regarding the setup of a SAN with ISCSI and multipath. We are upgrading our network to 10Gbit, so there are 2 10Gbit switches that are configured with MLAG for normal ethernet traffic. LAG groups are made on the appriorate NICs, so for ethernet only.

Now there's a SAN with 2 controllers. Each controller houses 4 x 10Gbit NICs. So my initial plan was to use all 4 NICs on each controller to maximize throughput, as there are multiple servers connecting to the SAN.

The servers connecting house 2 dedicated 10Gbit NICs for SAN traffic. Each NIC connects to one of the switches. There's no LAG on those interfaces, they are configured with a static address in the SAN network.

  1. Is it OK for the 2 controllers of the SAN to be in the same subnet? So to have one main SAN network. Or do I need an independent segment for each controller?
  2. What about the 4 NICs of each SAN controller? Do I just bundle 2 NICs in a LACP group on each controller? Or is LAG not needed at all? I'm a bit confused what a reasonable configuration would be. Connect 2 NICs of each controller to one switch? Would I assign each NIC a dedicated address in the SAN network and use all the 8 addresses in total for multipathing?

Well, I'm not a storage expert as one can see,...:-)

So I got an update on question number 1. Assuming I'm going to use 2 NICs of each SAN controller (not all 4 to keep it simple for the time being), it would make sense to use two different subnets. It seems that this increases the chance that MPIO is being used efficiently.

If all the NICs are on the same subnet there's no guarantee that MPIO is being used.

I'm still investigating answer 2 though. I found multiple articles describing a setup as outlined above, but just using 2 NICs of each SAN controller. I tested that out, but contradicting to all the articles I don't get 1 single ISCSI target, I get two (one for each controller). When configuring MPIO using both targets my throughput drops from 1800MB/s to around 30MB/s,...

Switches: 2 x Mellanox SX1012

Storage: QSAN XS5200 with 4 x SFP+ 10Gbit adapter per controller

Server: Supermicro 2028TP-HC1R-SIOM, 4 x SFP+ Intel X710

The current configuration looks as follows:

enter image description here

inzanez
  • 231
  • 1
  • 6
  • 1
    Please provide real specifics on the storage hardware and the switch make/manufacturer. Otherwise, it's not possible to give detailed advice. – ewwhite Dec 08 '16 at 12:56
  • 1
    Good point. Done. – inzanez Dec 08 '16 at 13:00
  • 1
    And what type of systems will be consuming your SAN resources? VMware? Linux? Windows? – ewwhite Dec 08 '16 at 13:14
  • 1
    It's Windows Server 2012 R2. Currently there are 4-5 machines planned. MPIO is active, multipath selected when configuring the targets on Windows ISCSI control panel. – inzanez Dec 08 '16 at 13:17

2 Answers2

4

Since you have iSCSI SAN, I suggest you go with MPIO splitting iSCSI networks and utilize different subnets. LACP does same failover as MPIO, however comes with performance penalty due to unsupported Multiple Connections per Session (MCS) meaning, it would not performance boost for same configuration. https://www.starwindsoftware.com/blog/lacp-vs-mpio-on-windows-platform-which-one-is-better-in-terms-of-redundancy-and-speed-in-this-case-2

Tweak the performance configuring NICs and switches Jumbo Frames value of 9k.

Mr. Raspberry
  • 3,878
  • 12
  • 32
  • 1
    Well, as I said, I used two NICs on each server on a different VLAN without LACP to connect to the two ISCSI targets that are presented by the SAN controllers. When only connecting to one target (so only one controller), speed is great, I can see that MPIO is being used (I connect twice, specify the different NICs from OS and the target IPs). As soon as I add the second target (from controller 2), speed drops dramatically. – inzanez Dec 08 '16 at 13:55
  • 1
    Is Round Robin polity enabled? – Mr. Raspberry Dec 08 '16 at 15:15
  • 1
    Yes, policy is set to round robin – inzanez Dec 08 '16 at 15:21
1

This is a case where you should follow your storage vendor's guidelines.

I don't know who will be consuming your SAN resources, but the general approach should be:

  • Your regular servers using LACP should have LACP bonds comprised of links to both switches.
  • Your MLAG should have some sort of cross-switch peer link.
  • Your iSCSI should be delivered via MPIO, using one or more ports from one controller to one switch on a specific VLAN, and one or more ports from the other controller to the other switch on a different VLAN.
  • Those iSCSI VLANs should be unique to each switch and not participate in the MLAG.
  • If your servers will be using iSCSI directly, go with the MPIO approach... two VLANs and one port from the server to each switch. This may complicate normal data if you only have two ports per server.
ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • 1
    I got 4 ports per server, so LAN will be configured with LACP, not touching the two SAN NICs at all. Currently I configured each controller with two VLANs as you see (1 NIC to Switch1, 1 NIC to Switch2). So each switch houses both VLANs. You suggest that I go from "Controller1" only to Switch1, and have both NICs of 'Controller1' in the same VLAN & subnet? To clarify my drawing above: RED would be VLAN 20, Blue VLAN 21... – inzanez Dec 08 '16 at 13:24