Link bonding across multiple switches?

Question

I've read up a little bit on bonding nics with ifenslave; what I'm having trouble understanding is whether there is special configuration needed in order to split the bonds across two switches. For example, if I have several servers that all have two nics each, and two separate switches, do I just configure the bonds and plug 1 nic from each into switch #1 and the other from each into switch #2? or is there more to it than that?

If the bonds are active-backup, will a nic failure on single machine mean that server may become disconnected since the rest of the machines are using the primary nic and it's using the secondary? Or do you link the switches with one cable as well?

score 6 · Accepted Answer · answered Oct 23 '12 at 23:15

From the Linux Kernel Network Bonding documentation:

In a topology such as the example above, the active-backup and broadcast modes are the only useful bonding modes when optimizing for availability; the other modes require all links to terminate on the same peer for them to behave rationally.

Note, the example referenced in the quote is for a single host with 2 (or more) NICs, with each NIC connected to a separate switch. The switches have a link between them (ISL or Inter-Switch Link) connecting them together.

score 5 · Answer 2 · answered Jul 12 '12 at 04:54

5

It depends on the bonding mode which you're using. The 802.3ad mode requires that the switch(es) support 802.3ad and are specifically configured for this. The other modes (balance-rr, active-backup, balance-xor, balance-tlb and balance-alb) don't require specific switch support.

I'm not sure how the inter-switch connectivity affects the choice of bonding mode, but I suspect that the switches need to be connected to each other and have STP enabled. This page looks like a good resource for bonding.

answered Jul 12 '12 at 04:54

mgorven

30,036
7
76
121

2

FYI... STP is orthogonal to ethernet bonding. – Mike Pennington Jul 12 '12 at 05:02
@MikePennington Yup, I realise that it's not directly related. – mgorven Jul 12 '12 at 05:06
As stated in the question, it would be using active backup. The switching issue is this: if the switches are not connected, then all nodes would have to be using the nic connected to the same switch, and a failure of that nic would isolate it. – Bryan Agee Jul 12 '12 at 06:26

score 3 · Answer 3 · edited Apr 13 '17 at 12:14

3

... what I'm having trouble understanding is whether there is special configuration needed in order to split the bonds across two switches.

If you want to bond a link across switches, first you need a switch that supports it. LACP (802.3ad) only supports bonding within a single switch. All the inter-switch ethernet bonding protocols I know of are proprietary.

Contact your switch vendor to understand whether they have a multi-chassis ethernet bonding protocol that works with your server OS.

EDIT:

as mgorven pointed out in the comments, you can bond across multiple switch without needing 802.3ad

edited Apr 13 '17 at 12:14

Community

1

answered Jul 12 '12 at 05:00

Mike Pennington

8,266
9
41
86

AFAICT the Linux bonding modes (apart from 802.3ad) don't require switch support. Do these modes not work when connected to multiple switches? – mgorven Jul 12 '12 at 05:09
2

Fair point, as long as you have all ports in the same vlan and you don't want to split traffic for a single mac-address simultaneously across multiple ethernet ports it *should* work – Mike Pennington Jul 12 '12 at 05:23
1

Using balance-xor without a group/channel will result in severe packet loss. – 3molo Sep 02 '12 at 07:18

score 2 · Answer 4 · answered Oct 22 '12 at 18:34

The only bonding mode that achieves the high-availability we're looking for is the active-backup bonding, with one cable from each bonding group attached to diverse switches, which themselves are interconnected. This allows for any given switch to fail, pushing all of the traffic to the other switches and NICs connected thereto; or any number of individual NICs can fail provided that no two are in the same bond and all switches are operational.

802.3ad requires managed switches with support for it, and xor- or rr-balanced require that the switches in question support port grouping of some kind.

Link bonding across multiple switches?

4 Answers4

Linked

Related