4

Linux server has 2 active network interfaces:

IF:eth1    IP:192.168.1.1/24    MAC:11:11:11:11:11:11   (1GbE)
IF:eth2    IP:192.168.1.2/24    MAC:22:22:22:22:22:22   (10GbE)

The idea is that the 10GbE interface (eth2) is the primary interface for communication with hosts on the network. I want to leave the second 1GbE interface (eth1) up as a failsafe. In the event that the 10GbE interface goes down: I'd still have an easy way in, can update DNS so hosts can connect, etc.

While observing interface statistics I noticed that all traffic was sending/receiving on the eth1 instead of eth2 despite the fact that all hosts on the network are addressing this interface. I confirmed DNS A record points to the IP of the correct interface. Additionally, I confirmed that addressing the interface by IP instead of FQDN produces the same result.

I cleared the ARP cache on my machine and pinged the eth1 interface by IP address. I inspect my ARP table and find the MAC address of eth1. I cleared the ARP cache again and pinged the eth2 interface by IP address. Again, I inspect my ARP table and find the MAC address of eth1 (not eth2).

If I bring down eth1, physically disconnect the interface, or put it on a different logical network - I get the expected behavior, traffic goes over my eth2 interface.

My question: Why does this happen? I am seeing some evidence that this is expected behavior on the linux kernel due to its "weak host model."

How can I keep both interfaces up, on the same network, and have them work in the way that I expect.

sardean
  • 833
  • 3
  • 14
  • 34

2 Answers2

3

Linux is designed to respond to ARP requests on any interface. It is assumed that the host owns the IP address and not the particular interface. What you are seeing is called ARP Flux.

You can change this behavior using sysctrl

arp_ignore - INTEGER

Define different modes for sending replies in response to received ARP requests that resolve local target IP addresses:

0 - (default): reply for any local target IP address, configured on any interface

1 - reply only if the target IP address is local address configured on the incoming interface

2 - reply only if the target IP address is local address configured on the incoming interface and both with the sender's IP address are part from same subnet on this interface

3 - do not reply for local addresses configured with scope host, only resolutions for global and link addresses are replied

David Houde
  • 3,160
  • 1
  • 15
  • 19
  • This makes perfect sense. Quick follow up question to see if I understand the arp_ignore options correctly. I presume this parameter is set for the kernel and not per-interface? If this is correct, will arp_ignore 1 produce the results I am seeking - where arp will reply with the MAC address of the interface of which it is assigned? – sardean Mar 06 '14 at 02:31
  • 1
    You are correct on both counts. Set it to 1 or 2 and you should see the correct ARP replies on the client. – David Houde Mar 06 '14 at 02:41
3

If your switch supports it, I would use 802.1ad link aggregation for providing failover.

With this feature, you bond the two interfaces together, and you can set one as active and one as passive interface. Your IP address would reside on the bonding interface, so there would be no IP address changes if one NIC fails.

Tero Kilkanen
  • 34,499
  • 3
  • 38
  • 58
  • +1, this was my first instinct when reading the question, as it would be a more desirable solution as long as the switch supports it. – David Houde Mar 06 '14 at 12:15
  • Thank you, this is a great idea. I didn't know that 802.1ad had an option for active/passive. Is it not a problem to mix a gigabit and 10gig interface in the bond? If that works, I will use this solution going forward. – sardean Mar 06 '14 at 13:36
  • I haven't tested the setup with different speed interfaces myself, but I don't see anything that would prevent it from working. – Tero Kilkanen Mar 06 '14 at 15:26
  • @TeroKilkanen after doing some additional research it appears as though most, if not all, 802.1ad implementations require identical speed links for members in the bond. – sardean Mar 07 '14 at 02:47