Using two bonded network interfaces in Ubuntu14.04

Question

I have 6 ethernet cards in my server. 4 of 1Gbps and 2 of 10Gbps, So I have configured all 4Gig cards as bond0 with static IP 144 and 10Gig cards as bond1 with IP 122. The problem I'm facing is that the server always re-directs the incoming traffic to any one bonded interface i.e either bond0 or bond1. And if I reboot the server it is likely to use the other bonded interface.

This is what I'm doing.

scp abc.xyz me@192.168.1.144:

I'm expecting bond0 to receive the traffic here but it doesn't happen so.

even if I do scp with IP 122 it selects bond1. And after I reboot the system and when I try scp to both IPs this time it selects bond0.

But what I expect is static behavior i.e if I do scp to IP 144 always the traffic should be picked by bond0 and if I do scp to 122 the traffic should be redirected to bond1.

Answer 1

What you describe is how Linux works by default when you have multiple interfaces in the same subnet.

Linux by default doesn't have a concept of "this interface and its MAC" and "that interface and its MAC", it just knows "my interfaces and MACs" and will reply to any ARP request on any interface.

You can configure each interface to ARP correctly with tunables:

net.ipv4.conf.all.arp_filter = 1
net.ipv4.conf.all.arp_announce = 2

To get traffic to flow how you want, you need to setup policy routing so that each bond has a separate routing table with the default gateway, and so there are IP rules to match on each bond's source address to put traffic into the desired routing table. This requires applications to bind to the source address to work.

However, this is a complex problematic configuration and I don't recommend it.

If you want different traffic separated on each bond interface, put one bond in a different broadcast domain and a different subnet to the other bond, and route traffic properly.