Linux storing arp cache entries using wrong interface

Question

I'm trying to get two virtual machines (running CentOS) to talk to host (Ubuntu 10.04) using VDE and the outside network.

So far I'm running into an ARP issue.

On Host I have a physical eth0 and a virtual tap0 interfaces bridged by br0. All on the same subnet.

br0       Link encap:Ethernet  HWaddr 78:e3:b5:90:88:df  
          inet addr:172.16.1.3  Bcast:172.255.255.255  Mask:255.0.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:1381 (1.3 KB)

br0:1     Link encap:Ethernet  HWaddr 78:e3:b5:90:88:df  
          inet addr:172.16.1.4  Bcast:172.255.255.255  Mask:255.0.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

eth0      Link encap:Ethernet  HWaddr 78:e3:b5:90:88:df  
          inet addr:172.16.1.1  Bcast:172.255.255.255  Mask:255.0.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:2362 (2.3 KB)
          Interrupt:33 Base address:0x6000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:778 errors:0 dropped:0 overruns:0 frame:0
          TX packets:778 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:62832 (62.8 KB)  TX bytes:62832 (62.8 KB)

tap0      Link encap:Ethernet  HWaddr b6:22:43:93:ed:60  
          inet addr:172.16.1.2  Bcast:172.255.255.255  Mask:255.0.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15 errors:0 dropped:3 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:0 (0.0 B)  TX bytes:1980 (1.9 KB)

When I ping this Host Ubuntu machine from another machine (or from the Guest), the arp cache entry is stored as a br0 entry, so the Ubuntu machine never responds to ping.

Address                  HWtype  HWaddress           Flags Mask            Iface
172.22.64.4                      (incomplete)                              eth0
172.22.64.4              ether   00:25:b3:0f:0b:14   C                     br0

Here's the interfaces file that actually works for setting up the IP addresses on all interfaces: auto lo iface lo inet loopback

auto eth0
iface eth0 inet manual
        address 172.16.1.1
        netmask 255.0.0.0
        gateway 172.16.1.255
        post-up ifconfig eth0 172.16.1.1 netmask 255.0.0.0

auto tap0
iface tap0 inet static
        address 172.16.1.2
        netmask 255.0.0.0
        gateway 172.16.1.255
        pre-up vde_tunctl -t $IFACE
        post-up ifconfig tap0 172.16.1.2 netmask 255.0.0.0
        post-up vde_switch -t $IFACE -s /tmp/vde-$IFACE -d -g rhuser -m 664
        post-down vde_tunctl -d $IFACE

auto br0
iface br0 inet static
        address 172.16.1.3
        netmask 255.0.0.0
        gateway 172.16.1.255
        post-up ifconfig br0 172.16.1.3 netmask 255.0.0.0
        post-up ifconfig br0:1 172.16.1.4 netmask 255.0.0.0
        bridge_ports eth0 tap0

I've tried turning off ARP on br0 by ip link set br0 arp off. I've also tried setting things like arp_ignore for br0. In both cases Ubuntu never responds to ARP requests at all. And this is my first question - why does eth0 not respond to ARP requests when a bridge exists. It does well when bridge is commented out from interfaces.

Is arp_filter useful in this situation? It's documentation says "you must use source based routing for this to work". Can someone throw me a link that explains how I set up "source based routing" (I'm pretty new to the topic).
Can arptables or ebtables be used to control this behavior? Looking at their documentation, I see info on how to react to apr requests and when to respond to them, but not much regarding apr cache behavior.
Generally, should I be doing something with sysctl and ARP or should I set my IP assignments differently? I see guides on the net that tell you how to do this trouble-free, but they all assign guests to a separate subnet, making bridge as a gateway. That would not work for me - I need all devices to be on the same subnet.

Any idea would do as long as:

VMs are on the same subnet as outside network.

VMs can talk to host in order to have NFS, etc.

Host does not necessarily need to talk to outside network.

VDE=http://vde.sourceforge.net/ – dmourati Jan 13 '12 at 03:27 — dmourati, Jan 13 '12 at 03:27

score 2 · Answer 1 · answered Jan 13 '12 at 03:26

2

This is the so called "arp problem."

http://www.ssi.bg/~ja/#arp_announce

See your linux/Documentation/networking/ip-sysctl.txt file for more info about the arp_announce and arp_ignore device flags.

I think you want arp_announce=1 for all interfaces.

Linux storing arp cache entries using wrong interface

1 Answers1