ARP replies contain wrong MAC address

Question

I've got a robot running linux with wired and wireless adapters. When I boot up, it connects to the wireless fine. When I assign an IP to the wired (either statically or with DHCP), it looks like it works. As in, ifconfig shows a proper IP and route shows proper routes. However, when I do an ARP request of the wired IP, the ARP reply contains the wireless MAC.

??? There's no bridge running on the robot, so why don't I get the wired MAC???

When the wire is disconnected, the wired IP replies to ping...

Why is the robot replying over the wireless interface to IP requests on the wired???

EDIT: both the wired and wireless adapters on the same IP subnet. I do an ARP request from a computer (tried with different computers) on the same IP subnet.

relevant ifconfig output:

eth0      Link encap:Ethernet  HWaddr 00:01:C0:04:BD:F7  
          inet addr:192.168.0.110  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
ra0       Link encap:Ethernet  HWaddr 24:3C:20:06:3E:6D  
          inet addr:192.168.0.101  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:59 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:31023598 (29.5 MiB)  TX bytes:85640627 (81.6 MiB)

relevant route output:

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 ra0
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0

It's a very cutdown linux, so I don't have tools like artptables, iptables, sysctl, brctl, etc.

EDIT: diagram as requested

EDIT: I am dumping traffic and looking at the ARP table. An ARP request of 192.168.0.110 returns an ARP reply containing 24:3C:20:06:3E:6D. The source MAC of the ARP reply packet is also 24:3C:20:06:3E:6D. I've tried fiddling with _filter, _ignore, and _announce, as mentioned here, but to no avail.

EDIT: setting a gateway (on either interface) makes no difference (as it shouldn't).

EDIT: this worked fine on a previous version of the OS (based on openembedded). is it possible they changed something?

Answer 1

What you are seeing is normal behavior when you have two interfaces on the same network. It is described in this LWN article.

Answer 2

When you say you get an ARP response for the wrong interface, are you actually dumping traffic or just looking at the resulting ARP table? It's possible you're getting ARP replies for both interfaces...

Anyway, I believe the answer to your problem lies in properly manipulating rp_filter and arp_filter. The documentation for each of them is included below.

I suggest first trying this:

echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter

You may need to make this change as well:

echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter

rp_filter - BOOLEAN
    1 - do source validation by reversed path, as specified in RFC1812
        Recommended option for single homed hosts and stub network
        routers. Could cause troubles for complicated (not loop free)
        networks running a slow unreliable protocol (sort of RIP),
        or using static routes.

    0 - No source validation.

    conf/all/rp_filter must also be set to TRUE to do source validation
    on the interface

    Default value is 0. Note that some distributions enable it
    in startup scripts.

arp_filter - BOOLEAN
    1 - Allows you to have multiple network interfaces on the same
    subnet, and have the ARPs for each interface be answered
    based on whether or not the kernel would route a packet from
    the ARP'd IP out that interface (therefore you must use source
    based routing for this to work). In other words it allows control
    of which cards (usually 1) will respond to an arp request.

    0 - (default) The kernel can respond to arp requests with addresses
    from other interfaces. This may seem wrong but it usually makes
    sense, because it increases the chance of successful communication.
    IP addresses are owned by the complete host on Linux, not by
    particular interfaces. Only for more complex setups like load-
    balancing, does this behaviour cause problems.

    arp_filter for the interface will be enabled if at least one of
    conf/{all,interface}/arp_filter is set to TRUE,
    it will be disabled otherwise

For a more thorough treatment, see this article:

http://www.embedded-bits.co.uk/tag/rp_filter/

Answer 3

I know this is an old issue but I recently encountered the exact same situation with an embedded device. The device has both an ethernet and wifi interface and the requirements are that both interfaces can be active and on the same network at any time, but network traffic must be routed through the "preferred" interface.

Most users wouldn't configure their devices this way but in theory it should be possible.

We first picked up the issues with Netgear routers because they would report an IP Address conflict - 2 MAC addresses were sharing a single IP. Apparently the router would start behaving badly in this scenario and mess up the users network.

I created a private network that only contained the router (ethernet + wifi), windows laptop (Ethernet only), and the embedded device (ethernet + wifi). Using wireshark, tcpdump on the device, and arp on windows I can see the following behaviour:

1. Ifconfig on the device shows distinct wln and Ethernet IP’s and distinct MAC addresses
2. Sometimes (very rarely) an arp –a from windows shows the correct IP-MAC combination.
3. Most of the time arp –a from windows shows both the wln and eth0 have the same MAC address
4. When pinging either wln or eth0 from windows, the ping response comes from wln and very rarely from eth0. tcpdump shows the wln only responded to 1 of the 4 pings (for example)
5. When windows sends an arp “who has” message for the eth0 IP – both the eth0 and wln interfaces respond saying they have that IP

I believe that item 3 is caused by item 5. The arp tables are being messed up because the wln is responding to arp messages that only eth0 should respond to. I believe item 4 is also caused by item 5. Ping is sent based on MAC address and since the last arp message received was from wln saying it has the eth0 IP, the pings are routed incorrectly to the wln interface.

After much digging and testing the solution was actually really simple. See this article - https://chrisdietri.ch/post/preventing-arp-flux-on-linux/

The Linux kernel network drivers are configured in such a way that when an arp request is received for a known interface (even if it is received on another interface) it will respond to the arp.

This setting resolves the issue:

echo 1 > /proc/sys/net/ipv4/conf/wln/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/eth0/arp_ignore

Explanation:

arp_ignore - INTEGER
Define different modes for sending replies in response to
received ARP requests that resolve local target IP addresses:
0 - (default): reply for any local target IP address, configured
on any interface
1 - reply only if the target IP address is local address
configured on the incoming interface
2 - reply only if the target IP address is local address
configured on the incoming interface and both with the
sender's IP address are part from same subnet on this interface
3 - do not reply for local addresses configured with scope host,
only resolutions for global and link addresses are replied

Answer 4

As this worked fine on a previous version of the OS (based on openembedded), my solution was to wait for the next version of the OS. My best guess was that the wireless kernel module was buggy.

Answer 5

Following up on Insyte's comment.

Lets do some naming:

• PC1 - Top Right
• PC2 - Top Left
• PC3 - Bottom Left

For you have your robot reachable from the 3 PCs via both wired and wireless media. And for they are on same subnet you cannot for sure tell that which way the arp request for your wired media has gone through. By that what I mean is when the switch broadcasts for an arp request, your robot receives it on both the interfaces [referring to your diagram] so for it receives the arp request for the IP on the wired media on the wireless media too chances are it replied with the physical address of the wireless media for the box does have that IP configured in it

I have had this issue in the past, it wasn't exact to yours but was similar. By default linux replies with the physical address of the interface it receives the arp request regardless of which interface the IP is configured onto. So in your case connect PC3 to the robot's eth0 interface directly and do an arp request for 192.168.0.101 it would reply you with the physical address of the eth0 interface instead of ra0.

My deployment scenario was:

[RTR] |------------eth0---[server]
|--------| switch1 |-----eth1-----[server]

Its the same switch, that both interfaces connect to. Hopeful that it would help you.

The router had primary and secondary IP address configured on its interface for two different networks on the two different interfaces on the server. But receiving an arp request on eth1 for IP address of eth0 it did reply with the physical address of eth1

To prevent it the following has so far worked for me

# echo 2 > /proc/sys/net/ipv4/conf/eth0/arp_announce
# echo 1 > /proc/sys/net/ipv4/conf/eth0/arp_ignore
# echo 2 > /proc/sys/net/ipv4/conf/ra0/arp_announce
# echo 1 > /proc/sys/net/ipv4/conf/ra0/arp_ignore

put it somewhere on your robot so you can get it be applied at boot.

Recommendations: I would recommend you have two different subnets be configured (say 192.168.1.x/24 on ra0 and 192.168.2.x/24 on eth0) you can use IP aliases on your PCs and your robot would be accessible over any of the two IPs. You cannot have two outbound paths for same subnet on a same host. Not unless there is something that makes your robot prefer one over another. Your robot can only take one path to send packets out of it.

Some readings: arp_announce, arp_ignore

Answer 6

i think there is a misconfiguration between your wireless AP and your switch. switch and AP are getting confused where to send packets. not sure about this though. also, i think you should try to define a gateway where programs can know where to send packets. something like

route add default gw 192.168.0.1