linux routing bug?

Question

I have been struggling with this not easily reproducible issue since a while. I am using linux kernel v3.1.0, and sometimes routing to a few IP addresses does not work. What seems to happen is that instead of sending the packet to the gateway, the kernel treats the destination address as local, and tries to gets its MAC address via ARP.

For example, now my current IP address is 172.16.1.104/24, the gateway is 172.16.1.254:

# ifconfig eth0 eth0      Link encap:Ethernet  HWaddr 00:1B:63:97:FC:DC
          inet addr:172.16.1.104  Bcast:172.16.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:230772 errors:0 dropped:0 overruns:0 frame:0
          TX packets:171013 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:191879370 (182.9 Mb)  TX bytes:47173253 (44.9 Mb)
          Interrupt:17

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.16.1.254    0.0.0.0         UG    0      0        0 eth0
172.16.1.0      0.0.0.0         255.255.255.0   U     1      0        0 eth0

I can ping a few addresses, but not 172.16.0.59:

# ping -c1 172.16.1.254
PING 172.16.1.254 (172.16.1.254) 56(84) bytes of data.
64 bytes from 172.16.1.254: icmp_seq=1 ttl=64 time=0.383 ms

--- 172.16.1.254 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.383/0.383/0.383/0.000 ms
root@pozsybook:~# ping -c1 172.16.0.1
PING 172.16.0.1 (172.16.0.1) 56(84) bytes of data.
64 bytes from 172.16.0.1: icmp_seq=1 ttl=63 time=5.54 ms

--- 172.16.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 5.545/5.545/5.545/0.000 ms
root@pozsybook:~# ping -c1 172.16.0.2
PING 172.16.0.2 (172.16.0.2) 56(84) bytes of data.
64 bytes from 172.16.0.2: icmp_seq=1 ttl=62 time=7.92 ms

--- 172.16.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 7.925/7.925/7.925/0.000 ms
root@pozsybook:~# ping -c1 172.16.0.59
PING 172.16.0.59 (172.16.0.59) 56(84) bytes of data.
From 172.16.1.104 icmp_seq=1 Destination Host Unreachable

--- 172.16.0.59 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

When trying to ping 172.16.0.59, I can see in tcpdump that an ARP req was sent:

# tcpdump -n -i eth0|grep ARP
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
15:25:16.671217 ARP, Request who-has 172.16.0.59 tell 172.16.1.104, length 28

and /proc/net/arp has an incomplete entry for 172.16.0.59:

# grep 172.16.0.59 /proc/net/arp
172.16.0.59      0x1         0x0         00:00:00:00:00:00     *        eth0

Please note, that 172.16.0.59 is accessible from this LAN from other computers.

Does anyone have any idea of what's going on? Thanks.

update: replies to the comments below:

there are no interfaces besides eth0 and lo
the ARP req cannot be seen on the other end, but that's how it should work. the main problem is that an ARP req should not even be sent at the first place
the problem persist even if I add an explicit route with the command "route add -host 172.16.0.59 gw 172.16.1.254 dev eth0"

I'm thinking this is some kind of default behavior, let's see the ARP table too? The other end's arp table may be useful here. — SpacemanSpiff, Nov 16 '11 at 15:39
How do you fix it? Does putting a host specific route get it working again? I wonder if you are somehow getting an ICMP redirect that makes the host think the destination is local. — Paul, Nov 17 '11 at 04:02
It seems like the arp reply isn't coming back. Can you tcpdump on the 172.16.0.59 host ? Is this a vm guest? Check network traffic on the host also. — AndreasM, Nov 17 '11 at 13:26
Can you please post the output of `ifconfig -a`? Do you have other interfaces/IPs assigned to this host? — Khaled, Nov 17 '11 at 15:20

score 7 · Accepted Answer · answered Nov 18 '11 at 14:28

7

It is indeed a linux kernel bug, probably since version 2.6.39. I have posted the question to lkml and netdev lists (see the thread at https://lkml.org/lkml/2011/11/18/191), and it was just discussed in a different netdev thread at http://www.spinics.net/lists/netdev/msg179687.html

The current solution now is either a reboot or to flush all routes and wait 10 minutes for the icmp redirects to expire. To prevent it to happen again,

echo 0 >/proc/sys/net/ipv4/conf/eth0/accept_redirects

helps.

answered Nov 18 '11 at 14:28

Balázs Pozsár

2,085
1
14
16

unfortunately the above doesn't seem to help.. – sivann Feb 11 '14 at 10:10
try doing it for all interfaces: find /proc/sys/net -name accept_redirects |while read x; do echo -n 0 >$x; done or maybe you have an other bug – Balázs Pozsár Feb 11 '14 at 13:49
Thanks, I had already enabled it for all interfaces. The IPs are from IPSEC tunnels (this machine has hundrends of them) and there are always 5-10 of them (172.x) listed in the arp table in eth0 interface listed with (incomplete) HWaddress, and missing HWtype. Those seem to expire, and new ones take their place, but sometimes a reboot is required. – sivann Feb 22 '14 at 09:41

score -1 · Answer 2 · answered Nov 18 '11 at 12:34

-1

172.16.X.X default subnet mask is 255.255.0.0, you have reconfigured it to 255.255.255.0 . So the hosts things 172.16.0.x and 172.16.1.x are on different subnets. thus it will try and ROUTE it through the default gateway.

Changing your subnet mask to 255.255.0.0 will solve the problem.

Can you provide a diagram. If you can't draw a network , it can't be fixed (old network engineers proverb...by me!).

Cheers,

answered Nov 18 '11 at 12:34

The Unix Janitor

2,388
14
13

What web app or light weight desktop app would you recommend for network diagram drawing? – Belmin Fernandez Nov 18 '11 at 14:04
it has nothing to do with what the "default" netmask usually is. anyway, see my answer above. – Balázs Pozsár Nov 18 '11 at 14:29
Thanks for the mark down. So, why do you think the router is generating icmp redirects. – The Unix Janitor Nov 21 '11 at 13:40
The router is generating redirects, because it things the host should be using a different gateway. I think your understanding of the problem is a bug. Unless you'd like to educate me otherwise – The Unix Janitor Nov 21 '11 at 13:42
Please read the threads linked in the accepted answer. The problem is that these routing information are not discarded even though they should be. It is not a problem with the router/gateway. – Balázs Pozsár Nov 25 '11 at 15:13
okay, that might be the case, but don't you think should work out why it's creating redirects? – The Unix Janitor Nov 28 '11 at 17:51
indeed it's a but the icmp redirects are not being flushed..accepted...however you should not rely on icmp redirects for routing...they are a security risk and should be turned off. – The Unix Janitor Nov 28 '11 at 17:59
http://books.google.co.uk/books?id=lD43ahAWm5wC&lpg=PA248&ots=RkCOxJ1i66&dq=icmp%20redirects%20misconfiguration&pg=PA248#v=onepage&q=icmp%20redirects%20misconfiguration&f=false read this!!!!!!! – The Unix Janitor Nov 28 '11 at 22:53
we do not rely on icmp redirects for routing, but I do not have enough space here to describe everything :) anyway, the network is correct as it is. – Balázs Pozsár Dec 15 '11 at 10:53

linux routing bug?

2 Answers2