8

I have a multihomed Ubuntu server with a set of interfaces that includes:

eth2: 10.10.0.131/24
eth3: 10.20.0.2/24

The default interface is eth2, with a gateway of 10.10.0.1. Here's what the routing table looks like:

root@c220-1:~# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.10.0.1       0.0.0.0         UG        0 0          0 eth2
10.10.0.0       0.0.0.0         255.255.255.0   U         0 0          0 eth2
10.20.0.0       0.0.0.0         255.255.255.0   U         0 0          0 eth3
10.30.0.0       0.0.0.0         255.255.255.0   U         0 0          0 eth0
10.40.0.0       0.0.0.0         255.255.0.0     U         0 0          0 eth1

From a separate network (192.168.3.5/24) I can reach this machine on the eth2 interface (the one with the default gateway), but not the eth3 interface. I can ping the eth3 interface from a router on the same network (10.20.0.1) without a problem.

If I ping 10.10.0.131 from 192.168.3.5, the packets reach the machine, but it doesn't send any replies:

c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 0, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 1, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 2, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 3, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 4, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 5, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 6, length 64
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 98: 192.168.3.5 > 10.20.0.2: ICMP echo request, id 5451, seq 7, length 64

If I ping from the router (10.20.0.1) on the same network, the server replies properly:

c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 114: 10.20.0.1 > 10.20.0.2: ICMP echo request, id 28899, seq 2932, length 80
73:10:73:e4:10:06 > c4:c8:80:90:22:eb, IPv4, length 114: 10.20.0.2 > 10.20.0.1: ICMP echo reply, id 28899, seq 2932, length 80
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 114: 10.20.0.1 > 10.20.0.2: ICMP echo request, id 28899, seq 2932, length 80
73:10:73:e4:10:06 > c4:c8:80:90:22:eb, IPv4, length 114: 10.20.0.2 > 10.20.0.1: ICMP echo reply, id 28899, seq 2932, length 80
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 114: 10.20.0.1 > 10.20.0.2: ICMP echo request, id 28899, seq 2932, length 80
73:10:73:e4:10:06 > c4:c8:80:90:22:eb, IPv4, length 114: 10.20.0.2 > 10.20.0.1: ICMP echo reply, id 28899, seq 2932, length 80
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 114: 10.20.0.1 > 10.20.0.2: ICMP echo request, id 28899, seq 2932, length 80
73:10:73:e4:10:06 > c4:c8:80:90:22:eb, IPv4, length 114: 10.20.0.2 > 10.20.0.1: ICMP echo reply, id 28899, seq 2932, length 80
c4:c8:80:90:22:eb > 73:10:73:e4:10:06, IPv4, length 114: 10.20.0.1 > 10.20.0.2: ICMP echo request, id 28899, seq 2932, length 80
73:10:73:e4:10:06 > c4:c8:80:90:22:eb, IPv4, length 114: 10.20.0.2 > 10.20.0.1: ICMP echo reply, id 28899, seq 2932, length 80

Note that, as per the answer in this similar question, I have rp_filter off on all of the interfaces, but it doesn't resolve the issue:

$ for i in eth0 eth1 eth2 eth3 all default
> do
> cat /proc/sys/net/ipv4/conf/$i/rp_filter
> done
0
0
0
0
0
0
Community
  • 1
Lorin Hochstein
  • 4,868
  • 15
  • 54
  • 72

2 Answers2

18

The problem is that since the default route is through eth2, the ping responses get sent through eth2 even though the requests were received on eth3. (If you tcpdump eth2 you should see the responses being sent.) There's then probably some device which is dropping the packets because they have an invalid source IP for the network they're on. You need some source policy routing in order to get the responses to be sent out the interface they were received on.

  1. Create a new routing table (only needs to be done once):

    echo 13 eth3 >> /etc/iproute2/rt_tables

  2. Add a default route to this new table going out eth3:

    ip route add default via 10.20.0.1 table eth3

  3. Add a policy rule to use this new table for packets with source address of eth3's IP:

    ip rule add from 10.20.0.2 lookup eth3
mgorven
  • 30,036
  • 7
  • 76
  • 121
  • D'oh, I didn't think to do the tcpdump on eth2, I do see the responses. The source policy routing worked. – Lorin Hochstein Mar 14 '13 at 19:34
  • 1
    Can the table label "eth3" be arbitrary, or does it need to match the interface name? – Lorin Hochstein Mar 14 '13 at 20:31
  • 1
    @LorinHochstein The name is arbitrary. – mgorven Mar 14 '13 at 21:02
  • to make it permanent add post-up and pre-down commands to the interfaces file (at least on debian based distros) in my case: `post-up /sbin/ip route add default via 192.168.0.1 table eth1` `post-up /sbin/ip rule add from 192.168.0.100 lookup eth1` `pre-down /sbin/ip route del default via 192.168.0.1 table eth1` `pre-down /sbin/ip rule del from 192.168.0.100 lookup eth1` – cusco Mar 10 '17 at 21:57
  • The table name is even optional, you can use a number. (name is only there to make it easier to read and understand the rules) – NiKiZe Aug 15 '21 at 14:36
1

From a separate network (192.168.3.5/24) I can reach this machine on the eth2 interface (the one with the default gateway), but not the eth3 interface. I can ping the eth3 interface from a router on the same network (10.20.0.1) without a problem.

It sounds like you are missing a route for 192.168.3.5/24 from the 10.30.0/24 subnet. You should add a network diagram and traceroutes for each network from each device.