Multiple GRE tunnels on the same host, only one routes incoming packets

Question

I'm having a routing problem on CentOS 6.2 (kernel 2.6.32-220)

Here's the setup: One host is on my local network. It's talking to two nearly identical hosts at a remote location. The two remote hosts are on all the same networks, acting as redundant backups for each other.

I set up two GRE tunnels from the local host, one to each remote host:

ip tunnel add name tunnel1 mode gre local 10.2.1.2 remote 10.2.1.1
ip link set dev tunnel1 up
ip route add 172.16.1.0/24 dev tunnel1 metric 101

ip tunnel add name tunnel2 mode gre local 10.2.1.4 remote 10.2.1.3
ip link set dev tunnel2 up
ip route add 172.16.1.0/24 dev tunnel2 metric 102

Outgoing packets route properly, no problem. Incoming packets are weird. It appears that whichever tunnel has the route with the higher metric (tunnel2 in the example above) will ignore incoming packets. They come in all right, and can be seen on the local machine with 'tcpdump -i tunnel2', but they are not routed properly to the local networks. They're simply dropped. I can switch the two metrics, and then tunnel1 will drop all incoming packets. The tunnel with the lower route metric will route packets properly, both incoming and outgoing.

Is this "working as designed?" What I'd like to have happen, of course, is for all packets on both tunnels to be properly forwarded. Is this possible?

---

Quick update: One person suggested (I'm not sure why) adding 'key' parameters to the 'ip tunnel' commands. I tried this, both with different keys for each tunnel and with identical keys for both tunnels, but neither trial had no effect.

Answer 1

Thanks to the netdev mailing list, I have an answer. There's a feature I'd never heard of, called "reverse path filtering". When turned on, this kernel routing feature will silently drop incoming packets if they arrive on an interface other than the interface that would be used by outgoing packets to the source address. It's intended as a security feature, and is a good default in most circumstances, particularly when you don't have iptables set up carefully.

CentOS (and presumably RHEL) turns on reverse path filtering by default in /etc/sysctl.conf (net.ipv4.conf.default.rp_filter = 1). I added the following commands to my setup script:

echo 0 > /proc/sys/net/ipv4/conf/tunnel1/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/tunnel2/rp_filter

This turns off reverse path filtering for both pseudo-interfaces, and everything works now.

Note that adding to sysctl.conf for my specific interfaces would not have worked for me, since these interfaces are created "on the fly", and sysctl.conf is only read at boot time. I could, of course, have changed the default in sysctl.conf, but I didn't want to disable the feature for my other interfaces.