Response packet on the same interface as incoming in LAN

Question

Currently, I'm struggling with the following scenario:

• I have a server with 2 interfaces in 2 separate LAN subnets. IF1, IF2
• I have a laptop which has an IP address from the first subnet
• When I try to connect from this particular laptop to the second IP address of the server, I don't get a response at all.

For example, when I try to ping 172.31.196.185 from the laptop with IP 172.31.190.129, I can see incoming requests in tcpdump on the ens224 interface, but there is no response request on any other interfaces after that.

Here is my network diagram:

       +-------------------------+
       |                         |
       |  Laptop 172.31.190.129  +---------+
       |                         |         |
       +-------------------------+         |
                                           |                 +-------------------------+
                                           |                 |                         |
                               +-----------+---------+       |       Linux Server      |
                          +----+                     |       |                         |
                          |    | LAN 172.31.190.0/23 +-------+ IF1  -  default gw      |
                   +------+--+ |                     |       | 172.31.190.63           |
    +----------+   |         | +---------------------+       |                         |
    | Internet +---+ Gateway |                               |                         |
    +----------+   |         | +---------------------+       |                         |
                   +------+--+ |                     |       |                         |
                          |    | LAN 172.31.196.0/23 +-------+ IF2                     |
                          +----+                     |       | 172.31.196.185          |
                               +---------------------+       |                         |
                                                             |                         |
                                                             |                         |
                                                             +-------------------------+

Also, I have this script:

IF1=ens160
IF2=ens224

P1_NET=172.31.190.0/23
P2_NET=172.31.196.0/23

IP1=172.31.190.63
IP2=172.31.196.185

P1=172.31.190.1
P2=172.31.196.1

ip route add $P1_NET dev $IF1 src $IP1 table T1
ip route add default via $P1 table T1
ip route add $P2_NET dev $IF2 src $IP2 table T2
ip route add default via $P2 table T2

ip route add $P1_NET dev $IF1 src $IP1
ip route add $P2_NET dev $IF2 src $IP2

ip rule add from $P1_NET dev $IF1 table T1
ip rule add from $P2_NET dev $IF2 table T2

Which is writted in accordance to this link: https://lartc.org/howto/lartc.rpdb.multiple-links.html

I have tried many different ways to make a policy-based routing in my case, but no one didn't succeed...

Answer 1

TL;DR

There is no need of iptables, marks, nor to relax Reverse Path Forwarding/Filter. Rather than the ip rules you used which won't match for outgoing packets, use the commands as described in the LARTC's documentation link you provided:

ip rule add from $IP1 table T1
ip rule add from $IP2 table T2

then everything will work fine.

---

Long version

Even if it's always possible to allow things to work by relaxing Reverse Path Forwarding/Filter using for example sysctl -w net.ipv4.conf.all.rp_filter=2 etc., thus allowing asymmetric routing, asymmetric routing should always be avoided. Especially considering that the Gateway (if acting as strict stateful firewall) or the Laptop might also not allow it for similar reasons. Using iptables and marks to correct a wrong behaviour will certainly not help to understand how will behave the already complex routing setup.

While the linked documentation uses the server's IP as source (for $IF2/ens224: 172.31.196.185) without interface for the ip rule, you are specifying the incoming interface in the rule (dev means iif here). That is the issue: this has not the effect one expects to happen. See the description of iif in ip rule:

iif NAME

select the incoming device to match. If the interface is loopback, the rule only matches packets originating from this host. This means that you may create separate routing tables for forwarded and local packets and, hence, completely segregate them.

While it's not clearly written, the reverse is true: packets originating from this host will only match the loopback interface: they will not match iif ens160 nor iif ens224, but only iif lo (yes iif means incoming interface and iif lo matches locally generated outgoing packets, consider this to translate to "incoming from local system [towards wherever routing rules will tell]"). This does matter.

What happens then:

1. Laptop tries to reach $IP2 (it doesn't nor should know that $IP2 could possibly be reached through Server's $IF1 and $IP1 on same LAN), sends packet through Gateway,

2. Gateway routes packet to Server's $IP2 belonging to $P2_NET, to its $IF2 interface,

3. Server receives a packet belonging to $P1_NET (from 172.31.190.129) on $IF2 interface,

4. Server's Strict Reverse Path rp_filter=1 checks reverse route,

5. As seen above, reverse route doesn't match any iif different than iif lo: the two new rules don't match, so the only remaining matching rule is the rule for the main routing table: through $IF1, as verified with:

    # ip route get 172.31.190.129 from 172.31.196.185
    172.31.190.129 from 172.31.196.185 dev ens160 uid 0 
        cache 
   

6. Reverse path is not using the interface the packet arrived from: packet is dropped.

For the same reasons, this current setup will not allow a correct access to Internet from Server when using $IP2: again it will not match the new rules and try to use $IF1 for this:

# ip route get 8.8.8.8 from 172.31.196.185
8.8.8.8 from 172.31.196.185 via 172.31.190.1 dev ens160 uid 0 
    cache 

So once the two rules are removed and changed, as documented in LARTC, to:

ip rule add from $IP1 table T1
ip rule add from $IP2 table T2

That is:

# ip rule add from 172.31.190.63 table T1
# ip rule add from 172.31.196.185 table T2

Things will work correctly, as a route lookup will tell (now using table T2):

# ip route get 172.31.190.129 from 172.31.196.185
172.31.190.129 from 172.31.196.185 via 172.31.196.1 dev ens224 table T2 uid 0 
    cache 

# ip route get 8.8.8.8 from 172.31.196.185
8.8.8.8 from 172.31.196.185 via 172.31.196.1 dev ens224 table T2 uid 0 
    cache 

Those 3 variants could also have worked (just putting the rule with table T2 for brevity):

ip rule add from 172.31.196.0/23 table T2
ip rule add from 172.31.196.185 iif lo table T2
ip rule add from 172.31.196.0/23 iif lo table T2

Note that as documented in the man page, iif lo will alter behaviour if Server is also routing other systems behind it (again, rules won't match for those systems): better not use it unless needed.

There is no need of iptables and marks for this. Using marks to change interface can cause additional troubles with routing, ARP requests and Reverse Path Filtering (using marks usually requires relaxing rp_filter), so better not use them if it can be avoided.