How can I route a subnet to a different interface with iproute2?

Question

so I am trying to setup routing with iproute, so a certain client subnet can access a firewalled internet uplink on another interface.

The server has four interfaces:

2: enp2s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether a0:36:9f:e6:7a:9e brd ff:ff:ff:ff:ff:ff
    inet 10.20.30.1/16 brd 10.20.255.255 scope global enp2s0f0
       valid_lft forever preferred_lft forever
    inet6 fe80::a236:9fff:fee6:7a9e/64 scope link 
       valid_lft forever preferred_lft forever
3: enp2s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether a0:36:9f:e6:7a:9f brd ff:ff:ff:ff:ff:ff
    inet 10.132.128.70/26 brd 10.132.128.127 scope global enp2s0f1
       valid_lft forever preferred_lft forever
    inet6 fe80::a236:9fff:fee6:7a9f/64 scope link 
       valid_lft forever preferred_lft forever
4: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether ac:1f:6b:00:d1:ce brd ff:ff:ff:ff:ff:ff
    inet 10.132.128.71/26 brd 10.132.128.127 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::ae1f:6bff:fe00:d1ce/64 scope link 
       valid_lft forever preferred_lft forever
5: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether ac:1f:6b:00:d1:cf brd ff:ff:ff:ff:ff:ff
    inet 172.16.10.1/21 brd 172.16.15.255 scope global eno2
       valid_lft forever preferred_lft forever
    inet6 fe80::ae1f:6bff:fe00:d1cf/64 scope link 
       valid_lft forever preferred_lft forever

Clients can connect to the 10.20.0.0/16 subnet and get an IP address via DHCP. They should then be able to access the internet over the interface enp2s0f1. This interface is a firewalled internet uplink with the gateway at 10.132.128.65. I have no control over the firewall.

The routing rules and tables look like this:

table local:

broadcast 10.20.0.0 dev enp2s0f0  proto kernel  scope link  src 10.20.30.1 
local 10.20.30.1 dev enp2s0f0  proto kernel  scope host  src 10.20.30.1 
broadcast 10.20.255.255 dev enp2s0f0  proto kernel  scope link  src 10.20.30.1 
broadcast 10.132.128.64 dev eno1  proto kernel  scope link  src 10.132.128.71 
broadcast 10.132.128.64 dev enp2s0f1  proto kernel  scope link  src 10.132.128.70 
local 10.132.128.70 dev enp2s0f1  proto kernel  scope host  src 10.132.128.70 
local 10.132.128.71 dev eno1  proto kernel  scope host  src 10.132.128.71 
broadcast 10.132.128.127 dev eno1  proto kernel  scope link  src 10.132.128.71 
broadcast 10.132.128.127 dev enp2s0f1  proto kernel  scope link  src 10.132.128.70 
broadcast 127.0.0.0 dev lo  proto kernel  scope link  src 127.0.0.1 
local 127.0.0.0/8 dev lo  proto kernel  scope host  src 127.0.0.1 
local 127.0.0.1 dev lo  proto kernel  scope host  src 127.0.0.1 
broadcast 127.255.255.255 dev lo  proto kernel  scope link  src 127.0.0.1 
broadcast 172.16.0.0 dev tap0  proto kernel  scope link  src 172.16.0.66 
local 172.16.0.66 dev tap0  proto kernel  scope host  src 172.16.0.66 
broadcast 172.16.3.255 dev tap0  proto kernel  scope link  src 172.16.0.66 
broadcast 172.16.8.0 dev eno2  proto kernel  scope link  src 172.16.10.1 
local 172.16.10.1 dev eno2  proto kernel  scope host  src 172.16.10.1 
broadcast 172.16.15.255 dev eno2  proto kernel  scope link  src 172.16.10.1

table main:

10.20.0.0/16 dev enp2s0f0  scope link  src 10.20.30.1 
10.132.128.64/26 dev eno1  proto kernel  scope link  src 10.132.128.71 
10.132.128.64/26 dev enp2s0f1  proto kernel  scope link  src 10.132.128.70 
172.16.0.0/22 dev tap0  proto kernel  scope link  src 172.16.0.66 
172.16.8.0/21 dev eno2  proto kernel  scope link  src 172.16.10.1

table default:

default via 10.132.128.65 dev eno1 onlink 

I tried setting up the route like this:

echo 200 clients >> /etc/iproute2/rt_tables
ip rule add from 10.20.0.0/16 lookup clients
ip route add default via 10.132.128.70 dev enp2s0f1 table clients
ip route flush cache

but that didn't work. So I have been trying to change the routing rule to

from 10.20.30.1 lookup clients
//and
from iif enp2s0f0 lookup clients

but I still never get a connection to the interface. I test with

ping -I enp2s0f0 10.132.128.70

Since the rule seems ok, I tried different routes:

ip route add 10.20.0.0/16 dev enp2s0f0 table clients
ip route add 10.132.128.64/26 via 10.132.128.65 dev enp2s0f1 table clients
ip route add 10.132.128.64/26 via 10.20.30.1 dev enp2s0f1 table clients
ip route add default 10.20.0.0/16 via 10.132.128.65 dev enp2s0f1 table clients

As you might have guessed, at this point I am just confusing myself. If I understood it correctly, it works like this:

ip route add {source_network} via {gateway} dev {output_device} table clients

Or did I get that wrong? Another thing is, when I do a

ip route get 10.132.128.65 // where we want to end up

The result is

10.132.128.65 dev eno1  src 10.132.128.71 
    cache 

So the route in the default table seems to overwrite the rule for the client table...but the client table rule is active since changes in the clients table change the connectivity for the clients in the 10.20.0.0/16 network.

If there is some iformation missing, I will gladly provide it.

Thanks in advance!

Answer 1

Your following configuration seems ok:

echo 200 clients >> /etc/iproute2/rt_tables
ip rule add from 10.20.0.0/16 lookup clients
ip route add default via 10.132.128.70 dev enp2s0f1 table clients

I assume that you are testing ping from your router and not from one of your client. Then your ping command ping -I enp2s0f0 10.132.128.70 is wrong. In fact 10.132.128.70 is a local address (an address of you router). Icmp requests will then stay local and won't be destinated to either enp2s0f1 or eno1.

Instead, you could try

ping -I enp2s0f0 10.132.128.65

Depending on the firewall, you may see or not ping/icmp reply but you can at least check that packets are sent through the right interface (enp2s0f1). To ensure that: you can use tcpdump:

tcpdump -i enp2s0f1 ip host 10.132.128.65