I have an EC2 machine running Ubuntu 20.04 with 2 ethernet interfaces. They are both connected to the same subnet and both are reachable from the outside. I have created 2 custom route tables, one for each interface and both just containing the route entries for the machine itself, the immediate subnet and a default gateway via the corresponding interface. I can add a rule for all traffic to use either of the custom route tables and it works: the machine will use the specified interface and the corresponding ip address for outbound communication. However if I add rules to use either custom route table based on packet markings that I apply via iptables, everything stops working and no traffic is leaving my machine.
Here are the main bits of relevant configuration:
$ sudo ip route
10.0.11.0/24 dev ens6 proto kernel scope link src 10.0.11.52
10.0.11.0/24 dev ens5 proto kernel scope link src 10.0.11.201
$ sudo ip route show table if1
default via 10.0.11.1 dev ens5 proto static metric 100
10.0.11.0/24 dev ens5 proto static scope link src 10.0.11.52
10.0.11.201 dev ens5 proto static scope link metric 100
$ sudo ip route show table if2
default via 10.0.11.1 dev ens6 proto static metric 200
10.0.11.0/24 dev ens6 proto static scope link src 10.0.11.201
10.0.11.52 dev ens6 proto static scope link metric 200
$ ip rule
0: from all lookup local
46: from all fwmark 0x3 lookup if1
47: from all fwmark 0x2 lookup if2
50: from 10.0.11.201 lookup if1
100: from 10.0.11.52 lookup if2
32766: from all lookup main
32767: from all lookup default
$ sudo iptables -S -t mangle
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A OUTPUT -m owner --uid-owner 1002 -j MARK --set-xmark 0x3/0xffffffff
-A OUTPUT -m owner --uid-owner 1003 -j MARK --set-xmark 0x2/0xffffffff
My goal is to route traffic from the user with uid 1002 over interface ens5 (if1) and traffic from user with uid 1003 via ens6 (if2) but I'm doing something wrong somewhere and it drives me nuts.... I can see that marked packets are being treated differently, so something is happening. If I assign a default route to the main routing table and then just mark traffic for uid 1002, all users can make outbound connections except for uid 1002. So I know the marking works, I know marked packets get routed differently and I know my custom route tables work but somewhere in this constellation there's an error and I can't find it. Any help would be greatly appreciated at this point!
--edit--
I arrived at this by basing my setupn on the approach outlined here: Trying to route outgoing http requests through VPN on Ubuntu Server 12.04 and some customizations as I don't wanna route traffic based on specific ports but originating uid.
