2
TL;DR:
In a bit of a complex routing scenario, ICMP and DNS requests/messages works as expected from host to the internet through an embedded router, but web browsers (so i guess TCP and/or HTTP) do not. How can I debug issue with RPDM+routing tables? it seems like when I add a default routing for the main table of the router, some RPDM rules are ignored and I don't understand the behavior I see.
Full explanation:
I have a Linux embedded device which is connected to 2 ISPs simultaniously, and provide 2 host devices (via eth0, eth1) exclusive access to each ISP. This means host 1 will exclusively use ISP connection 1 and host 2 will exclusively use ISP connection 2. at the same time, I want the embedded device to share a connection to one of the ISPs with a host (lets say, through ISP connection 2 - eth3).
I use RPDM to separate the handling for each device:
32762: from all iif eth3 lookup 102
32763: from all iif eth2 lookup 101
32764: from 192.168.1.102 lookup 102
32765: from 192.168.1.101 lookup 101
where the routing tables are configured as for example as:
default dev eth2 scope link
123.123.123.123 dev eth2 scope link
124.124.124.124 dev eth3 scope link
127.0.0.0/8 dev lo scope link
192.168.1.0/24 dev bridge0 scope link
I use masquerade on both ISP connections:
iptables -t nat -A POSTROUTING -o eth2 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth3 -j MASQUERADE
As long as the device has no default route configured, and the hosts each has DNS configured correctly (i.e. the DNS addresses of the ISPs) - everything works as expected, with no internet access to the device itself.
When I add a default route to the main routing table of the device (lets assume its for eth3), the device gains internet access, but host 1 (which should use eth2) loses connectivity, and some packets are lost somewhere in the way. ICMP from host 1 works OK, as well as DNS. Wireshark on host 1 shows that there are many TCP retransmissions. tcpdump on the device shows that some packets from host 1 end up on eth 3.
How can I figure out why this is not working? it seems like the default route that I add takes precedence over the rules from the RPDM, which should not happen..
setup output here.
Clarification:
I am familiar with iptables/TRACE target. I added trace for every packet going in from the bridge, but saw none going out to eth3 - only to eth2, as opposed to the tcpdump log. maybe im looking for the wrong thing?
+ +
| |
| |
| |
| |
+---------------------------------------------------------+
|---------------------------------------------------------|
|| |eth2 (ISP connection)| |eth3 (ISP connetion 2)||
|| |IP 123.123.123.123 | |IP 124.124.124.124 ||
|| +---------------------+ +-----------------------|
|| ||
|| "Embedded Device" ||
|| +--------+------------+ ||
|| +-------+bridge0 + DHCP server+-----+ ||
|| | |IP 192.168.1.1 | | ||
|| | +---------------------+ | ||
|| | | ||
|--------+------------+ +------+------------|
|| eth0 | |eth1 ||
|| (local connection)| |(local connection)||
|| IP 192.168.10.1 | |IP 192.168.10.2 ||
|---------------------------------------------------------|
+---------------------------------------------------------+
| |
| |
| |
+--------------------+ +--------------------+
|--------------------| |--------------------|
|| host 1 || || host 2 ||
|| IP 192.168.1.100|| || IP 192.168.1.101 ||
|--------------------| |--------------------|
+--------------------+ +--------------------+