Debug RPDM/Routing setup with working ICMP+DNS nut no TCP

Question

TL;DR:

In a bit of a complex routing scenario, ICMP and DNS requests/messages works as expected from host to the internet through an embedded router, but web browsers (so i guess TCP and/or HTTP) do not. How can I debug issue with RPDM+routing tables? it seems like when I add a default routing for the main table of the router, some RPDM rules are ignored and I don't understand the behavior I see.

Full explanation:

I have a Linux embedded device which is connected to 2 ISPs simultaniously, and provide 2 host devices (via eth0, eth1) exclusive access to each ISP. This means host 1 will exclusively use ISP connection 1 and host 2 will exclusively use ISP connection 2. at the same time, I want the embedded device to share a connection to one of the ISPs with a host (lets say, through ISP connection 2 - eth3).

I use RPDM to separate the handling for each device:

32762: from all iif eth3 lookup 102 
32763: from all iif eth2 lookup 101 
32764: from 192.168.1.102 lookup 102 
32765: from 192.168.1.101 lookup 101 

where the routing tables are configured as for example as:

default dev eth2 scope link 
123.123.123.123 dev eth2 scope link 
124.124.124.124 dev eth3 scope link 
127.0.0.0/8 dev lo scope link 
192.168.1.0/24 dev bridge0 scope link

I use masquerade on both ISP connections:

iptables -t nat -A POSTROUTING -o eth2 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth3 -j MASQUERADE

As long as the device has no default route configured, and the hosts each has DNS configured correctly (i.e. the DNS addresses of the ISPs) - everything works as expected, with no internet access to the device itself.

When I add a default route to the main routing table of the device (lets assume its for eth3), the device gains internet access, but host 1 (which should use eth2) loses connectivity, and some packets are lost somewhere in the way. ICMP from host 1 works OK, as well as DNS. Wireshark on host 1 shows that there are many TCP retransmissions. tcpdump on the device shows that some packets from host 1 end up on eth 3.

How can I figure out why this is not working? it seems like the default route that I add takes precedence over the rules from the RPDM, which should not happen..

setup output here.

Clarification:

I am familiar with iptables/TRACE target. I added trace for every packet going in from the bridge, but saw none going out to eth3 - only to eth2, as opposed to the tcpdump log. maybe im looking for the wrong thing?

            +                                 +
            |                                 |
            |                                 |
            |                                 |
            |                                 |
+---------------------------------------------------------+
|---------------------------------------------------------|
||  |eth2 (ISP connection)|       |eth3 (ISP connetion 2)||
||  |IP 123.123.123.123   |       |IP 124.124.124.124    ||
||  +---------------------+       +-----------------------|
||                                                       ||
||                    "Embedded Device"                  ||
||               +--------+------------+                 ||
||       +-------+bridge0 + DHCP server+-----+           ||
||       |       |IP 192.168.1.1       |     |           ||
||       |       +---------------------+     |           ||
||       |                                   |           ||
|--------+------------+               +------+------------|
||  eth0              |               |eth1              ||
||  (local connection)|               |(local connection)||
||  IP 192.168.10.1   |               |IP 192.168.10.2   ||
|---------------------------------------------------------|
+---------------------------------------------------------+
        |                                   |
        |                                   |
        |                                   |
+--------------------+              +--------------------+
|--------------------|              |--------------------|
||  host 1          ||              || host 2           ||
||  IP 192.168.1.100||              || IP 192.168.1.101 ||
|--------------------|              |--------------------|
+--------------------+              +--------------------+

Answer 1

Just for future users coming around:

The issue was with a packet accelerator used in the embedded device that overrides linux routing tables (named SFE, provided by qualcomm). it simply ignored some of the more complex rules.

its open source and a patch was supplied.

good luck!