first of all, this is not an every-day routing issue. The setup is fairly complex, so let me state it before.
I got a router with, lets keep it simple, 3 interfaces. eth0, eth1, eth2. eth2 is used for pppoe. eth0 & eth1 have the clients.
Okay so far so good, all basic.. Now here comes the tricky thing: I create a bunch of macvlan-interfaces on top of eth0 and eth1, the name schema is:
g1eth0 : g1 for gate1, eth0 indicates on what physical interface its laying on
This I got for every uplink I provide, lets say 3, 1 pppoe and 2 VPNs. These are then merged into bridges named after the gate.
So far we got these interfaces:
<iface>:<description>
eth0 : our 1st subnet is here
eth1 : our 2nd subnet is here
eth2 : our pppoe is hooked here
ppp0 : our pppoe uplink
tap0 : our vpn1 uplink
tap1 : our vpn2 uplink
g1eth0 : advertised gate over uplink1 on clients in eth0
g1eth1 : advertised gate over uplink1 on clients in eth1
g2eth0 : advertised gate over uplink2 on clients in eth0
g3eth1 : advertised gate over uplink3 on clients in eth1
gate1 : bridge containing g1eth0 and g1eth1
gate2 : bridge containing g2eth0
gate3 : bridge containing g3eth1
As I said, a bunch of interfaces... Notice that an uplink can be advertised over several physical interfaces, thats why we got the bridges.
Alright now lets take a look at the routing rules:
32763: from all fwmark 0x3 lookup 202
32764: from all fwmark 0x2 lookup 201
32765: from all fwmark 0x1 lookup 200
Okay this is not so spectacular, obviously, it only checks what FWMARK a pkg has and pushes it to the according table.
The routing tables:
200:
default via 1.2.3.4 dev ppp0 src 4.3.2.1
201:
default via 5.6.7.8 dev tap0 src 8.7.6.5
202:
default via 9.10.11.12 dev tap1 src 12.11.10.9
Okay the IPs are just for to fill the gaps, you should be familiar with the syntax ;)
Right now we got the routing tables, routing rules and the interfaces - but we're missing out the pkg marking, so this is being done in iptables:
iptables -t mangle -A PREROUTING -i gate1 -s 10.0.0.0/16 -j MARK --set-xmark 0x1/0xffffffff
iptables -t mangle -A PREROUTING -i gate2 -s 10.0.0.0/16 -j MARK --set-xmark 0x2/0xffffffff
iptables -t mangle -A PREROUTING -i gate3 -s 10.0.0.0/16 -j MARK --set-xmark 0x3/0xffffffff
Okay for explanation, we mark all pkgs comming in our bridges with the right value for the routing rules.
Now I also had to do some tweaks in arp_announce and arp_ignore so that the right MAC is being advertised for the g*eth*-interfaces. This post is getting rather full, so I will skip describing it, both are set to 2.
The filter:FORWARD chain is empty for now, it just logs the pkgs it gets.
Now NAT'ing:
iptables -t nat -A POSTROUTING -s 10.0.0.0/16 -j MASQUERADE.
All default policies for iptables are ACCEPT.
tcpdump shows that the incomming pkgs are directed to the right MAC according to the g*eth*-interfaces.
mangle:PREROUTING counters for the rules increment as they should.
ip_forward verified to be 1.
filter:FORWARD counters are NOT incrementing.
I got LOG rules in every chain, but the pkgs seem to vanish once passed the mangle:PREROUTING.
Any ideas why?
Addition I: I placed a TRACE rule in PREROUTING as the comment suggested me, ironically it doesn't show any of the pings my clients are running.
Addition II: After some playing around with the rules,tracing,promisc,... I noticed that I see the data getting in on ethX but not on gateX. Seems like the brigde-interface is just dropping it, no wonder the kernel cant get it into forward.
Why does my bridge-interface do this?
bridge name bridge id STP enabled interfaces
gate1 8000.dead000200b5 no g1eth0
g1eth1
