I've two interfaces, let's say eth0
and eth0.4000
as vlan.
Both have a default-gateway.
Everything works as expected when a process listens on the interface directly.
But not for hostPort
bindings of Kubernetes.
vlan.gw-mac > eth0-mac, ethertype 802.1Q (0x8100), length 78: vlan 4000, p 0, ethertype IPv4 (0x0800), clientIP.38712 > vlanIP.80: Flags [S]
eth0-mac > eth0.gw-mac, ethertype IPv4 (0x0800), length 74: vlanIP.80 > clientIP.38712: Flags [S.]
The SYN comes from vlan.gateway
, gets forwarded to the Container but the answer SYN-ACK leaves the stack through eth0.gw
and not the correct vlan.gw
but tcpdump shows that the sourceIP is vlanIP
.
The routing tables looking good:
# ip route get to <clientIP> from <vlanIP> dev eth0.4000
<clientIP> from <vlanIP> via <vlan.gw> dev eth0.4000 table 1 uid 0
The hostPort mapping gets created through the CNI-Plugin portmap which uses DNAT and SNAT (details linked). So the gateway lookup happens to early. When I manually add a route from the container-ip to lookup table 1 it works using the vlan interface but breaks eth0.
So the questions is - what has to be done that the routing happens after NAT replaced the container-ip with the interface-ip?