We have the following scenario, a docker host with two NICs in two subnets, ens8: IP 192.168.100.74/24 and ens9: IP 172.20.102.24/25, the default bridge docker0 IP 172.17.0.1/16 and a second bridge app_net IP 172.21.0.1/24. 192.168.100.1 is the default gateway on host (internet gateway).
The docker container have a fixed outgoing IP using SNAT, see Four ways to connect a docker container to a local network - Using NAT. This is done by two commands, assign target address to host interface ip addr add 192.168.100.234/24 dev ens8 and add iptables rule iptables -t nat -I POSTROUTING -s 172.21.0.4 -j SNAT --to-source 192.168.100.234. Containers are created using IPs from bridge app_net.
On the host I can either ping/traceroute IPs in subnet 192.168.100.x and in 172.20.102.x and internet e.g. google.com. Within a container, I can ping/traceroute IPs in subnet 192.168.100.x, I can also ping e.g. google.com, but I cannot ping anything in 172.20.102.x, except 172.20.102.24. See the following output.
# traceroute -I 192.168.100.10
traceroute to 192.168.100.10 (192.168.100.10), 30 hops max, 46 byte packets
1 172.21.0.1 (172.21.0.1) 0.004 ms 0.003 ms 0.002 ms
2 192.168.100.10 (192.168.100.10) 0.180 ms 0.158 ms 0.028 ms
# traceroute -I google.com
traceroute to google.com (216.58.214.46), 30 hops max, 46 byte packets
1 172.21.0.1 (172.21.0.1) 0.004 ms 0.013 ms 0.002 ms
2 fritz.box (192.168.100.1) 2.144 ms 3.114 ms 2.697 ms
3 server.provider.net (xx.yy.250.5) 6.947 ms 7.042 ms 4.305 ms
4 etc...
# traceroute -I 172.20.102.24
traceroute to 172.20.102.24 (172.20.102.24), 30 hops max, 46 byte packets
1 172.20.102.24 (172.20.102.24) 0.004 ms 0.003 ms 0.002 ms
# traceroute -I 172.20.102.9
traceroute to 172.20.102.9 (172.20.102.9), 30 hops max, 46 byte packets
1 172.21.0.1 (172.21.0.1) 0.004 ms 0.003 ms 0.002 ms
2 * * *
3 * * * (etc. no response)
If I do not add the iptables SNAT rule, containers can ping/traceroute everything, also machines on subnet 172.20.102.x.
I found an article about multiple NICs and docker(#) with a stripped down solution on serverFault, but neither this, nor a deep look into advanced docker networking(#) gave me a clue, why I cannot reach other machines in the ens9 subnet, except the network card itself from within a docker container.
Am I missing something, or is this network scenario just wrong?
(#) Not allowed to post more than two links because of reputation, but the serverFault link together with Google "Docker over multiple network interfaces" should help to find the links.
