In short, everything works until I try to access my apache server in a docker container from the internet. The incoming packets reach the docker container, but the outgoing packets get dropped by the NAT interface on docker-machine.
My setup is
- typical home LAN setup with a router connected to the internet
- osx machine running docker (ie - docker-machine aka boot2docker)
- a docker container running apache (exposing port 80 as 11111)
- I have apache running on osx (my host machine) as well which will serve as a troubleshooting tool.
- I added a bridged interface to docker-machine which connects docker-machine directly to my LAN and it gets a IP from my router.
- on my router (192.168.1.1) I've port forwarded 11111 to my docker-machine IP (192.168.1.102)
what I can do
- I can connect to apache from my host and from another computer on my
LAN (all 192.168.1.X) because of the bridge interface on
docker-machine. both 192.168.1.102:11111 and using
http://:11111 - so obviously my domain resolves it's IP properly and port forwarding on my router works fine.
- I can also easily access the web server on my host machine (osx) from the internet
what I've tried
- port forwarding on my host (osx) using pfctl
- using --net=host on my container
I've narrowed down the problem (using tcpdump and other experimentation) that when I connect to the web server in the container from a computer on my LAN the packets flow through the bridged interface (192.168.1.102) in docker-machine. When I connect from the internet the incoming packet flows through 192.168.1.102, but the return packet does not. Instead it goes through the NAT interface on docker-machine and gets dropped. I've proved this by doing "ifconfig eth0 down" on the NAT interface from inside docker-machine. Now obviously this screws up my docker-machine because I can no longer run docker commands and it kills my current ssh session. But when I did this, connectivity to the web server in the container from the internet works! So I proved that the return packet is the problem.
Now, can I use iptables inside docker-machine to route the packet properly so that it goes out the bridged interface instead of the NAT interface? I've tried it without success. Here is my iptables rules on docker-machine. I want to route any packets coming from my container with a source port of 11111 and route them to eth1 (the bridged interface).
sudo iptables -I FORWARD 1 -i docker0 -o eth1 -p tcp --syn --sport 11111 -m conntrack --ctstate NEW -j ACCEPT
docker@default:~$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
ACCEPT tcp -- anywhere anywhere tcp spt:11111 flags:FIN,SYN,RST,ACK/SYN ctstate NEW
DOCKER-ISOLATION all -- anywhere anywhere
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain DOCKER (1 references)
target prot opt source destination
Chain DOCKER-ISOLATION (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere
Am I going about this all wrong? Is there a standard way to set this up that I haven't explored?
