iptables: containers NAT and external IP

Question

I've a Debian 10 server (public IP 85.xxx.xxx.xxx at enp6s0) running a bunch of LXC containers on a network bridge cbr0. I've the following iptables rules in place so the containers can communicate to the outside world:

iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -i cbr0 -j ACCEPT
iptables -A FORWARD -i cbr0 -o enp6s0 -j ACCEPT
iptables -A FORWARD -i enp6s0 -o cbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -t nat -A POSTROUTING -o enp6s0 -j MASQUERADE

I've created a container (10.0.0.1) that has nginx and redirected incoming http traffic to that container:

iptables -t nat -A PREROUTING -i enp6s0 -p tcp --dport 80 -j DNAT --to-destination 10.0.0.1:80
iptables -A FORWARD -i enp6s0 -o cbr0 -p tcp --dport 80 -j ACCEPT

Everything works as expected, I'm able to browse the website on the container etc.

Now I've created a second container (10.0.0.2) that needs fetch some data from the nginx container. If I use the internal 10.0.0.1 IP for the communication my second container is able to wget pages. However if I try to use the server's external IP:

root@second-container:~# nc example.com 80
example.com [85.xxx.xxx.xxx] 80 (http) : Connection refused

My current workaround is to manually add entries at /etc/hosts for all the domains hosted by nginx so they will resolve to 10.0.0.1 instead of their public IP.

What rules do I need to also allow communication using the external server IP?

Thank you.

Here is an additional diagram of the issue:

Have you redirected incoming IP traffic from OUTSIDE your network to reach the container, IE is your router directing any incoming WAN traffic it receives on port 80 to the 10.0.0.1 lxc, and can you browse it externally? But you can't reach the 10.0.0.1 container internally by using the external IP address? If so then the issue likely lies with your router rather than your lxc config (which already appears to have port 80 open). Your router may be configured to only forward traffic from the WAN interface, or it's possible it just doesn't support NAT loopback. — Will, Nov 03 '20 at 21:26
I get your point however there's no router involved. The server has a public IP provided by the ISP's optical network terminal directly. As I described the second block of rules redirects the port 80 of the server to the port 80 of the 10.0.0.1 container. And yes I can browse the websites externally. Unfortunately the second container can't reach the first container using the public IP. — TCB13, Nov 03 '20 at 21:37
This is the common problem about hairpin routing. The proper solution is that you set up the local DNS with the local address so that you do not wast routing resources with NAT, which is resource intensive. The local DNS should point to the local address so that traffic remain local and can be bridged rather than routed. — Ron Maupin, Nov 03 '20 at 22:07
@RonMaupin as I expected. I'm currently doing that "proper solutions" with `/etc/hosts` inside the container, but is there a way to use the external address? I've some specific domains and situations where I can't predict this kinds of connections will happen and I would like to have a global fallback for them. — TCB13, Nov 03 '20 at 22:21
There's a router somewhere, even if you don't have a physical router that your WAN connects into. Where have you set up the rule to forward port 80 to the internal IP address of your 10.0.0.1 container? That's your router, because it's routing traffic from the WAN IP to the local IP. Either it's set to only route traffic that comes in on the WAN connection to that IP, or it just isn't capable of NAT loopback. — Will, Nov 03 '20 at 22:34
@Will the router is the server itself if I follow your logic. Anyways RonMaupin named the issue properly. I was able to find someone running into the exact same problem here https://newspaint.wordpress.com/2018/09/13/hairpin-for-lxc-containers-using-iptables/ — TCB13, Nov 03 '20 at 23:00
@RonMaupin can you post your comment as answer so I can accept it? — TCB13, Nov 08 '20 at 11:17

score 2 · Accepted Answer · answered Nov 08 '20 at 14:56

This is the common problem about hairpin routing. The proper solution is that you set up the local DNS with the local address so that you do not waste routing resources with NAT, which is resource intensive. The local DNS should point to the local address so that traffic remains local and can be bridged rather than routed.

score 0 · Answer 2 · answered Nov 08 '20 at 18:28

This is a common problem called a hairpin NAT. The solution I generally prefer is to just add SNAT (source address translation). Therefore the device that already does DNAT at the same time would do SNAT for the same packet.

The overhead is miniscule. And, most importantly, you leave your DNS pure, simple, predictable.

iptables: containers NAT and external IP

2 Answers2