How accessing a NATed, virtualized (LXC) server over public IP from client itself on a bridged network?

Question

Setup

    ____________________________             ____________________________ 
    | Host                     |             | Client                   |
    | Public IP:   66.66.66.66 |             |                          |
    | Internal IP: 10.0.3.1    | <---------> | Internal IP: 10.0.3.192  |
    ----------------------------             ----------------------------

Host with public IP (=66.66.66.66)
Linux Container (LXC), called client in the rest of the post, started with veth network option on which a webserver is running.
On the host, the firewall is configured with ufw and the respective ports are opened and forwarded to the client, this works.

iptables port forwarding in the /etc/ufw/before.rules:

# nat Table rules
*nat
:PREROUTING ACCEPT [0:0]

-A PREROUTING -i eth0 -p tcp --dport 80 -j DNAT --to-destination 10.0.3.192:80
-A PREROUTING -i eth0 -p tcp --dport 443 -j DNAT --to-destination 10.0.3.192:443
# don't delete the 'COMMIT' line or these rules won't be processed
COMMIT

/etc/network/interfaces on host:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
    address 66.66.66.66
    netmask 255.255.255.0
    network 66.66.66.0
    broadcast 66.66.66.255
    gateway 66.66.66.1
    # dns-* options are implemented by the resolvconf package, if installed
    dns-nameservers 8.8.8.8

/etc/network/interfaces on client:

# The loopback network interface
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address 10.0.3.192
    netmask 255.255.255.0
    gateway 10.0.3.1
    broadcast 10.0.3.255

Port scan from client

Starting Nmap 5.21 ( http://nmap.org ) at 2014-04-18 18:46 CEST
Nmap scan report for 66.66.66.66
Host is up (0.00011s latency).
PORT    STATE  SERVICE
80/tcp  closed http
443/tcp closed https

Port scan from "outside"

Starting Nmap 6.40 ( http://nmap.org ) at 2014-04-18 19:54 CEST
Nmap scan report for 66.66.66.66
Host is up (0.41s latency).
PORT    STATE SERVICE
80/tcp  open  http
443/tcp open  https

Nmap done: 1 IP address (1 host up) scanned in 1.28 seconds

iptables configuration:

$ iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DNAT       tcp  --  anywhere             anywhere             tcp dpt:http to:10.0.3.192:80
DNAT       tcp  --  anywhere             anywhere             tcp dpt:https to:10.0.3.192:443
DNAT       tcp  --  anywhere             anywhere             tcp dpt:43211 to:10.0.3.192:22
DNAT       tcp  --  anywhere             anywhere             tcp dpt:http-alt to:10.0.3.192:8080

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  tcp  --  anywhere             10.0.3.192           tcp dpt:http
MASQUERADE  tcp  --  anywhere             10.0.3.192           tcp dpt:https
MASQUERADE  all  --  10.0.3.0/24         !10.0.3.0/24

Versions

Ubuntu 12.04 (host as well as client)
LXC (1.0.3-0ubuntu1~ubuntu12.04.1~ppa1)

Desired result

I want to access the webserver on port 80 on the client from the client itself over the public IP, e.g.

$ curl http://66.66.66.66
curl: (7) couldn't connect to host

should give the same result as

$ curl http://10.0.3.192
<html>.....

Strangely $ ping 66.66.66.66 from the client inside the private network works though.

What I tried

I see that this problem is very much related to the so called Hairpin NAT / Loopback NAT, though I could not configure the masquerading with the following rules, such that it works (in /etc/ufw/before.rules after the PREROUTING entries):

# nat Table rules
*nat
:POSTROUTING ACCEPT [0:0]

-A POSTROUTING -o eth0 -d 10.0.3.192 -p tcp --dport 80 -j MASQUERADE
-A POSTROUTING -o eth0 -d 10.0.3.192 -p tcp --dport 443 -j MASQUERADE

# don't delete the 'COMMIT' line or these nat table rules won't be processed
COMMIT

My guess to the solution is that the bridged network or just my incapability of adapting the MASQUERADE rule correctly to my setup is the problem. Any suggestions and comments, also to the setup in general are appreciated.

Related questions or articles, but not helping

LXC networking with public IP
About different network modes of LXC: http://containerops.org/2013/11/19/lxc-networking/
https://coderwall.com/p/k0gutq
Using NAT for container with private IPs (explicitly written that it will not work from inside the private network): https://openvz.org/Using_NAT_for_container_with_private_IPs
Clean LXC + NAT configuration: https://coderwall.com/p/k0gutq
iptables - how to access local server using external ip (disadvise of doing so, suggest setting up DNS or editing host file, but based on FQDN): http://www.tomshardware.co.uk/forum/12532-42-iptables-access-local-server-external
Is NAT Loopback on my router a security problem? (Gives SNAT rule,: https://security.stackexchange.com/questions/16343/is-nat-loopback-on-my-router-a-security-problem
NAT loopback using iptables: http://for-invent.com/nat-loopback-using-iptables/

A.B · Answer 1 · 2014-04-24T20:48:00.153

(I wrote first that's it's a routed network not a bridged network. Now I see it's using LXC, so I don't really know. but if PREROUTING works already, I sure hope what I wrote below will work)

You're using the PREROUTING chain, this chain alters (about to be) routed packets only, that is packets coming from somewhere else. Packets generated on Host itself are not routed (they're just well... output, as any host can do), so this chain never receives curl's packets. curl just tried to connect on Host as usual. There's an other chain to catch locally generated packets: OUTPUT.

So, you also duplicate the DNAT rules to the (-t nat) OUTPUT chain, with some modifications: OUTPUT doesn't want input interface. You replace -i eth0 with -o lo ! -s 127.0.0.0/8 or just -d 66.66.66.66 or something else but you need some restriction else any web request to anywhere will go to Client. First example is independent of Host's IP(s), second one is shorter, as you like. It's not a typo, if you connect to 66.66.66.66 from itself, it's a local packet, so it goes through the lo interface. But then because the 127.0.0.1-like destinations can't be rerouted ( curl http://127.0.0.1/ would timeout instead of saying couldn't connect ) 127.0.0.0/8 is put as an exception.

And that's it. Everything else is as usual handled by conntrack. You should remove the 2 port specific MASQUERADE rules. They're not needed and they may even interfere (I think if they work, the result is that Client will always see Host as source on the web server)

so, the short answer is (using -t nat):

-A PREROUTING -i eth0 -p tcp --dport 80 -j DNAT --to-destination 10.0.3.192
-A PREROUTING -i eth0 -p tcp --dport 443 -j DNAT --to-destination 10.0.3.192
-A OUTPUT -o lo ! -s 127.0.0.0/8 -p tcp --dport 80 -j DNAT --to-destination 10.0.3.192
-A OUTPUT -o lo ! -s 127.0.0.0/8 -p tcp --dport 443 -j DNAT --to-destination 10.0.3.192

Unfortunately this did not work for me. I got the error "iptables: Invalid argument. Run `dmesg' for more information.", and when running dmesg I see "[3181260.407457] x_tables: ip_tables: DNAT target: only valid in nat table, not filter". So it seems DNAT is not allowed in an OUTPUT filter. Any ideas? — Matthew O'Riordan, Jun 24 '15 at 10:44