iptables and Nginx reverse proxy for localhost

Question

Summary

I use Nginx as a reverse proxy and in particular its functionality proxy_bind in order to send the real client IP to the backend. I followed this documentation from nginx.com.
It works well, both when the HTTP server is hosted on another machine or directly on the one of the reverse proxy, except in the second case if the HTTP server listens on localhost.

=> I can't get this configuration works:

# /etc/nginx/conf.d/nginx-revprox-test.conf
server {
    listen 80;
    location / {
        proxy_pass  http://127.0.0.1:8080;
        proxy_bind  $remote_addr transparent;
    }
}

Details

Hypotheses:

• IP addresses:
  • Reverse proxy/Local backend: 192.168.1.90 (eth0) & 172.17.0.1 (eth1)
  • External backend: 172.17.0.2
  • Client: 192.168.1.91
• Ports:
  • frontend: 80 (no TLS)
  • backends: 8080

• Case n°1: The HTTP server is hosted on the external backend (http://172.17.0.2:8080).

  # /etc/nginx/conf.d/nginx-revprox-test.conf
  server {
      listen 192.168.1.90:80;
      location / {
          proxy_pass  http://172.17.0.2:8080;
          proxy_bind  $remote_addr transparent;
      }
  }
  

  The manual says we have to configure iptables consequently:

  iptables -t mangle -A PREROUTING -p tcp -s 172.17.0.2 --sport 8080 -j MARK --set-xmark 0x1/0xffffffff
  ip rule add fwmark 1 lookup 100
  ip route add local 0.0.0.0/0 dev lo table 100
  

  => It works fine.

• Case n°2: The HTTP server is hosted on the same machine than the reverse proxy and listens on eth1 (http://172.17.0.1:8080).

  # /etc/nginx/conf.d/nginx-revprox-test.conf
  server {
      listen 192.168.1.90:80;
      location / {
          proxy_pass  http://172.17.0.1:8080;
          proxy_bind  $remote_addr transparent;
      }
  }
  

  According to this answer and this diagram, only OUTPUT and POSTROUTING apply to locally generated packets. We need to change our iptables rule consequently:

  iptables -t mangle -A OUTPUT -p tcp -s 172.17.0.2 --sport 8080 -j MARK --set-xmark 0x1/0xffffffff
  

  => It works too.

• Case n°3: The HTTP server is hosted locally but listen on lo (http://127.0.0.1:8080).

  # /etc/nginx/conf.d/nginx-revprox-test.conf
  server {
      listen 192.168.1.90:80;
      location / {
          proxy_pass  http://127.0.0.1:8080;
          proxy_bind  $remote_addr transparent;
      }
  }
  

  => [!] I can't get it works, even with an OUTPUT rule in iptables.
  curl http://192.168.1.90:80 does not work from the client (it only works from the server but that's not the point).

I missed something about iptables. Could you help me?

Answer 1

You can avoid Netfilter marking for case 3 - the necessary filtering is also supported directly by ip rule. The following will route all traffic from the backend server to the nginx frontend:

# add a new routing table that routes all packets back to this server
ip route add local 0.0.0.0/0 dev lo table 100

# add a rule for choosing that routing table:
ip rule add from 127.0.0.1/32 ipproto 6 sport 8080 iif lo lookup 100
# ^ individual parts:
# from 127.0.0.1/32 - select just one loopback IP, the one with nginx on it
# ipproto 6  - select just TCP
# sport 8080 - select just traffic from the backend HTTP server port
# iif lo     - select just traffic originating from this machine
# lookup 100 - route these matching packets via routing table 100

I found this by trial-and-error with the help of ip rule manpage and the packet flow diagram (1, 2).

---

You could also use Netfilter marking, but I would recommend the solution above as it avoids enabling route_localnet (which can cause security issues). For completeness, this is how to make it work:

1. Enable routing on local interfaces (as per https://unix.stackexchange.com/a/158256/72419):

   sysctl -w net.ipv4.conf.all.route_localnet=0
   

2. Configure the routing table:

   ip rule add fwmark 1 lookup 100
   ip route add local 0.0.0.0/0 dev lo table 100
   

3. Let Netfilter mark the packet in the OUTPUT chain:

   iptables -t mangle -A OUTPUT -p tcp -s 127.0.0.1 --sport 8080 -j MARK --set-xmark 0x1/0xffffffff
   

   The OUTPUT chain is critical - locally produced packets do not go through the PREROUTING table.

You might be able to avoid the security issues associated with route_localnet, though - https://github.com/yrutschle/sslh/blob/master/doc/tproxy.md#transparent-proxy-to-one-host fixes this with

iptables -t raw -A PREROUTING ! -i lo -d 127.0.0.0/8 -j DROP
iptables -t mangle -A POSTROUTING ! -o lo -s 127.0.0.0/8 -j DROP