Why is apache logging an internal IP when using an external domain?

Question

I have a Raspberry Pi running which we're using in the office for small test projects we don't want on our main development server. It is running apache. The DNS is handled via Cloudflare, but in DNS Only mode. Currently, no IP restriction is in place while I test. The A record for mysite.domain.tld is pointing to my static IP address 123.123.123.123 and only a standard business fibre router/modem sit between the Internet and the Pi.

When I visit mysite.domain.tld from, for instance, my phone without wifi, the cell carrier IP shows up. When I wget from a remote server, its' IP shows up in the logs. Everything works as expected.

However, when I go to mysite.domain.tld from within the same network that the Pi is located in, apache logs the router gateway IP 192.168.1.1. I'd expect to see my public IP address, because my connection to the domain name resolves, via Cloudflare, to the public IP. But instead I see a local network IP in the logs.

There is nothing set in /etc/hosts (I'm on macOS) and on the router, only port-forwarding for connections from port 443 are router to the Pi on the same port - nothing regarding the domain name is reference anywhere. When I ping mysite.domain.tld it gives me the Cloudflare IP address, which is what I'd expect.

It seems like somewhere along the chain, the fact that my IP address and the public IP address of the Pi are being matched so it's overriding the IP with the internal gateway IP. What is actually happening here? I don't mind per se, I just want to make sure that I can rely on the fact that 192.168.* IPs can be trusted when setting up IP restriction on the firewall.

Note: CF-Connecting-IP and similar headers are not sent by Cloudflare here, I presume that only happens when not in DNS Only mode. And it seems to be only when I use the same network connection that the Pi is using.

Answer 1

Your router is running Linux, and this behaviour is easy to implement on any stock Linux distro. I can guess what rules must be present in your router's firewall for it to work like this. But be aware this is only a speculation, we don't know how exactly rules look in reality.

When you forwarded a port to your webserver, it added a specific DNAT rule, which likely looks like this:

iptables -t nat -A PREROUTING -p tcp -d <your-external-address> --dport 443 -j DNAT --to-destination <raspberry-pi-address>

In words, this means: "before deciding whether this packet is destined to the device or must be forwarded, check if the packet destination address is your external address and destination port is 443. If it matches, change destination address to the raspberry pi's LAN address". Notice, this rule doesn't filter by the interface.

Also it definitely has the SNAT-type rule (for providing an access to the internet for the LAN), probably it looks like this:

iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -j MASQUERADE

In other words, after the router decides where the packet must go, before sending packet, it changes its source address to the whichever address an outbound interface has (if the packet was sent from the LAN). Again, nothing is filtered by an interface.

Now consider your HTTPS connection. You are specifying the host name in the browser, right? You set it up so it is resolved to your router's external address, which becomes the destination address. Your source address is inside LAN. So, it seems both of these rules apply to packets of your connection.

Processing them, router first encounters DNAT rule, checks the destination address and port and decides to change destination address to raspberry pi's one. Then it finds out the interface the packet must go out is LAN one. Then it checks partially translated packet against second rule, and finds out the source address is from LAN. So it replaces source address of the packet with the address of LAN interface, 192.168.1.1. This is what your raspberry pi sees.

The NAT operation is stateful, i.e. it also maintains a table record which says which was replaced with what and how to detect subsequent forward and reply packets, so it translates all of them correctly. Yes, it turns out Linux can do DNAT and SNAT at the same time within the same stream.

Can you trust this behaviour? I don't know. If the router's firmware was open source, we had a chance to check. Without source, we can't be sure. This is always the case when the source is closed, that's why closed source products must be avoided if you are concerned about security.