0

Apache is on 127.0.100.2 and Nginx is on 127.0.100.3, with Traffic Control performing DNAT from the former to the latter (for source 127.0.0.1) and SNAT from the latter to the former (for destination 127.0.0.1):

tc qdisc add dev lo root handle 1:0 prio
tc filter add dev lo parent 1:0 protocol ip prio 1 u32 \
  match ip src 127.0.0.1 match ip dst 127.0.100.2 \
  action nat ingress 127.0.100.2 127.0.100.3 
tc filter add dev lo parent 1:0 protocol ip prio 1 u32 \
  match ip src 127.0.100.3 match ip dst 127.0.0.1 \
  action nat egress 127.0.100.3 127.0.100.2

Cache hits work, misses don't, which I understand: Upon a cache miss, content is delivered by Apache,† but NAT will prevent that. What configuration should be used?

Nginx configuration file:

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=my_cache:10m max_size=1g 
    inactive=60m use_temp_path=off;
server {
  listen 127.0.100.3;
  location / {
    proxy_cache my_cache;
    proxy_pass http://127.0.100.2;
  }
}

† Extra question: Technically speaking, how does Nginx handle a cache miss? Wireshark doesn't reveal anything obvious: I see SYN; SYN,ACK; ACK; HTTP HEAD; ACK between 127.0.0.1 and 127.0.100.3, then a similar TCP handshake between 127.0.0.1 and 127.0.100.2 followed by HTTP GET and payload delivery. Presumably, the final ACK between 127.0.0.1 and 127.0.100.3 is doing something clever, but what?

user2768
  • 41
  • 5

2 Answers2

2

What configuration should be used?

Generally it depends on your very circumstances. But obviously if you don't want NAT to jump in between Nginx and Apache, Nginx should connect to Apache using different dest. or src. address. For e. g., you can assign second IP for Apache to listen on, and point proxy_pass there, or make Nginx to connect using a different outgoing IP. That's kinda blunt but should do.

Since tc's NAT is stateless other options (like handling TCP's SYN/ACKs) are kinda cumbersome.

poige
  • 9,171
  • 2
  • 24
  • 50
  • If I understand correctly, your solution essential defines a special path for Apache-Nginx traffic, is that right? (Rather than using different destination and source addresses, different ports would suffice.) I'd favour a cleaner, lighter, simpler solution, albeit, perhaps no such solution exists. – user2768 Mar 24 '20 at 15:59
  • I don't immediately see how Nginx connecting _using a different outgoing IP_ helps, since Nginx connecting to Apache isn't problematic: For cache misses, 127.0.0.1 requests content directly from Apache, which is why my configuration doesn't work. – user2768 Mar 24 '20 at 16:03
  • If by "cache misses" you mean Nginx's cache misses, it means Nginx can't fetch requested obj from own cache and therefore Nginx relays such a request to Apache. It's not `127.0.0.1` who's doing that _but a process_ instead. Nginx. Its outgoing req. source IP can be found out with `ip ro get …dest…`, unless limited by `proxy_bind` or with other enforcement. – poige Mar 24 '20 at 16:52
  • as to the ports instead of IPs — whatever, it's up to you — as declared "Generally it depends on your very circumstances" – poige Mar 24 '20 at 16:54
  • I expected Nginx to relay the request to Apache, but that doesn't seem to be the case, as I noted at the end of my question. (If Nginx were to relay the request, then that'd surely suffice as a solution.) – user2768 Mar 24 '20 at 17:06
  • 1
    :) you're making up some funny theories, but in reality request is handled in turns starting from the 1st process which owns corresponding TCP socket. – poige Mar 24 '20 at 17:49
  • I have a Wireshark trace (when `tc` rules aren't in place): I see SYN; SYN,ACK; ACK; HTTP HEAD; ACK between 127.0.0.1 and Nginx, then a similar TCP handshake between 127.0.0.1 and Apache followed by HTTP GET and payload delivery. That's certainly not what I (nor you, it seems) expected to see! I tried to explain it. For instance, maybe Nginx spoofed the source. But that doesn't make sense (127.0.0.1 didn't initiate, so it wouldn't work). – user2768 Mar 24 '20 at 18:16
  • That second handshake is initiated by Nginx, as explained in [another answer](https://serverfault.com/a/1009920/564695) – user2768 Mar 31 '20 at 16:42
  • obviously it's, that's why I suggested using different IP for that communication. Another answer tells you to do exactly that, but just later than mine did. – poige Apr 01 '20 at 01:17
  • I also told you "about Its outgoing req. source IP can be found out with ip ro get …dest…, unless limited by `proxy_bind`", how that "another answer" is telling somewhat more? I'd say something ether-fishy is going on here. – poige Apr 01 '20 at 02:24
  • I couldn't understand nor use your answer. You did say _make Nginx to connect using a different outgoing IP_, but didn't explain Nginx was using 127.0.0.1 *and* 127.0.100.1 (due to default behaviour), nor that `proxy_bind` would solve this. (I don't know why you suggested _you can assign second IP for Apache to listen on, and point proxy_pass there_, since that isn't necessary.) Of course, some will understand you and be able to use your answer. I couldn't. That's why I favour the other answer. There's nothing _ether-fishy...going on here_. Thanks for your support – user2768 Apr 01 '20 at 06:28
  • Again. I gave you exact command to find out which source IP would be used, citing it: "Its outgoing req. source IP can be found out with `ip ro get …dest…`" I've flagged this, I want moderators to step in and make it fair, because now it's not. – poige Apr 01 '20 at 11:01
1

I think you're expecting the nginx connection to apache to come from 127.0.100.3 because you have configured nginx to listen on that IP.

The address bound for listening (incoming) sockets doesn't apply to outgoing connections. With your current configuration, the connection from nginx to apache should come from 127.0.0.1 as that's the primary IP bound to the loopback interface and thus the default for outgoing connections using that interface.

You can use proxy_bind to override that default and bind a specific source address for nginx outgoing connections.Relevant documentation on docs.nginx.com

I believe you'll achieve the behavior you expect by adding proxy_bind 127.0.100.3; to your location / { ... } configuration.

Also, I'd suggest you consider using 127.0.100.4 for your proxy_bind, to ease your filtering rules.

etherfish
  • 1,747
  • 10
  • 12