How can I get nginx not to override x-forwarded-for when proxying?

Question

I have an nginx server behind a load balancer, the nginx server passes requests on to a variety of services, but in this case a docker container running apache. The load balancer sets an X-Forwarded-For correctly, but by the time it gets to the docker container, X-Forwarded-For has been set to the LB IP.

I have this in nginx config:

/etc/nginx/conf.d/real_ip.conf
set_real_ip_from {{LB IP}};
real_ip_header X-Real-IP;
real_ip_recursive on;

and this is the virtualhost:

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name *.domain domain;
    include /etc/nginx/snippets/domain_ssl.conf;

  add_header X-Nginx-Debug "hi";

  proxy_pass_request_headers on;

  location    / {
    proxy_pass_request_headers on;
    proxy_pass  http://container-php;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Remote-Addr $remote_addr;
    proxy_set_header X-Real-IP $http_x_real_ip;
    proxy_set_header X-Header-Test "Hello World - $http_x_forwarded_for";
    proxy_set_header X-Forwarded-Proto $scheme;
  }
}

But what I get from the container is:

array(19) {
  ["Connection"]=>
  string(7) "upgrade"
  ["Host"]=>
  string(19) "domain"
  ["X-Forwarded-For"]=>
  string(12) "{{LB IP}}"
  ["X-Header-Test"]=>
  string(13) "Hello World -"
  ["X-Forwarded-Proto"]=>
  string(5) "https"
  ["cache-control"]=>
  string(9) "max-age=0"
  ["sec-ch-ua"]=>
  string(64) "" Not;A Brand";v="99", "Google Chrome";v="97", "Chromium";v="97""
  ["sec-ch-ua-mobile"]=>
  string(2) "?0"
  ["sec-ch-ua-platform"]=>
  string(9) ""Windows""
  ["upgrade-insecure-requests"]=>
  string(1) "1"
  ["user-agent"]=>
  string(114) "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36"
  ["accept"]=>
  string(135) "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
  ["sec-fetch-site"]=>
  string(4) "none"
  ["sec-fetch-mode"]=>
  string(8) "navigate"
  ["sec-fetch-user"]=>
  string(2) "?1"
  ["sec-fetch-dest"]=>
  string(8) "document"
  ["accept-encoding"]=>
  string(17) "gzip, deflate, br"
  ["accept-language"]=>
  string(26) "en-GB,en-US;q=0.9,en;q=0.8"
}

Notably X-Real-IP, X-Fowarded-For don't seem to be set, nor does remote_addr. Files served directly from nginx have x-forwarded-for set properly, so the LB is sending down the right header.

Have I missed a step?

Answer 1

I think the problem is in your real ip configuration.

set_real_ip_from {{LB IP}};

real_ip_header X-Real-IP;

real_ip_recursive on;

When real_ip_header should be (in your case) set to X-Forwarded-For. I'll assume your X-Forwarded-For header from LB looks like this :

X-Forwarded-For: {{Original client ip}}, {{LB ip}}

So when you set real_ip_header (The header used to replace client ip) to X-Forwarded-For it will match the original client ip. The Original client should now be under the variable $realip_remote_addr, which you can address to proxy_set_header X-Forwarded-For :

proxy_set_header X-Forwarded-For $realip_remote_addr

Please let me know if I have been of any help !

Answer 2

This is the statement overriding the X-Forwarded-For header:

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

Assuming you want to preserve the original client IP in that header, you should write:

proxy_set_header X-Forwarded-For $remote_addr;

Answer 3

From your description your problem isn't directly related to nginx but with apache.

Depending on your configuration your traffic flow might look something like:

outside -> nginx -> apache -> php (running as fpm)

or

outside -> nginx -> apache + php module

You should look into the apache config to make sure that out if apache isn't discarding the x-forwarded-for and x-real-ip headers when passing the request to PHP.

If your traffic flow is in line with the first example, you have apache proxying the request also to php which could lead to a lot more issues if nginx and apache aren't "in sync".

If you were handling the php proxying with nginx alone you would just need to add the following to your php location config:

location ~ \.php$ {
        #...other rules
        fastcgi_param REMOTE_ADDR $http_x_real_ip;
        #...other rules
}

This way, PHP's REMOTE_ADDR would be correctly set.

---

In regards to your nginx config if your nginx {{LB IP}} is static, you can set it directly in the config.

/etc/nginx/conf.d/real_ip.conf
set_real_ip_from {{LB IP}};
real_ip_recursive on;

You don't need real_ip_header X-Real-IP; in your config file. That will override the set_real_ip_from directive.

Keep in mind that you need to have the real_ip module enabled in nginx in order for this to work.

Answer 4

I eventually solved this in a way that works, but doesn't actually solve the root problem. As far as I can tell, the upstream LB is setting "X-Forwarded-For" to the remote address, and nginx's magic attempting set that correctly for the reverse proxy was always getting it wrong and setting it to the LB address or an empty string.

Instead, I switched from HTTP to PROXY protocol for the LB->Server bit, set the following in /etc/nginx/proxy_params:

proxy_set_header X-Real-IP       $proxy_protocol_addr;
proxy_set_header X-Forwarded-For $proxy_protocol_addr;

and this in conf.d:

 /etc/nginx/conf.d/real_ip.conf
set_real_ip_from {{ LB_IP }};
real_ip_header proxy_protocol;

adapted from this:

https://www.x33u.org/docs/kubernetes-stuff/hetzner-loadbalancer-setup/

and now everything works.