4

Similar questions have been asked before here and here but none of them matched or solved the issue I am having. After a few hours of desperate problem solving I found a solution that was so unexpected yet simple that I wanted to share it in Q&A style.

Background

A company has two servers: a reverse proxy (Ubuntu 20.04 + Nginx 1.17) and an upstream app server (also Ubuntu 20.04 + Nginx 1.17). The reverse proxy handles various subdomains e.g. http://demo.example.com and maps them to a directory at the upstream server e.g. http://12.12.12.12:8000/demo/ by using proxy_pass directive.

Then, a client makes an HTTP request where the path points to a directory but has no trailing slash e.g. http://demo.example.com/items. The default Nginx behavior is to make 301 redirection to the same URL with the trailing slash added e.g. http://demo.example.com/items/ as stated in location docs.

If the 301 redirection happens at the upstream app server then the normal behavior is that the redirection location path becomes http://12.12.12.12:8000/demo/items/. This is not a problem because by default, the reverse proxy will replace 12.12.12.12:8000/demo/ part with the original host and thus the client receives a proper 301 redirection to demo.example.com/items/. This is what I expected to happen.

Problem

However, with the configuration below, the redirection location the client receives becomes http://demo.example.com:8000/demo/items/ instead. It seems like the reverse proxy only partially rewrites the redirection URL.

Site config at the reverse proxy:

server {
  listen 80;
  listen [::]:80;

  index index.html index.htm;

  server_name demo.example.com;

  location / {
    proxy_pass http://12.12.12.12:8000/demo/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }
}

Site config at the app server:

server {
  listen 8000;
  listen [::]:8000;

  root /var/www/example.com;
  index index.html index.htm;

  location / {
    try_files $uri $uri/ =404;
  }
}

What I tried

To solve the issue but with no success, I tried various combinations and values of proxy_redirect, absolute_redirect, server_name_in_redirect, and port_in_redirect directives as suggested in the similar questions. Either the directives had no effect or only partly fixed the problem like hiding the port.

Some combinations partly disabled the redirection and allowed client to access the items page (at items/index.html) without trailing slash http://demo.example.com/items. This however broke relative links to images as the browser tried to find them at /product.jpg instead of /items/product.jpg. The redirection was truly necessary.

I tried the following rewrite directive: rewrite ^/(.*) /demo/$1 break; with proxy_pass http://12.12.12.12:8000/. It worked as expected but did not affect the redirection.

I also tried replacing the try_files line with try_files $uri $uri/index.html $uri/ =404 as suggested here but almost obviously with no success. The redirection from http://demo.example.com/items stubbornly kept its location at http://demo.example.com:8000/demo/items/.

I feel it should not be this complex. What am I missing?

Akseli Palén
  • 161
  • 1
  • 6

1 Answers1

2

A solution, and possibly the solution, came after multiple hard hours of trying to understand the problem and exact behavior of Nginx's redirection mechanism. The solution was rather simple really:

Remove the proxy_set_header Host $host from the reverse proxy config:

server {
  ...

  location / {
    proxy_pass http://12.12.12.12:8000/demo/;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }
}

Or alternatively, use proper proxy_redirect value:

server {
  ...
  location / {
    proxy_pass http://12.12.12.12:8000/demo/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_redirect http://demo.example.com:8000/demo/ http://demo.example.com/;
  }
}

The reason behind this is that proxy_set_header Host $host delegates the upstream server the original hostname. The nginx at the upstream server applies this host to redirections. Therefore the redirection location is already http://demo.example.com:8000/demo/items/ when dispatched by the upstream server and before the reverse proxy modifies it in any way. Understanding this was the first half of the solution.

The second half deals with how proxy_pass and proxy_redirect affect the redirection. If proxy_redirect is not defined it still has default value default. Its default and rather hidden behavior is to take the redirections from the upstream and replace exact matches of the value of proxy_pass with the original host. In other words, given proxy_pass http://abc, only those redirects where Location header contains http://abc are modified. Finally, if no match is found, the Location header is left unaltered.

Therefore, the original issue was caused because the upstream server already replaced its IP address with the host header provided by the reverse proxy. That caused mismatch between the value of proxy_pass and the redirection location, thus preventing the activation of the default proxy_redirect directive. Furthermore, the unaltered redirect was passed to client and bounced the client to the invalid URL.

I would say this was a rather complex interaction between Host header, proxy_pass, default proxy_redirect, and built-in 301 redirections. Fortunately, it got solved!

Akseli Palén
  • 161
  • 1
  • 6