NGINX double proxy not allowing websocket connections

Question

I have two nginx proxies setup on my machine, one to unwrapp SSL and the other to do application-specific proxying (only the second one is version controlled). When I only had one proxy I was able to make successful websocket connections, but after moving to two all websocket upgrade requests are responded to with a 502 Bad Gateway error. I can confirm that normal http/https requests are working with my double proxy setup. Here's my current config.

Proxy 1

server {

        # SSL configuration
        #
        # listen 443 ssl default_server;
        # listen [::]:443 ssl default_server;
        #
        # Note: You should disable gzip for SSL traffic.
        # See: https://bugs.debian.org/773332
        #
        # Read up on ssl_ciphers to ensure a secure configuration.
        # See: https://bugs.debian.org/765782
        #
        # Self signed certs generated by the ssl-cert package
        # Don't use them in a production server!
        #
        # include snippets/snakeoil.conf;

        server_name staging.ambitx.io;

        location / {
                proxy_pass http://127.0.0.1:81;
                include proxy_params;
        }

    listen [::]:443 ssl ipv6only=on; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/staging.ambitx.io/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/staging.ambitx.io/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}
server {
    if ($host = staging.ambitx.io) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

        listen 80 default_server;
        listen [::]:80 default_server;

        server_name staging.ambitx.io;
    return 404; # managed by Certbot

}

proxy_params

proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

Proxy 2

(running on docker with port 81 on the host machine bound to port 80 on the container)

resolver 127.0.0.11 ipv6=off;

server {
  listen       80;
  listen  [::]:80;

  location / {
    root /var/www/staticfiles;
    index  index.html index.htm;
    try_files $uri /index.html =404;
  }
  
  location /ws {
    access_log off;

    proxy_pass       http://wsserver;

    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
  }

  location /api {
    proxy_pass http://apiserver;
  }
}

I originally had the statements below in the location /ws block of Proxy 2's config...

proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

but I removed them because they would be overwriting the headers set by Proxy 1.

Any ideas? Please let me know if you need more information.

score 0 · Accepted Answer · answered Jun 01 '22 at 02:28

Of course you should not drop the WebSocket proxying support from any of your intermediate proxy servers. WebSocket proxying is a very special task, to allow protocol switching and keep WebSocket connection established and alive you should return its support to your first nginx proxy as well:

...
location / {
    proxy_pass http://127.0.0.1:81;
    include proxy_params;
}
location /ws {
    proxy_pass http://127.0.0.1:81;
    include proxy_params;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
}
...

Removing all the three proxy_set_header directives from your second nginx proxy was also a bad idea. While X-Real-IP and X-Forwarded-For headers will be passed to your WebSocket app exactly as being set by your first proxy, the Host header is a special one. Unless being set explicitly, it will be passed equal to the upstream name used in the proxy_pass directive, i.e. Host: wsserver. As you can read from the proxy_set_header directive documentation:

By default, only two fields are redefined:
proxy_set_header Host $proxy_host;
proxy_set_header Connection close;

(the second one will surely break any attempt to establish a WebSocket connection as well). So to keep the original Host header from the client request (usually a good idea, you can read more about this header here), return the

proxy_pass Host $http_host;

line to both location / { ... } and location /ws { ... } of your second nginx proxy configuration.

This worked for me, thanks. For the purpose of keeping proxy 1 agnostic of proxy 2 (since proxy 1 isn't currently version controlled), I decided to use the "more sophisticated example" (the second one) listed on this website: https://nginx.org/en/docs/http/websocket.html instead and just add it to the `location / { ... }` block in proxy 1. That way if I change the websocket url but forget about proxy 1 it won't break. The only downside to this that I can think of is that it won't be able to handle http/2.0, but my server wasn't setup to handle it anyways. — Simon Richard, Jun 02 '22 at 23:41
That's sound reasonable, I prefer those way of proxying WebSocket too. About HTTP/2, you may found [this](https://serverfault.com/questions/765258/use-http-2-0-between-nginx-reverse-proxy-and-backend-webserver) thread quite interesting. — Ivan Shatsky, Jun 03 '22 at 11:18

NGINX double proxy not allowing websocket connections

1 Answers1