nginx reverse stream proxy with multiple ports to the same server

Question

I'm trying to use nginx as a reverse proxy to two different servers. The servers require the use of client-side certificates for authentication, which means nginx is configured as a stream proxy leveraging the map $ssl_preread_server_name for SNI inspection to send to the correct server.

This works great for the pair of servers it's hosting now. Both listen on 443 but provide completely different services, but the redirection via SNI is working great.

The trouble is that one of the servers also uses port 9997 for communication (TLS) and we need to add more of these into the mix. Currently we're just hard-coding the traffic in nginx to the one server that uses 9997. This wont work as we move forward and have additional servers hosting content on 9997

How can I configure nginx to stream both 443 and 9997 to the box that needs those communications, while also continuing to send 443 to the other server when needed?

It needs to be dynamic so that the traffic is sent to the RIGHT server.

Here's the config that works now (some info redacted):

#user  nobody;
worker_processes  1;

error_log   /var/log/nginx/error.log;
#pid        logs/nginx.pid;


events {
    worker_connections  1024;
}

stream {

    map $ssl_preread_server_name $upstream {
        server1.domain.com server1;
        server2.domain.com server2;
    }

    server {
        listen 443;
        proxy_pass $upstream;

        ssl_preread on;
    }

    server {
        listen 9997;
        proxy_pass 1.2.3.4:9997;
    }


    upstream server1 {
        server 1.2.3.4:443;
    }

    upstream server2 {
        server 1.2.3.5:443;
    }

}

score 3 · Accepted Answer · edited Sep 12 '18 at 10:09

3

Below config should work for you

stream {

    map $ssl_preread_server_name:$server_port $upstream {
      server1.domain.com:443 server1;
      server2.domain.com:443 server2;
      server1.domain.com:9997 server3;
    }

    server {
      listen 443;
      proxy_pass $upstream;

      ssl_preread on;
    }

    server {
      listen 9997;
      proxy_pass $upstream;
      ssl_preread on;
    }


    upstream server1 {
      server 1.2.3.4:443;
    }

    upstream server2 {
      server 1.2.3.5:443;
    }
    upstream server3 {
      server 1.2.3.4:9997;
    }
}

edited Sep 12 '18 at 10:09

Altenrion

103
5

answered Nov 29 '17 at 09:21

Harrison Hao

46
2

1

I had same issue with you. After tried many days, it finally works out for me by adding a $server_port. – Harrison Hao Nov 29 '17 at 09:23

Danny Apostolov · Answer 2 · 2022-02-26T07:48:02.007

Here is a possible solution for using nginx as a router without upstream definitions. upstream can be undesirable when the server definitions include named hosts because nginx will fail to start if a named host could not be resolved (which could be the case with docker networks). The following configuration assumes that only HTTPS traffic will be routed and redirects all HTTP traffic to HTTPS.

This configuration is designed for nginx running in a docker instance and sharing a network with other docker services. dockerservice1 and dockerservice2 are expected to listen on port 443. The resolver 127.0.0.11 ipv6=off valid=1s; line enables host name resolution by the docker's local DNS resolver (127.0.0.11), so that dockerservice1 and dockerservice2 can be resolved to ips from the docker internal network. With this configuration nginx will always start, regardless of the up status of the referenced docker servces/hosts.

The configuration for logging can be omitted but is useful for diagnostic purposes.

# /etc/nginx/nginx.conf from the nginx docker instance

user nginx;
worker_processes auto;
pid /run/nginx.pid;
error_log /var/log/nginx/error.log;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 1024;
}

http {
        server {
                listen 80 default_server;
                server_name _;
                return 301 https://$host$request_uri;
        }

        log_format basic
                '$time_local router:RD $status $request_time client: $remote_addr '
                'http://$host$request_uri -> https://$host$request_uri';

        access_log /var/log/nginx/access.log basic;
        error_log /var/log/nginx/error.log;
}

stream {
        map $ssl_preread_server_name:$server_port $upstream {

                hostnames;

                mydomain.com:443 dockerservice1;
                www.mydomain.com:443 dockerservice1;
                test.mydomain.com:443 dockerservice2;
                
                default dockerservice1;
        }

        server {
                listen 443;
                resolver 127.0.0.11 ipv6=off valid=1s;
                proxy_pass $upstream:443;
                ssl_preread on;
        }
        log_format basic
                '$time_local router:RT $status $session_time client: $remote_addr '
                '$ssl_preread_server_name:$server_port -> $upstream ($upstream_addr) '
                'bytes from/to client $protocol $bytes_sent $bytes_received '
                'bytes from/to upstream $upstream_bytes_sent/$upstream_bytes_received '
                'upstream time: $upstream_connect_time';

        access_log /var/log/nginx/access.log basic;
        error_log /var/log/nginx/error.log;
}

NOTE: with Docker version 20.10.12, build e91ed57, the DNS resolver at 127.0.0.11 seems to work unreliably if there are a docker service and a docker host name with the same name, on different docker projects/docker-compose.yml files (and presumably on the same network), e.g.

# project1/docker-compose.yml
version: "3"
services:
  sharedName:
    container_name: sharedName
    hostname: sharedName
    networks:
      - default
      - external
    ...

networks:
  external:
    name: someNetwork

# project2/docker-compose.yml
version: "3"
services:
  sharedName:
    container_name: OTHERContainerName
    hostname: OTHERContainerName
    networks:
      - default
      - external
    ...

networks:
  external:
    name: someNetwork

The expected behavior in the above case would be for the host name sharedName to always resolve to the ip of the container created from project1/docker-compose.yml. The observed behavior is that sharedName is randomly resolved to the ip of either container sharedName or OTHERContainerName. This is also valid for a larger number of projects/containers.

A possible solution seems to be to enforce unique host names that have no duplicates among service names.

nginx reverse stream proxy with multiple ports to the same server

2 Answers2