Docker Nginx reverse proxy configuration

Question

Context:

I'm fairly new to nginx, and figuring out how to incrementally make this work. I've worked around this problem for a while till now, where it's actually hindering development on one of my apps as a result of improper behavior (headers not being passed between each host, request bodies being empty, etc).

I have three apps in php-fpm containers, and an nginx server in its own container. The nginx container acts as a single access point for serving requests to the php-fpm containers based on the requested hostname (basic vhost configuration). Everything works fine when using try_files, but when switching to proxy_pass the problem presents itself.

The nginx configurations are nearly identical for each of the php-fpm hosts, with the exception of the server_name being changed to match whichever container the request is supposed to be processed by.

My research resources are linked at the bottom of the page.

Problem:

This is where I'm confused. Depending on what I set proxy_pass to will change the error I see in the nginx logs. Under no circumstance when using proxy_pass will requests actually complete successfully. Here are the various settings with their associated error:

Setting:

proxy_pass http://appX/; # upstream name

Error:

nginx | 2018/05/08 22:19:48 [error] 10224#10224: *8442 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 172.18.0.1, server: appX.local, request: "GET / HTTP/1.1", upstream: "http://172.18.0.3:9000/", host: "appX.local"
nginx | 172.18.0.1 - - [08/May/2018:22:19:48 +0000] "GET / HTTP/1.1" 502 568 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" "-"

Setting:

proxy_pass https://appX/; # upstream name

Error:

nginx | 172.18.0.1 - - [08/May/2018:22:20:50 +0000] "GET / HTTP/1.1" 502 568 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" "-"
nginx | 2018/05/08 22:20:50 [error] 13589#13589: *8446 peer closed connection in SSL handshake (104: Connection reset by peer) while SSL handshaking to upstream, client: 172.18.0.1, server: appX.local, request: "GET / HTTP/1.1", upstream: "https://172.18.0.3:9000/", host: "appX.local"

Setting:

proxy_pass http://appX.local/; # hostname

Error (Loops a bunch of times (logically, based on the redirect to https):

nginx | 172.18.0.6 - - [08/May/2018:22:21:49 +0000] "GET / HTTP/1.0" 301 185 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" "172.18.0.1"

Setting:

proxy_pass https://appX.local/; # hostname

Error (Loops a bunch of times and I don't know why):

nginx | 172.18.0.1 - - [08/May/2018:22:23:35 +0000] "GET / HTTP/1.1" 500 588 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" "-"

Setting:

proxy_pass https://appX:9000/;

Error:

nginx | 2018/05/08 22:29:26 [error] 14426#14426: *9563 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: appX.local, request: "GET / HTTP/1.1", upstream: "https://172.18.0.6:9000/", host: "appX.local"
nginx | 172.18.0.1 - - [08/May/2018:22:29:26 +0000] "GET / HTTP/1.1" 502 568 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" "-"

Setting:

proxy_pass https://appX.local:9000/;

Error:

nginx | 2018/05/08 22:27:39 [error] 14426#14426: *9559 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: appX.local, request: "GET / HTTP/1.1", upstream: "https://172.18.0.6:9000/", host: "appX.local"
nginx | 172.18.0.1 - - [08/May/2018:22:27:39 +0000] "GET / HTTP/1.1" 502 568 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" "-"

Attempted Remedies:

• I've tried changing the ports in the server directives to non-standard ports as well as appending a port number (both the one specified in the listen directive, as well as the one specified in the upstream directive).
• I've tried not using SSL and making this work over a standard HTTP connection.
• I've removed all of the other proxy settings and tried with just proxy_pass
• This works with try_files $uri $uri/ /index.php?$args, but that bypasses the proxying.
• I've tried to match the various configurations in my research resources and am experiencing the errors regardless of the changes I make.

Configurations:

Hostname are in my host's /etc/hostname file and all respond as to be expected.

nginx.conf

worker_processes 1;

daemon off;

events {
    worker_connections 1024;
}

error_log   /var/log/nginx/error.log warn;
pid         /var/run/nginx.pid;

http {
    default_type application/octet-stream;
    include /etc/nginx/conf/mime.types;

    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
    '$status $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for"';

    access_log /var/log/nginx/access.log main;

    sendfile on;

    keepalive_timeout 65;

    include /etc/nginx/conf.d/*.conf;
}

/etc/nginx/conf.d/appX.conf:

upstream appX {
        server appX:9000;
}

server {
        listen 80;
        listen [::]:80;
        server_name appX.local appX;
        return 301 https://$server_name$request_uri;
}

server {
        listen 443 ssl;
        listen [::]:443 ssl;
        server_tokens off;

        ssl on;
        ssl_certificate /etc/nginx/certs/appX.crt;
        ssl_certificate_key /etc/nginx/certs/appX.key;
        ssl_dhparam /etc/nginx/certs/dhparam.pem;

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;
        ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
        ssl_ecdh_curve secp384r1;
        ssl_session_cache shared:SSL:10m;
        ssl_session_tickets off;
        resolver 8.8.8.8 8.8.4.4 valid=300s;
        resolver_timeout 5s;

        server_name appX.local appX;

        root /var/www/appX;
        index index.php index.html;

        location / {
                proxy_read_timeout    90;
                proxy_connect_timeout 90;
                proxy_redirect        off;

                proxy_set_header                X-Real-IP $remote_addr;
                proxy_set_header                X-Scheme $scheme;
                proxy_set_header                X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header                X-Forwarded-Proto $scheme;
                proxy_set_header                X-Forwarded-Host $server_name;
                proxy_set_header                Host $host;
                proxy_set_header                X-Real-IP $remote_addr;
                proxy_set_header                X-Forwarded-Port 443;
                proxy_set_header                Authorization $http_authorization;
                proxy_pass_header               Authorization;
                proxy_next_upstream             error timeout invalid_header;

                proxy_hide_header               X-Powered-By;
                proxy_hide_header               X-Pingback;
                proxy_hide_header               Link;

                proxy_cache_bypass              $http_pragma $http_authorization;

                proxy_ssl_session_reuse off;
                proxy_ssl_server_name on;

                proxy_pass https://appX;
        }
}

docker-compose.yml

version: '3.3'
services:
  nginx:
    image: evild/alpine-nginx:1.9.15-openssl
    container_name: apps_nginx
    volumes:
      - ./app-one:/var/www/app-one/:ro
      - ./app-two:/var/www/app-two/:ro
      - ./app-three:/var/www/app-three/:ro
      - ./nginx/conf/nginx.conf:/etc/nginx/conf/default.conf:ro
      - ./nginx/conf.d:/etc/nginx/conf.d:ro
      - ./certs:/etc/nginx/certs
    ports:
      - 80:80
      - 443:443
    expose:
      - "80"
      - "443"
    depends_on:
      - php-app-one
      - php-app-two
    environment: 
      TZ: "America/Los_Angeles"
    networks:
      default:
        aliases:
          - app-one.local
          - app-two.local

  php-app-one:
    environment: 
      TZ: "America/Los_Angeles"
    image: joebubna/php
    container_name: apps_php_app-one
    restart: always
    volumes:
      - ./app-one:/var/www/app-one
    ports:
      - 9001:9000
    networks:
      - default

  php-app-two:
    environment: 
      TZ: "America/Los_Angeles"
    image: joebubna/php
    container_name: apps_php_app-two
    restart: always
    volumes:
      - ./app-two:/var/www/app-two
    ports:
      - 9000:9000
    networks:
      - default


  php-app-three:
    image: joebubna/php
    container_name: apps_php_app-three
    restart: always
    volumes:
      - ./app-three:/var/www/app-three
      - ./fastcgi_params:/var/www/fastcgi_params
    ports:
      - 9002:9000
    depends_on:
      - db
    environment:
      WORDPRESS_DB_HOST: db:3306
      WORDPRESS_DB_USER: root
      WORDPRESS_DB_PASSWORD: root
      WORDPRESS_DB_NAME: app_three
      TZ: "America/Los_Angeles"
    networks:
      - default

  db:
    image: mysql:5.6
    container_name: apps_mysql
    volumes:
      - db-data:/var/lib/mysql
      - ./mysql/my.cnf:/etc/mysql/conf.d/ZZ-apps.cnf:ro
    environment:
      MYSQL_ROOT_PASSWORD: root
      MYSQL_USER: user
      MYSQL_PASSWORD: password
      MYSQL_DATABASE: cora
      TZ: "America/Los_Angeles"
    ports:
      - 3306:3306
    expose:
      - "3306"
    networks:
      - default


volumes:
  db-data:

networks:
  default:
    driver: bridge

Any help would be greatly appreciated. I've spent quite a few hours on this today and I'm pretty well stuck as to what I've missed in all the reading and trial and error.

Research resources:

https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/

https://www.liaohuqiu.net/posts/nginx-proxy-pass/

https://gist.github.com/soheilhy/8b94347ff8336d971ad0

https://stackoverflow.com/questions/43500469/nginx-proxy-jwilder-nginx-proxy-connection-reset-by-peer-502-bad-gateway

https://groups.google.com/forum/#!topic/xnat_discussion/-S6eYsnwrXg

https://stackoverflow.com/questions/37346560/nginx-server-closes-ssl-connection-for-some-clients

https://stackoverflow.com/questions/38375588/nginx-reverse-proxy-to-heroku-fails-ssl-handshake

https://stackoverflow.com/questions/40580617/nginx-reverse-proxy-upstream-not-working

https://umbrella.cisco.com/blog/2015/11/03/lets-talk-about-proxies-pt-2-nginx-as-a-forward-http-proxy/

Use Nginx as Reverse Proxy for multiple servers

EDIT: Based on input from a couple kind individuals here, I have concluded this route has deviated too far from my original problem to warrant further pursuit. This can be chalked up to confusion and too many hours staring at the same problem. If a moderator wants to close this question, please feel free to do so.

Answer 1

The hostnames for docker are not specified in /etc/hostname or /etc/hosts. Docker has its own inter container communication. So when you refer to a container then you can use the service name that you specified in your docker-compose.yml to refer to the container. So at the moment you have the following containers:

1. nginx: you can refer to it as nginx as the hostname. You should be able to ping the container using the hostname nginx.
2. php-app-one: hostname: php-app-one
3. php-app-two
4. php-app-three
5. db

Btw if you want to specify a different hostname for the container then you can use this in your docker-compose.yml

container_name: 'some-container-name-different-to-the-name-of-service'

Getting nginx+php-fpm to work together, but each in their own container is pretty tough. Here is a github repo where I setup everything manually without docker-compose.yml. Nginx and php-fpm are each in their own container: https://github.com/timogoosen/mediawiki-dockerized/blob/master/nginx/sites-enabled/site.conf That is the nginx config. You will see the nginx container connects to php-fpm container using a hostname and port.

fastcgi_pass php-fpm-container:9000;

See the README, it is pretty descriptive: https://github.com/timogoosen/mediawiki-dockerized/tree/master/nginx

I suggest break the problem up into small parts as you have been doing. First get nginx and php-fpm each in their own container working then serve a random page that just contains:

<?php
phpinfo();
php>

Once you get that working then go from there.

Here are some links to help you: https://stackoverflow.com/questions/39901311/docker-ubuntu-bash-ping-command-not-found https://stackoverflow.com/questions/30172605/how-to-get-into-a-docker-container https://stackoverflow.com/questions/30151436/how-to-run-nginx-docker-container-with-custom-config