7

I have a Docker setup with one Django container and one nginx serving static files. I have nginx configured in the standard way:

upstream main_web {
    server web:8000;
}
server {
    location / {
        proxy_pass http://main_web;
        #...
    }

}

I use the /etc/hosts entry to get the IP address of the Django container.

When I restart the Django container, its IP address is updated, which reflects to the hosts file. But nginx keeps giving a 502 Bad Gateway error. Things work fine when I manually restart nginx.

Isn't there a way to tell nginx to resolve the IP again if it's not reachable?

user1496984
  • 281
  • 2
  • 3
  • 8

2 Answers2

7

When you use a variable to specify the domain name in the proxy_pass directive, NGINX re‑resolves the domain name when its TTL expires. You must include the resolver directive to explicitly specify the name server (NGINX does not refer to /etc/resolv.conf). More here

Using Docker, you can check your current dns resolver and add that to your nginx configuration. Check /etc/resolv.conf.

For example using docker-compose you can set the name of the service in your nginx vhost configuration. Asume we have this setup:

docker-compose.yml

version: "3"
services:
  my-backend-service: # we will use this name in our nginx vhost conf
    image: some-image
  lb:
    image: nginx
    volumes:
      - ".docker/etc/nginx/default.conf:/etc/nginx/conf.d/default.conf"
    ports:
      - 80:80

A few things about the above configuration, note that I do not add a container-name to the backend service, so I can scale it freely. Also pay attention to the service name, we will use that in Nginx vhost conf so it resolves to the instances ip via DNS. Finally, Nginx is listening on port 80 and the backend-services are listening on 8080 (as an example, could be anything)

default.conf

resolver 127.0.0.11 valid=10s; # 127.0.0.1 comes from /etc/resolv.conf on the nginx container

server {
    location / {
        set $backend_servers my-backend-service;
        proxy_pass http://$backend_servers:8080;
    }
}

To test this:

  • docker-compose up -d
  • docker-compose scale my-backend-service=5
  • docker-compose logs -f my-backend-service

Now you can start making requests to localhost and Nginx is going to use the DNS resolver you specify to get the ip addresses of the backend services, and will re-resolve the domain name when the TTL expires.

lloiacono
  • 193
  • 1
  • 7
  • 1
    This is nice. Questions: is `10s` a decent duration for the TTL? I guess if it is too short, it will happen too often and slow down *something* (what?), while if it is too long, there will be extended downtime. From my logs, I still have a [0-10] seconds interval where some requests are lost. Ain't there a way **to force the `resolver` to refresh itself**? This would make total sense... – Augustin Riedinger Mar 05 '20 at 13:56
  • I've stumbled upon same problem. It happened when I was deploying new version of application. Nginx upstream was using domain address like `php:9000`. On deploy, containers were recreated, but somehow nginx dns cache pointed to old IP address of `php` service. Workaround was to use static ip addresses for services inside `docker-compose.yml` file and point to IP in nginx configuration. I know it might not be possible to do in every situation (probably not when using swarm) but it fixed nginx 502 errors. – piotrekkr Oct 30 '20 at 06:29
0

See: http://tenzer.dk/nginx-with-dynamic-upstreams/ for a method to get nginx to re-resolve the IP using a variable in the proxy_pass setting.

It's also possible with ha_proxy 1.6 with its 'Server IP resolution using DNS at runtime' feature.

user1458424
  • 111
  • 1