3

My current setup is nginx as a load balancer, in front of a bunch of upstream servers.

I want to be able to deploy with no change in user experience. This means no 502s and no increase page times.

My understanding is that when nginx -s reload is issued, nginx will gracefully spawn new threads to handle new connections for the new configuration, but will still finish handling requests that were in-flight at the time of reload. Moreover, I have read somewhere that marking an upstream as down in the upstream block, and reloading, will take that upstream from the rotation, but will still finish any in-flight connections to it.

Is any/all of this correct?

The current plan is to have a small process in the nginx host, that will basically mutate nginx configuration and issue these graceful reloads when asked to do so through an internal REST API.

My deployment tool will then connect that process and add/remove my upstreams from nginx config in sequence, after graceful shutdown and boot up/warm-up for each upstream is complete.

Does this make sense? How are other people doing this? Are there any already available tools for this?

I haven't had a lot of luck searching for info on this...

Márcio Martins
  • 141
  • 1
  • 7
  • [Nginx Plus](https://www.nginx.com/resources/deployment-guides/load-balance-apache-tomcat/) supports dynamic reconfiguration, but that costs money. I wonder if you could do DNS based load balancing and have two Nginx load balancers, but you'd have to test your clients will switch to the other server if one goes down. – Tim Aug 29 '17 at 20:31

1 Answers1

1

Yes, you are correct in thinking that nginx continues handling the current requests when reloading, and will start processing new requests with the new configuration.

http://nginx.org/en/docs/beginners_guide.html#control

Old worker processes, receiving a command to shut down, stop accepting new connections and continue to service current requests until all such requests are serviced. After that, the old worker processes exit.

Marking an upstream as down doesn't significantly differ from removing it from the configuration. It will not accept new requests in both cases.

We frequently reload the nginx config (generated using cog https://nedbatchelder.com/code/cog/) and it does not cause any downtime as far as the proxy is concerned, for upstream locations. As long as the upstream itself is ready to process requests, there is no downtime.

Vince
  • 164
  • 1
  • 5
  • I took the time and tested all my assumptions. Everything is working well. Basically my upstream block is its own file which in turn is managed by a small server that takes REST calls to add or remove/suspend servers from it. Once done it sends the reload signal and everything works like a charm with no impact at all for customers :) – Márcio Martins Sep 01 '17 at 14:05