We have a situation where a site starts to serve a 502 Bad Gateway but doesn't seem to recover after the upstream servers rebound. The nginx server is setup to proxy/load balance requests for two upstream servers. It looks like the database server will start to get a high load average, causing the web servers (upstream servers) to serve content slowly and timeout (according to the nginx server). Nginx will then serve a 502, which makes sense given the situation.
What is strange though is that it seems that nginx doesn't pick up on when the web servers rebound and we must restart nginx to get it to start serving the site again. Is there a good way to fix this? I'm looking through the proxy settings currently to see if there is something to set but not having much luck finding anything.
Looking at nginx logs we see entries like (just pulling three examples of errors in this timeframe):
2013/06/12 13:53:40 [error] 29840#0: *258391 upstream timed out (110: Connection timed out) while reading response header from upstream, client: n.n.n.n, server: www.example.org, request: "GET / HTTP/1.1", upstream: "http://n.n.n.n:80/", host: "www.example.org"
2013/06/12 13:54:11 [error] 29840#0: *261105 no live upstreams while connecting to upstream, client: n.n.n.n, server: www.example.org, request: "GET /HTTP/1.1", upstream: "http://example_rack/", host: "www.example.org"
2013/06/12 13:54:46 [alert] 29840#0: *261470 stalled cache updating, error:0 while closing request, client: n.n.n.n, server: n.n.n.n:80