I have a server running Django on top of Gunicorn and Nginx, and I experienced a small amount of downtime last night. After restarting Nginx and Gunicorn the server went back to normal, but I can't figure out what caused the outage. Inspecting the log outlined where the problem happened: firstly, there were around 100 lines like this:
2014/03/04 15:48:47 [emerg] 21790#0: *19536658 posix_memalign(16, 4096) failed (12: Cannot allocate memory), client: xx.xx.xx.xx, server: www.mysite.com, request: "GET /static/images/loading.gif HTTP/1.1", host: "www.mysite.com"
Then the actual errors came. There were about 300 of these:
2014/03/04 15:49:04 [error] 21790#0: *19532341 connect() failed (110: Connection timed out) while connecting to upstream, client: xx.xx.xx.xx, server: www.mysite.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8081/", host: "www.mysite.com"
...and about 100 of these:
2014/03/04 15:51:32 [error] 21789#0: *19529583 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: xx.xx.xx.xx, server: www.mysite.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8081/", host: "www.mysite.com"
...along with a few of these peppered throughout the log:
2014/03/04 15:51:22 [emerg] 21791#0: *19539287 malloc(1024) failed (12: Cannot allocate memory) while waiting for request, client: xx.xx.xx.xx, server: 0.0.0.0:80
As for the Django error logs, I was getting a lot of errors with this message:
OperationalError: could not fork new process for connection: Cannot allocate memory
This looks like some kind of out-of-memory error, but at the time my swap was completely empty and I haven't seen anything in any of the logs to suggest any process was killed. Can anyone shed some light on what might be happening here?
