Nginx v Redis: Upstream Prematurely Closed Connection

Question

My AWS EC2 instance, running Node, needs to pass data to my Redis cache. This has been working fine for months. But, for the passed few days, it's broken.

All requests either timeout after approx 60 seconds, or I get a 502 error.

I added extra logging to the node function. This revealed that it all breaks down when the service trying to talk to Redis.

I have rebooted the redis node several times, but that did not work. I rebuilt the Cloudformation stack several times, but that did not work. i have terminated and created new EC2 instances - same effect.

I have checked the following:

Security Groups checked: all good
IAM permissions checked: all good
IAM roles checked: all good Redis endpoint URL: correct
Access key/secret checked: all good

I looked at the nginx logs and found this telling bit of info:

2019/01/11 09:24:33 [error] 3826#0: *6732 upstream prematurely closed connection while reading response header from upstream, client: , server: my.server.url, request: “GET /path/of/request HTTP/1.1”, upstream: “http://ip.ad.dr.ess:PORT/path/of/request”, host: “my.server.url”

I have two nginx configs. The parent looks like this:

# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log;
#error_log  /var/log/nginx/error.log  notice;
#error_log  /var/log/nginx/error.log  info;

pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" - $request_time X-Forwarded-For=$http_x_forwarded_for Host=$host';

    log_format KVP 'ip="$remote_addr" time="$time_local" request="$request" '
                    'status_code=$status request_time=$request_time '
                    'host="$http_host" '
                    'body_bytes_sent=$body_bytes_sent referrer="$http_referer" user_agent="$http_user_agent" remote_user="$remote_user"';

    access_log /var/log/nginx/access.log main;
    error_log /var/log/nginx/error.log;

    #sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  300s;

    #gzip  on;

    # Load modular configuration files from the /etc/path/to/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/path/to/*.conf;

    index   index.html index.htm;

}

And the following one is included:

proxy_read_timeout 300s;

limit_req_zone $binary_remote_addr  zone=api:10m rate=10r/s;


# A virtual host using mix of IP-, name-, and port-based configuration
#

upstream NAME_REMOVED {
    server xxx.x.x.x:;
}

server {
    listen       80;

    server_name ;

    real_ip_header X-Forwarded-For;
    # internal addresses in TEST
    set_real_ip_from xx.xxx.0.0/16;
    # internal addresses in PRODUCTION
    set_real_ip_from xx.xxx.0.0/16;

    gzip on;
    gzip_comp_level 6;
    gzip_vary on;
    gzip_min_length  1000;
    gzip_proxied any;
    gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript;
    gzip_buffers 16 8k;
    client_max_body_size 100M;

    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Host $http_host;
        proxy_set_header X-NginX-Proxy true;

        proxy_read_timeout 300s;
        proxy_send_timeout 300s;
        proxy_connect_timeout 300s;

        proxy_pass http://app;
        proxy_redirect off;

        limit_req zone=api burst=10;
    }

    location ~ /.git/ {
        deny all;
    }
}

The timeouts shown above were added by me to try to fix this issue, but didn't have the desired effect.