1
My AWS EC2 instance, running Node, needs to pass data to my Redis cache. This has been working fine for months. But, for the passed few days, it's broken.
All requests either timeout after approx 60 seconds, or I get a 502 error.
I added extra logging to the node function. This revealed that it all breaks down when the service trying to talk to Redis.
I have rebooted the redis node several times, but that did not work. I rebuilt the Cloudformation stack several times, but that did not work. i have terminated and created new EC2 instances - same effect.
I have checked the following:
Security Groups checked: all good
IAM permissions checked: all good
IAM roles checked: all good
Redis endpoint URL: correct
Access key/secret checked: all good
I looked at the nginx logs and found this telling bit of info:
2019/01/11 09:24:33 [error] 3826#0: *6732 upstream prematurely closed connection while reading response header from upstream, client: , server: my.server.url, request: “GET /path/of/request HTTP/1.1”, upstream: “http://ip.ad.dr.ess:PORT/path/of/request”, host: “my.server.url”
I have two nginx configs. The parent looks like this:
# For more information on configuration, see: # * Official English Documentation: http://nginx.org/en/docs/ # * Official Russian Documentation: http://nginx.org/ru/docs/ user nginx; worker_processes 1; error_log /var/log/nginx/error.log; #error_log /var/log/nginx/error.log notice; #error_log /var/log/nginx/error.log info; pid /var/run/nginx.pid; events { worker_connections 1024; } http { include /etc/nginx/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" - $request_time X-Forwarded-For=$http_x_forwarded_for Host=$host'; log_format KVP 'ip="$remote_addr" time="$time_local" request="$request" ' 'status_code=$status request_time=$request_time ' 'host="$http_host" ' 'body_bytes_sent=$body_bytes_sent referrer="$http_referer" user_agent="$http_user_agent" remote_user="$remote_user"'; access_log /var/log/nginx/access.log main; error_log /var/log/nginx/error.log; #sendfile on; #tcp_nopush on; #keepalive_timeout 0; keepalive_timeout 300s; #gzip on; # Load modular configuration files from the /etc/path/to/conf.d directory. # See http://nginx.org/en/docs/ngx_core_module.html#include # for more information. include /etc/path/to/*.conf; index index.html index.htm; }
And the following one is included:
proxy_read_timeout 300s; limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s; # A virtual host using mix of IP-, name-, and port-based configuration # upstream NAME_REMOVED { server xxx.x.x.x:; } server { listen 80; server_name ; real_ip_header X-Forwarded-For; # internal addresses in TEST set_real_ip_from xx.xxx.0.0/16; # internal addresses in PRODUCTION set_real_ip_from xx.xxx.0.0/16; gzip on; gzip_comp_level 6; gzip_vary on; gzip_min_length 1000; gzip_proxied any; gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript; gzip_buffers 16 8k; client_max_body_size 100M; location / { proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $remote_addr; proxy_set_header Host $http_host; proxy_set_header X-NginX-Proxy true; proxy_read_timeout 300s; proxy_send_timeout 300s; proxy_connect_timeout 300s; proxy_pass http://app; proxy_redirect off; limit_req zone=api burst=10; } location ~ /.git/ { deny all; } }
The timeouts shown above were added by me to try to fix this issue, but didn't have the desired effect.