Persistent 502s from AWS ALB

Question

This is our architecture: Cloudflare -> ALB 1 -> Nginx API Gateway -> ALB 2 -> (Nginx Sidecar -> Application)

The application and the sidecar are on the same box and communicate via unix domain socket.

We see a steady but small stream of HTTP 502s from ALB 2. As far as we can tell these requests do no make it to the application. From analyzing the packet capture data, our team figured that the sidecar is returning a connection reset response to ALB 2 resulting in the 502s from the ALB.

We have made sure that the keepalive_timeout on our sidecar nginx conf is greater than the ALB's idle timeout (90s).

ALB 2's listening rule is to listen at HTTPS 443 and forward to the target group.

Relevant config from Nginx API Gateway:

sendfile on;
  tcp_nopush on;
  tcp_nodelay on;

  server_tokens off;
  more_clear_headers Server;

  underscores_in_headers on;
  proxy_buffer_size 128k;
  proxy_buffers 4 256k;
  proxy_busy_buffers_size 256k;
  large_client_header_buffers 4 512k;
  client_max_body_size 50M;

  keepalive_timeout 120s;
  client_header_timeout 120s;

Relevant config from the sidecar:

 sendfile on;
  tcp_nopush on;
  tcp_nodelay on;

  server_tokens off;

  underscores_in_headers      on;
  proxy_buffer_size           128k;
  proxy_buffers               4 256k;
  proxy_busy_buffers_size     256k;
  large_client_header_buffers 4 512k;
  client_max_body_size        50M;

  keepalive_timeout 650s;
  client_header_timeout 650s;
  keepalive_requests 10000;
  lingering_timeout 650s;

  upstream app {
    server unix:/var/tmp/puma.sock fail_timeout=0;
  }

  server {
    listen 8000;

    location / {
      proxy_pass http://app;
      proxy_buffering off;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Request-Start "t=${msec}";
      proxy_set_header Host $http_host;
    }
  }

The application is a Rails app running on Puma. Relevant Puma config:

bind("unix:///var/tmp/puma.sock?backlog=3")

We are pretty perplexed about the source of the 502s. ;(

Persistent 502s from AWS ALB

0 Answers0