Puma Socket connection time out or unavailable under high loads

Question

I have a Rails 4.2 app that is currently running on an Ubuntu server with Nginx and Passenger and it get's a lot of traffic which Passenger doesn't handle very well (very often processes hang).

I decided to replace Passenger with Puma as I did with other apps on another server where things improved drastically but with this one as soon as i deployed the new version running on Puma, i noticed problems started happening, getting a lot of 502 bad gateway errors and looking in the logs i saw a lot of either of these errors:

puma.sock failed (11: Resource temporarily unavailable)

[error] 6658#6658: *5788 upstream timed out (110: Connection timed out) while reading response header from upstream

After googling around and ended up trying several things including the following Sysctl tweaks:

/etc/sysctl.conf

# Increase number of incoming connections
net.core.somaxconn = 65535

# Increase number of incoming connections backlog
net.core.netdev_max_backlog = 65536

Then reloaded with sudo sysctl -p

I've also tweaked the following Nginx configs:

/etc/nginx/nginx.conf

user www-data;
worker_processes auto;
pid /run/nginx.pid;
worker_rlimit_nofile 400000;

events {
        worker_connections 10000;
        use epoll;
        multi_accept on;
}
http {
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        keepalive_requests 100000;
        server_tokens off;

        server_names_hash_bucket_size 256;

[...]
}

Here's my Puma config:

workers 3
preload_app!

threads 1, 6

app_dir = File.expand_path("../..", __FILE__)
shared_dir = "#{app_dir}/shared"

# Default to production
rails_env = ENV['RAILS_ENV'] || "production"
environment rails_env

# Set up socket location
bind "unix://#{shared_dir}/sockets/puma.sock"

# Logging
if rails_env == "production"
  stdout_redirect "#{shared_dir}/log/puma.stdout.log", "#{shared_dir}/log/puma.stderr.log", true
end

# Set master PID and state locations
pidfile "#{shared_dir}/pids/puma.pid"
state_path "#{shared_dir}/pids/puma.state"

on_worker_boot do
  #reconnect to mongo
  Mongoid::Clients.clients.each do |name, client|
    client.close
    client.reconnect
  end

  #reconnect to redis
  $redis.redis.client.reconnect
end

before_fork do
  Mongoid.disconnect_clients
end

I've also tried specifying the backlog value when biding to the socket like so:

bind "unix://#{shared_dir}/sockets/puma.sock?backlog=1024"

Here's the nginx config for the app:

upstream pumamyapp {
  server unix:///var/www/myapp/shared/sockets/puma.sock;
}

server {
  listen   80;

  listen 443 ssl; # managed by Certbot
  ssl_certificate /etc/letsencrypt/live/myapp/fullchain.pem; # managed by Certbot
  ssl_certificate_key /etc/letsencrypt/live/myapp/privkey.pem; # managed by Certbot
  include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

  server_name www.myapp.com;
  rewrite ^(.*) https://myapp.com$1 permanent;
}

server {
  listen   80;

  listen 443 ssl; # managed by Certbot
  ssl_certificate /etc/letsencrypt/live/myapp/fullchain.pem; # managed by Certbot
  ssl_certificate_key /etc/letsencrypt/live/myapp/privkey.pem; # managed by Certbot
  include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

  root /var/www/myapp/public;
  server_name myapp.com;

  if ($ssl_protocol = "") {
    rewrite     ^   https://$server_name$request_uri? permanent;
  }

  client_max_body_size 100M;

  location ~* ^/assets/ {
    # Per RFC2616 - 1 year maximum expiry
    # http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
    expires 1y;
    add_header Cache-Control public;

    # Some browsers still send conditional-GET requests if there's a
    # Last-Modified header or an ETag header even if they haven't
    # reached the expiry date sent in the Expires header.
    add_header Last-Modified "";
    add_header ETag "";
    break;
  }

  location /cgi-bin {
    return 404;
  }

  location /setup.cgi {
    return 404;
  }

  location / {
    try_files $uri @app;
  }

  location @app {
    proxy_pass http://pumamyapp;

    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header Host $http_host;

    proxy_headers_hash_max_size 512;
    proxy_headers_hash_bucket_size 128;

    proxy_redirect off;
  }

}

I had to rollback to the previous version that runs on passenger because the site was unusable, any idea what is wrong and how can I make it right?