I've been running into an issue where a Rails app server (nginx/puma) and a PostgreSQL data server communicate consistently when on the same VLAN on our DMZ, yet when the database is isolated to another VLAN and the app server remains on the DMZ, a user hitting the app server only eventually runs into 504 (Gateway Timeout) errors from nginx. These eventual timeouts do not seem related to actual end-user usage of the apps (potential under-allotment of connections, used-up connections etc.) as I have noticed this issue can happen on a weekend, when almost certainly no users are in the system. From the first 504 gateway timeout, all subsequent requests to the server error out with more 504 gateway timeout pages. I would say that this is due to a suboptimal connection configuration on my part, yet when both servers are on the same DMZ and not connecting through a firewall the whole thing works. When the pair is in the "bad" configuration, connections work but only for a variable period of time, usually an hour or so.
Puma configuration is as follows:
#!/usr/bin/env puma
directory "/var/www/my_app/current"
preload_app!
environment "production"
daemonize true
pidfile "/var/www/my_app/shared/tmp/pids/my_app.pid"
state_path "/var/www/my_app/shared/puma/my_app.state"
stdout_redirect '/var/www/my_app/shared/log/production.log', '/var/www/my_app/shared/log/production_err.log', false
threads 0, 16
bind "unix:///var/www/my_app/shared/tmp/sockets/my_app.sock"
workers 8
on_worker_boot do
require "active_record"
ActiveRecord::Base.connection.disconnect! rescue ActiveRecord::ConnectionNotEstablished
ActiveRecord::Base.establish_connection(YAML.load_file("/var/www/my_app/current/config/database.yml")["production"])
end
before_fork do
ActiveRecord::Base.connection.disconnect! rescue ActiveRecord::ConnectionNotEstablished
end
Nginx configuration is as follows:
upstream my_app {
server unix:///var/www/my_app/current/tmp/sockets/my_app.sock;
}
server {
listen 80 default;
listen [::]:80 default;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl default;
listen [::]:443 ssl default;
server_name my_server.domain.com;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload";
root /var/www/my_app/current/public;
ssl_certificate /etc/ssl/certs/my_app_crt;
ssl_certificate_key /etc/ssl/private/my_app_key;
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
ssl_prefer_server_ciphers on;
#See https://weakdh.org/
ssl_dhparam /etc/ssl/private/dhparams.pem;
client_max_body_size 500M;
location / {
if (-f $document_root/maintenance.html) {
return 503;
}
proxy_pass http://my_app; # match the name of upstream directive which is defined above
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
}
location ~* ^/assets/ {
# Per RFC2616 - 1 year maximum expiry
expires 1y;
add_header Cache-Control public;
# Some browsers still send conditional-GET requests if there's a
# Last-Modified header or an ETag header even if they haven't
# reached the expiry date sent in the Expires header.
add_header Last-Modified "";
add_header ETag "";
break;
}
error_page 503 @maintenance;
location @maintenance {
rewrite ^(.*)$ /maintenance.html break;
}
}
I'm thinking the firewall may be the problem, but we see nothing regarding blocked connections in our Palo Alto firewall. We've tried opening only postgresql traffic, and then broadening to only tcp traffic on port 5432, and the issues persists. The postgres configuration is pretty bog-standard, with a max_connections that outranks the max possible connections that can be made by the app server.