11: Resource temporarily unavailable, while connecting to upstream + Bad Gateway (Nginx)

Question

For all the sites on my new server, it seems as though once nginx has served some amount of traffic, it starts serving "ERROR 502 - Bad Gateway!".

Let's take a look at some logs and config...

OS: Ubuntu 16.04 (Server)
Hardware: 6 cores, 12GB RAM
Panel: ISPConfig
nginx version: 1.10.3

From /var/www/example/log/error.log:

11: Resource temporarily unavailable) while connecting to upstream, client: 216.170.123.179, server: example.com, request: "GET /KHFxygk9 HTTP/1.1", upstream: "fastcgi://unix:/var/lib/php7.0-fpm/web3.sock:", host: "example.com", referrer: "http://example.com/"
2017/08/10 06:01:06 [error] 15408#15408: *31372 connect() to unix:/var/lib/php7.0-fpm/web3.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 192.3.114.12, server: example.com, request: "GET /jiPNQjwp HTTP/1.1", upstream: "fastcgi://unix:/var/lib/php7.0-fpm/web3.sock:", host: "example.com", referrer: "http://example.com/"

Here are my config files:

/etc/php/7.0/fpm/pool.d/www.conf

[www]    
user = www-data
group = www-data

listen = /run/php/php7.0-fpm.sock

listen.owner = www-data
listen.group = www-data

pm = dynamic
pm.max_children = 12    
pm.start_servers = 2    
pm.min_spare_servers = 1    
pm.max_spare_servers = 3    
pm.max_requests = 500

/etc/nginx/nginx.conf

user www-data;
worker_processes auto;
pid /run/nginx.pid;

events {
        worker_connections 768;
}

http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;
        gzip_disable "msie6";


        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

systemctl status php7.0-fpm

● php7.0-fpm.service - The PHP 7.0 FastCGI Process Manager
   Loaded: loaded (/lib/systemd/system/php7.0-fpm.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2017-08-10 04:44:16 UTC; 1h 30min ago
  Process: 20310 ExecReload=/bin/kill -USR2 $MAINPID (code=exited, status=0/SUCCESS)
  Process: 20302 ExecReload=/usr/lib/php/php7.0-fpm-checkconf (code=exited, status=0/SUCCESS)
  Process: 13060 ExecStartPre=/usr/lib/php/php7.0-fpm-checkconf (code=exited, status=0/SUCCESS)
 Main PID: 13070 (php-fpm7.0)
   Status: "Processes active: 20, idle: 12, Requests: 6874, slow: 0, Traffic: 0.3req/sec"
   CGroup: /system.slice/php7.0-fpm.service
       ├─13070 php-fpm: master process (/etc/php/7.0/fpm/php-fpm.conf)                      
       ├─13072 php-fpm: pool apps-{fpm_domain}                                              
       ├─13073 php-fpm: pool apps-{fpm_domain}                                              
       ├─13074 php-fpm: pool ispconfig                                                      
       ├─13075 php-fpm: pool ispconfig                                                      
       ├─13076 php-fpm: pool web1                                                           
       ├─13077 php-fpm: pool web1                                                           
       ├─13079 php-fpm: pool web2                                                           
       ├─13080 php-fpm: pool web3                                                           
       ├─13081 php-fpm: pool web3                                                           
       ├─13082 php-fpm: pool web4                                                           
       ├─13083 php-fpm: pool web4                                                           
       ├─13084 php-fpm: pool web5                                                           
       ├─13085 php-fpm: pool web5                                                           
       ├─13087 php-fpm: pool www                                                            
       ├─13088 php-fpm: pool www                                                            
       ├─16186 php-fpm: pool web3                                                           
       ├─16603 php-fpm: pool web3                                                           
       ├─16604 php-fpm: pool web3                                                           
       ├─16605 php-fpm: pool web3                                                           
       ├─16609 php-fpm: pool web3                                                           
       ├─16614 php-fpm: pool web3                                                           
       ├─16629 php-fpm: pool web3                                                           
       ├─16630 php-fpm: pool web3                                                           
       ├─18019 php-fpm: pool web2                                                           
       ├─18026 php-fpm: pool web2                                                           
       ├─18027 php-fpm: pool web2                                                           
       ├─18028 php-fpm: pool web2                                                           
       ├─18369 php-fpm: pool web2                                                           
       ├─18370 php-fpm: pool web2                                                           
       ├─18410 php-fpm: pool web2                                                           
       ├─18411 php-fpm: pool web2
       └─18413 php-fpm: pool web2                                                           
Aug 10 04:44:16 web1.example.com systemd[1]: Stopped The PHP 7.0 FastCGI Process Manager.
Aug 10 04:44:16 web1.example.com systemd[1]: Starting The PHP 7.0 FastCGI Process Manager...
Aug 10 04:44:16 web1.example.com systemd[1]: Started The PHP 7.0 FastCGI Process Manager.

Have you fixed this issue? – Vijay Anand Pandian Jan 03 '20 at 09:55 — Vijay Anand Pandian, Jan 03 '20 at 09:55

score 15 · Answer 1 · answered Jan 07 '20 at 13:05

These errors could be caused by lack of allowed maximum number of socket connections (mostly default is 128). You can view the current limit by executing cat /proc/sys/net/core/somaxconn and it's highly recommended to increase this limit:

sudo echo "net.core.somaxconn = 65535" | tee -a /etc/sysctl.conf
sudo sysctl -p

And check whether it has changed cat /proc/sys/net/core/somaxconn.

Also these errors could be caused by lack of number of incoming connections backlog queue (mostly default is 1000). Check the current limit executing cat /proc/sys/net/core/netdev_max_backlog. It's recommended to increase the limit:

sudo echo "net.core.netdev_max_backlog = 65535" | tee -a /etc/sysctl.conf
sudo sysctl -p

And check whether it has changed cat /proc/sys/net/core/netdev_max_backlog

Note: it's impossible to follow my advice if these troubles appeared on your OpenVZ VPS, because you can't change kernel parameters using this type of virtualization. The better option in this case is trying to use port listening in your php-fpm pool config instead of socket (e.g. listen = 9000). Check your busy ports for finding available one by executing sudo netstat -tunlp.

This was good, it brought me the following results: status: 1xx=0 2xx=10000 3xx=0 4xx=0 5xx=0 From... status: 1xx=0 2xx=8136 3xx=0 4xx=0 5xx=1864 However, seeing some errors now like these on nginx errors, what do you think Alex? failed (24: Too many open files) while connecting to upstream, — SomeDudeSomewhere, Apr 03 '20 at 17:32
How can you test the current usage towards these limits, in order to know which limit is being hit? — Quolonel Questions, Jul 08 '21 at 15:00

score 2 · Answer 2 · answered Aug 10 '17 at 12:03

2

you have some mismatch in cfg, in php define listen = /run/php/php7.0-fpm.sock but nginx try to connect to fastcgi://unix:/var/lib/php7.0-fpm/web3.sock. Set the same value on both places (php and nginx)

answered Aug 10 '17 at 12:03

Quantim

1,269
11
13

score 1 · Answer 3 · answered Aug 10 '17 at 08:48

1

You have to increase your php-fpm max children's setting pm.max_children. The current value is 12.

Can you attach php-fpm error log also?

answered Aug 10 '17 at 08:48

dkokmadis

546
2
9

If max_children has to be increased much higher than that, it's either a very poorly designed application, or the server has a lot of cores. – Zenexer Aug 12 '17 at 10:45

11: Resource temporarily unavailable, while connecting to upstream + Bad Gateway (Nginx)

3 Answers3

Linked