Nginx + php-fpm "504 Gateway Time-out" error with almost zero load (on a test-server)

Question

After debugging for 6-hours - I am giving this up :|

We have a nginx+php-fpm+mysql in LAN with almost 100 wordpress (created and used by different designers/developers all working on test wordpres setup)

We are using nginx without any issues from long.

Today, all of a sudden - nginx started returning "504 Gateway Time-out" out of the blue...

I checked nginx error log for a virtual host...

2010/09/06 21:24:24 [error] 12909#0: *349 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 21:25:11 [error] 12909#0: *349 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 21:25:11 [error] 12909#0: *443 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET /info.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 21:25:12 [error] 12909#0: *443 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 22:08:32 [error] 12909#0: *1025 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 22:09:33 [error] 12909#0: *1025 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 22:09:40 [error] 12909#0: *1064 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET /info.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 22:09:40 [error] 12909#0: *1064 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 22:24:44 [error] 12909#0: *1313 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 22:24:53 [error] 12909#0: *1313 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"

As I run php-fpm on port 9000 via TCP mode, I ran "netstat | grep 9000" and noticed something unusual... (Pasting partial output here for ease of read)

tcp        9      0 localhost:9000          localhost:36094         CLOSE_WAIT  14269/php5-fpm  
tcp        0      0 localhost:46664         localhost:9000          FIN_WAIT2   -               
tcp     1257      0 localhost:9000          localhost:36135         CLOSE_WAIT  -               
tcp     1257      0 localhost:9000          localhost:36125         CLOSE_WAIT  -               
tcp        9      0 localhost:9000          localhost:36102         CLOSE_WAIT  14268/php5-fpm  
tcp        0      0 localhost:46662         localhost:9000          FIN_WAIT2   -               
tcp      745      0 localhost:9000          localhost:46644         CLOSE_WAIT  -               
tcp        0      0 localhost:46658         localhost:9000          FIN_WAIT2   -               
tcp     1265      0 localhost:9000          localhost:46607         CLOSE_WAIT  -               
tcp        0      0 localhost:46672         localhost:9000          ESTABLISHED 12909/nginx: worker
tcp     1257      0 localhost:9000          localhost:36119         CLOSE_WAIT  -               
tcp     1265      0 localhost:9000          localhost:46613         CLOSE_WAIT  -               
tcp        0      0 localhost:46646         localhost:9000          FIN_WAIT2   -               
tcp     1257      0 localhost:9000          localhost:36137         CLOSE_WAIT  -               
tcp        0      0 localhost:46670         localhost:9000          ESTABLISHED 12909/nginx: worker
tcp     1265      0 localhost:9000          localhost:46619         CLOSE_WAIT  -               
tcp     1336      0 localhost:9000          localhost:46668         ESTABLISHED -               
tcp        0      0 localhost:46648         localhost:9000          FIN_WAIT2   -               
tcp     1336      0 localhost:9000          localhost:46670         ESTABLISHED -               
tcp        9      0 localhost:9000          localhost:36108         CLOSE_WAIT  14274/php5-fpm  
tcp     1336      0 localhost:9000          localhost:46684         ESTABLISHED -               
tcp        0      0 localhost:46674         localhost:9000          ESTABLISHED 12909/nginx: worker
tcp     1336      0 localhost:9000          localhost:46666         ESTABLISHED -               
tcp     1257      0 localhost:9000          localhost:46648         CLOSE_WAIT  -               
tcp     1336      0 localhost:9000          localhost:46678         ESTABLISHED -               
tcp        0      0 localhost:46668         localhost:9000          ESTABLISHED 12909/nginx: wo

There are plenty of "CLOSE_WAIT" & "FIN_WAIT2" pairs as highlighted below (in above output):

tcp     1337      0 localhost:9000          localhost:46680         CLOSE_WAIT  -               
tcp        0      0 localhost:46680         localhost:9000          FIN_WAIT2   -

Please note port 46680 in above.

I enabled mysql slow queries error log, but it didn't work.

As of now restarting php5-fpm every minute via a cronjob (see command below) keeping everything running "smoothly" but I hate patchwork and want to solve this...

1 * * * * service php5-fpm restart > /dev/null

I searched extensively on Google - got no help. As mentioned, this a test-server in LAN, CPU load is never crossed 0.10 and memory usage is also below 25% (System has 2GB RAM and ubuntu-server installed) So if you find its time-confusing to help me out, please atleast drop a hint.

Thanks in advance for help.

-Rahul

(note - this is reposting of - http://forum.nginx.org/read.php?11,127694)

Update: I found answer, which is posted below.

score 35 · Accepted Answer · edited Apr 20 '16 at 02:32

35

I found answer on my posting on the nginx forum - http://forum.nginx.org/read.php?2,127854

The answer, in my case, is to set:

request_terminate_timeout=30s

in php-fpm config (usually /etc/php5/fpm/php-fpm.conf)

Note, you can use values other than 30s also.

I used it to match my value in main php.ini file which is:

max_execution_time = 30

Thanks All. :-)

edited Apr 20 '16 at 02:32

chicks

3,639
10
26
36

answered Sep 08 '10 at 11:07

rahul286

1,647
4
20
25

5

This config may also be found in www.conf file. Thanks for the answer though, this seems to have done the trick. – eddiemoya Aug 16 '13 at 23:06
2

This is a pool-level directive, you will get an error message when trying to put it into the php-fpm.conf's `[global]` section. It works there only if you also have your pool configs in there. Also: [request_terminate_timeout docs](http://php.net/manual/en/install.fpm.configuration.php#request-terminate-timeout). – tanius Jan 30 '15 at 22:19
If this is the correct answer which I REALLY NEEED then this Friday will be the best of 2015 yet. – Philip Feb 20 '15 at 08:49
2

I found that putting `request_terminate_timeout=30s` into my `php-fpm.conf` file caused a (111 Connection Refused) error. When I moved it to my `www.conf` file it worked. – AJB Apr 20 '16 at 00:17
On CentOS 7.2 when using php7, request_terminate_timeout is located in: /etc/php-fpm.d/www.conf – nadavkav Jan 01 '17 at 13:45

score 19 · Answer 2 · answered Nov 12 '12 at 20:42

19

Here how it resolved my issue:

make following changes to /etc/nginx/nginx.conf in http { section

proxy_connect_timeout  600s;
proxy_send_timeout  600s;
proxy_read_timeout  600s;
fastcgi_send_timeout 600s;
fastcgi_read_timeout 600s;

and then restart nginx

/etc/init.d/nginx restart

answered Nov 12 '12 at 20:42

Vijay Kumar

219
2
2

2

Yeah, that really doesn't look like it has anything to do with the problem the person asking the question has. – HopelessN00b Nov 19 '12 at 11:39
3

but luckily that's what I needed :) – luchaninov Apr 02 '13 at 20:30
This didn't fix my issue, but it allowed me to see the actual error instead of the timeout message, which led me to the actual issue. – Michael Sep 21 '18 at 06:34

score 4 · Answer 3 · answered Sep 07 '10 at 07:49

4

If you are using php 5.3, increase the backlog.

If you are using php 5.2, backport the patch to increase the backlog size from 128.

Also, use a unix socket rather than a TCP socket. unix:/tmp/php5-cgi.sock (or the relevant path)

answered Sep 07 '10 at 07:49

karmawhore

3,865
17
9

I'd have to agree, especially with the unix socket usage. – Matt Sep 07 '10 at 13:05
Thanks karmawhore for answer. I found a solution on nginx mailing list. – rahul286 Sep 08 '10 at 11:03
@rahul286 which answere? iam interested! – breiti Dec 15 '11 at 19:50
@breiti see my anser below - http://serverfault.com/a/179136/17440 – rahul286 Dec 22 '11 at 12:44

score 3 · Answer 4 · answered Mar 02 '13 at 09:49

3

Great, thanks

request_terminate_timeout=30s

It works perfectly for me

but, I had to insert the line in this file : "/etc/php5/fpm/pool.d/www.conf" that is to say in the "Worker Section".

PHP 5.3.21-1 - Wordpress 3.5.1

http://php-fpm.org/wiki/Configuration_File

answered Mar 02 '13 at 09:49

Franck

31
1

I had a combination of factors that end up causing the 502 error your recipe did the magic trick! thanks you so much! – Jorge Vicente Mendoza Mar 08 '16 at 03:29

score 2 · Answer 5 · answered Jun 10 '13 at 08:30

2

in my case (same nginx error message), some problematic php scripts are not ending to execute and waiting for something, resulting no more php5-fpm children for nginx to pick.

fix:

add execution time limit other mentioned it this post. request_terminate_timeout=30s
raise children number. and everything worked like a charm. pm.max_spare_servers=16 pm.min_spare_servers=2

now everything worked like a charm.

answered Jun 10 '13 at 08:30

c2h2

759
2
8
20

I had a long running external connection request in my php script. Look for those long running tasks and put a timeout for them. – Ali Nadalizadeh Dec 01 '17 at 21:05

score 1 · Answer 6 · answered Nov 26 '15 at 17:58

1

For me same issue has occured after removing rabbitmq from server. Nothing of the above wasnt usefull, reinstalling all php5 modules solved the problem. I had Debian 8.2 on that server. Hope will be helpful for someone.

answered Nov 26 '15 at 17:58

Taggart Comet

31
1

Mawardy · Answer 7 · 2020-03-09T11:42:13.717

In our case the problem was that php-fpm couldn't open a session on an NFS share , it seems like one of the web nodes locked the directory , after restarting the NFS and only allowed 1 node to access it the issue was resolved ,Thank God

You should always check your php-fpm logs , in my case the location was at /var/opt/remi/php72/log/php-fpm

you should also enable slow scripts logs for better debugging and check the www-slow.log file located in the logs directory

to make sure slowlog is working make sure the following settings present in www.conf

slowlog = /var/opt/remi/php72/log/php-fpm/www-slow.log

request_slowlog_timeout = 5s

this should log any script that exceeds 5 seconds and logs the location in the php code that makes the problem in /var/opt/remi/php72/log/php-fpm/www-slow.log , you can change the path to meet your environment

score 0 · Answer 8 · answered Mar 18 '11 at 19:28

0

This may also help folks:

Depending on what your setup is you should look at the fastcgi configuration params as well as php ... in my case (I am using apache2 + php5-fpm) and max_execution time is also dependent on how long the fastcgi module waits for response (-idle-timeout) ...

http://www.fastcgi.com/mod_fastcgi/docs/mod_fastcgi.html#FastCgiExternalServer

answered Mar 18 '11 at 19:28

farinspace

173
1
1
12

why use apache2?? I mean u can use nginx directly to interact with php5-fpm. No need to use Apache when u have nginx! – rahul286 Mar 21 '11 at 15:57
If your using nginx, if others are NOT using nginx hopefully this will help them out. :-) ... I came across this page searching for Apache2 + php5-fpm related question – farinspace Mar 21 '11 at 18:27
ok. I thought you are using Nginx with Apache for PHP scripts like some pople used it in past. – rahul286 Mar 22 '11 at 16:05

score 0 · Answer 9 · answered Apr 12 '12 at 10:42

0

I had the same problem and I solved it by completely removeing Apache:

yum remove httpd

After that I reccomend restaring both PHP and NGINX:

/etc/init.d/nginx restart
/etc/init.d/php-fpm restart

answered Apr 12 '12 at 10:42

Nikolay

9
1

1

We did not have apache on our server then. Glad to know your case as it may help us in future. – rahul286 Apr 12 '12 at 12:44

Nginx + php-fpm "504 Gateway Time-out" error with almost zero load (on a test-server)

9 Answers9

Linked