502 errors on random (not all) pages

Question

A few weeks ago I noticed in the Google Webmaster tools that a few of my URL's were returning 502's.

One such URL is http://www.sau.com.au/forums/topic/437438-around-the-bay-wrap-up/

This URL returns the 502 for any desktop skin but works fine when on mobile (different skin)

Note that it returns the 502 very quickly, suggesting to me that it is not an execution or some form of timeout.

Besides a few more like this, all other URL's on the site are fine.

The only log entry to help at all is this one;

2015/06/29 09:33:39 [error] 19650#0: *4431763 upstream prematurely closed FastCGI stdout while reading response header from upstream, client: 94.228.34.203, server: sau.com.au, request: "GET /forums/topic/437438-around-the-bay-wrap-up/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm/php-fpm.sock:", host: "www.sau.com.au", referrer: "http://www.sau.com.au/forums/forum/100-victoria/"

I have also restarted APC which didn't help. It had only 1.5% fragmentation.

I can't find any limit-reached entries. The server is specced quite high so PHP has a lot of memory, request sizes and also long timeouts.

I tried the following that I read about but no diff.

fastcgi_buffer_size 10240k;
fastcgi_buffers 4 10240k;

I'm reluctant to make any big changes because this is only on some pages. One thread suggested updating PHP to 5.5.

I dont know where to look now for further assistance. What should my next step be?

Some info;

nginx version: nginx/1.6.2
PHP 5.3.3 (fpm-fcgi) (built: Oct 30 2014 20:14:56)

UPDATE 1

PHP error log has nothing written to it. Error logging is enabled though;

php_admin_value[error_log] = /var/log/php-fpm/www-error.log
php_admin_flag[log_errors] = on

UPDATE 2

dmesg doesn't seem to be updating and there is no kernel.log;

-rw-r--r--   1 root   root       41179 Dec 18  2014 dmesg

RESOLVED

I was installing XDebug as proposed by @sa289 in this answer and to do so I needed to update from PHP 5.3 to PHP 5.5. I was hoping that would fix the issue on its own, but it didn't. So, I installed XDebug and added the zend_extension to php.ini, restarted PHP and voilà the 502's are gone. I commented out the zend extension again and the 502's returned. XDebug to the rescue. No firm grasp on the cause or true resolution but this is good enough for me.

Hi @EEAA - There is nothing written to PHP log. Despite logging being enabled (see update) — Christian, Jun 29 '15 at 01:35
Have you checked dmesg or your kernel.log for crashes of php-fpm (segfaults etc.)? You should also be able to see them in your php-fpm.log. — Oliver, Jul 01 '15 at 07:20
Hi @Oliver. dmesg is not updating and there is no kernel.log. Grepping logs for 'segfault' turns up nothing. — Christian, Jul 01 '15 at 07:50
In php-fpm.log you won't find 'segfault', but you may be able to find processes 'exited on signal 11' instead. — Oliver, Jul 01 '15 at 07:54
Grepped all logs for 'exited' and 'signal 11' and got nothing. Even zgrepped in php log dir. — Christian, Jul 01 '15 at 08:06
Time to learn how to use `strace` it seems. Especially if things aren't being logged .... — Joseph Kern, Jul 01 '15 at 08:22
I had similar 502 issues few years ago and I got rid of them by uninstalling APC and installing Zend OPC cache for opcode cache. For PHP 5.3 you got it in the repo under php-pecl-zendopcache. — Nemke, Jul 01 '15 at 21:07
Thanks @JosephKern and Nemke. Will have a look into both of those. — Christian, Jul 03 '15 at 06:02

score 1 · Accepted Answer · answered Jul 07 '15 at 00:39

1

I'd see where it might be crashing by using Xdebug. Xdebug can be used to do something called tracing where it will tell you every line of code that executes and this way you can see the last line of code that was run before crashing so it will point you in the right direction. You can have it only trace when a certain cookie is set so you aren't generating trace logs for every request. Check out especially the following configuration settings:

xdebug.trace_enable_trigger
xdebug.trace_output_dir
xdebug.collect_params
xdebug.trace_output_name (I like "trace.%p.%t" because it puts the PID in the filename which can be helpful in certain cases)

See http://xdebug.org/docs/all_settings for details on these configuration parameters as well as others.

answered Jul 07 '15 at 00:39

sa289

1,308
2
17
42

Yes! Great idea, thanks. I'll have a look at XDebug and report back. – Christian Jul 07 '15 at 05:50
@Christian Were you able to solve the issue? – sa289 Jul 13 '15 at 20:16
I'm sorry @sa289 I haven't had time to try, i'll try to tonight. – Christian Jul 14 '15 at 06:24
@Christian Was my answer able to address your question? If so, I'd appreciate if you mark it as accepted and/or upvote it so I can get credit for it. If not, let me know. Thanks – sa289 Jul 24 '15 at 18:21
Sorry @sa289 I haven't forgotten this but I have not been able to install XDEBUG yet as its not in my repos. I'll be trying to do it this week as well as maybe updating to php 5.5. I'll report back. Apologies for the delay. – Christian Jul 27 '15 at 02:21
See my update in the question - XDebug inadvertently fixed the problem. Thanks! – Christian Jul 28 '15 at 07:18
@Christian how interesting. As a side note, in some past testing we did, having XDebug installed slows down performance significantly. Now of course that may be worth it to you, but I wanted to let you know in case that makes it worth looking into trying to find the cause more. – sa289 Jul 28 '15 at 14:50
I've noticed a big jump in whats logged, so I need to disable all the additional logging because it's killing the box during high traffic. But, otherwise its working well. Thanks again. – Christian Jul 29 '15 at 01:51

502 errors on random (not all) pages

1 Answers1