9

I have recently upgrade my magento from 1.5 to 1.9 and when ever I add a certain product to basket, I started to receive this error: 502 Bad Gateway

There were no log entries in the var/log/ folder: enter image description here

So, I had a look at my nginx errors and I found the following entries in nginx-errors.log:

2015/04/09 10:58:03 [error] 15208#0: *3 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 46.xxx.xxx.xxx, server: dev.my-domain.co.uk, request: "POST /checkout/cart/add/uenc/aHR0cDovL2Rldi5zYWx2ZW8uY28udWsvdGludGktYmF0aC1wYWludGluZy1zb2FwLTcwbWwuaHRtbD9fX19TSUQ9VQ,,/product/15066/form_key/eYLc3lQ35BSrk6Pa/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fcgi-www-data.sock:", host: "dev.my-domain.co.uk", referrer: "http://dev.my-domain.co.uk/tinti-bath-painting-soap-70ml.html"
2015/04/09 11:04:42 [error] 15208#0: *13 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 46.xxx.xxx.xxx, server: dev.my-domain.co.uk, request: "POST /checkout/cart/add/uenc/aHR0cDovL2Rldi5zYWx2ZW8uY28udWsvdGludGktYmF0aC1wYWludGluZy1zb2FwLTcwbWwuaHRtbD9fX19TSUQ9VQ,,/product/15066/form_key/eYLc3lQ35BSrk6Pa/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fcgi-www-data.sock:", host: "dev.my-domain.co.uk", referrer: "http://dev.my-domain.co.uk/tinti-bath-painting-soap-70ml.html"
2015/04/09 11:05:03 [error] 15208#0: *16 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 46.xxx.xxx.xxx, server: dev.my-domain.co.uk, request: "POST /checkout/cart/add/uenc/aHR0cDovL2Rldi5zYWx2ZW8uY28udWsvdGludGktYmF0aC1wYWludGluZy1zb2FwLTcwbWwuaHRtbD9fX19TSUQ9VQ,,/product/15066/form_key/eYLc3lQ35BSrk6Pa/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fcgi-www-data.sock:", host: "dev.my-domain.co.uk", referrer: "http://dev.my-domain.co.uk/tinti-bath-painting-soap-70ml.html"
2015/04/09 11:12:07 [error] 15273#0: *1 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 46.xxx.xxx.xxx, server: dev.my-domain.co.uk, request: "POST /checkout/cart/add/uenc/aHR0cDovL2Rldi5zYWx2ZW8uY28udWsvdGludGktYmF0aC1wYWludGluZy1zb2FwLTcwbWwuaHRtbD9fX19TSUQ9VQ,,/product/15066/form_key/eYLc3lQ35BSrk6Pa/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fcgi-www-data.sock:", host: "dev.my-domain.co.uk", referrer: "http://dev.my-domain.co.uk/tinti-bath-painting-soap-70ml.html"

I have installed magento on a custom LEMP stack, here are the configurations:

This error only appears to be happening when I add a specific product to basket in my upgraded magento and every time the error occurs, I can see a core.XXXXX file (which is approx 350mb) in the public_html folder.

Any idea why my php-fpm is crashing like this? How can I find the cause and fix it?

Here's the last entries on my Linux (CentOS) server, when I run dmesg command:

php-fpm[14862]: segfault at 7fff38236ff8 ip 00000000005c02ba sp 00007fff38237000 error 6 in php-fpm[400000+325000]
php-fpm[15022]: segfault at 7fff38351ff0 ip 00000000005bf6e5 sp 00007fff38351fb0 error 6 in php-fpm[400000+325000]
php-fpm[15021]: segfault at 7fff38351ff0 ip 00000000005bf6e5 sp 00007fff38351fb0 error 6 in php-fpm[400000+325000]
php-fpm[15156]: segfault at 7fff38351ff0 ip 00000000005bf6e5 sp 00007fff38351fb0 error 6 in php-fpm[400000+325000]
php-fpm[15024]: segfault at 7fff38351ff0 ip 00000000005bf6e5 sp 00007fff38351fb0 error 6 in php-fpm[400000+325000]
php-fpm[15223]: segfault at 7fff8d1d5fd8 ip 00000000005c02ba sp 00007fff8d1d5fe0 error 6 in php-fpm[400000+325000]
php-fpm[15222]: segfault at 7fff8d1d5fd8 ip 00000000005c02ba sp 00007fff8d1d5fe0 error 6 in php-fpm[400000+325000]
php-fpm[15225]: segfault at 7fff8d1d5fd8 ip 00000000005c02ba sp 00007fff8d1d5fe0 error 6 in php-fpm[400000+325000]
php-fpm[15227]: segfault at 7fff8d1d5fd8 ip 00000000005c02ba sp 00007fff8d1d5fe0 error 6 in php-fpm[400000+325000]
php-fpm[15362]: segfault at 7fff3118afd0 ip 00000000005c0ace sp 00007fff3118afa0 error 6 in php-fpm[400000+325000]

I analysed the core dump with gdb and this is what I see for the first two frames: http://pastebin.com/raw.php?i=aPvB1sWv (doesn't make much sense to me)...

Latheesan
  • 347
  • 2
  • 6
  • 18
  • 4
    Your PHP is segfaulting, and the fact that the coredump is about 350mb suggest there's an enormous memory leak in PHP... it up to date ? –  Apr 09 '15 at 16:30
  • Yes I am using the latest nginx, php-fpm and this is my php version: `[root@DevServer ~]# php -v PHP 5.5.23 (cli) (built: Mar 19 2015 09:03:54) Copyright (c) 1997-2015 The PHP Group Zend Engine v2.5.0, Copyright (c) 1998-2015 Zend Technologies` – Latheesan Apr 09 '15 at 16:32
  • i've read elsewhere that upgrading to php 5.6.x might help, so I did - but the error is still occurring. – Latheesan Apr 10 '15 at 08:40
  • I analysed the core dump with gdb and this is what I see for the first two frames: http://pastebin.com/raw.php?i=aPvB1sWv (doesn't make much sense to me)... – Latheesan Apr 10 '15 at 09:07
  • 1
    had similar behaviour under ubuntu and solved by using ppa:ondrej/php5-5.6. perhaps you can try an alternative repo: http://rpms.famillecollet.com/enterprise/$releasever/php56/mirror (i would prever 5.6 too) – Kevin Horst Dec 18 '15 at 16:31
  • are you running your linux server on bare metal or is it some "virtual root server"? – omoser Feb 22 '16 at 14:59
  • are there only 2 frames in the dump ? the last frames are more interesting then the first ones. – smaftoul May 23 '16 at 22:59
  • Hi - Encountered similar issue while working on code to handle large file downloads. Saw other posts and solutions and found increasing the php.ini memory_limit setting made the difference for downloads, so wanted to share. If not wanting to change the php.ini file, could also likely use ini_set('memory_limit', '1024M') in your code and replace 1024M with what you need. Hope this is helps a few people. – John Oct 11 '18 at 21:48

2 Answers2

5

Such errors usually occurs when server is running out of resources, assuming that you're running most recent stable versions of php5-fpm:

  1. Check that php5-fpm has enough memory (there's no oom-killer killing the process)

  2. There's enough space on the disk

  3. Make sure to check the open file limits on the server. You're interested especially in the hard limit (-Hn):

    $ ulimit -Hn
4096
$ ulimit -Sn
1024

Check number of currently opened file descriptors on the server:

    sysctl fs.file-nr
    fs.file-nr = 1440       0       790328

Modern servers are capable of handling many files, usually ulimits are set to unnecessary low values.

Then check nginx.conf, at the beginning there's something like:

    worker_processes 4;
    events {
      worker_connections 1024;
    }

If you're proxying request for each connection you'd need 2 file handles. Which means that in case of many connections you'd reach the limit quite quickly.

nginx has a worker_rlimit_nofile directive for restricting opened files per worker process (top level directive like worker_processes 4;):

    worker_rlimit_nofile    1024;

Just do the math and calculate how many opened file descriptors you'd need when all connections are used (a bit extreme case). Also consider all other services running on that server.

Tombart
  • 2,013
  • 3
  • 27
  • 47
1

try to rise buffer size at php.ini, .htaccess or httpd.conf (because PHP_INI_PERDIR)

#buffer-size
output_buffering = 65535

At second add it to your /etc/httpd/conf.d/fcgid.conf

FcgidMaxRequestLen 1073741824
FcgidOutputBufferSize 1073741824
MaxRequestLen 1073741824
  
FcgidMaxRequestsPerProcess 500
FcgidMaxProcesses 300
FcgidConnectTimeout 180
FcgidIOTimeout 2000
FcgidIdleTimeout 60
FcgidFixPathinfo 1
FcgidProcessLifeTime 60
FcgidIdleScanInterval 15
FcgidBusyTimeout 180
FcgidBusyScanInterval 30
FcgidErrorScanInterval 3
FcgidZombieScanInterval 3
Arthur Veselov
  • 111
  • 1