2

We have a standard lighttpd deployment with PHP-CGI and our error logs are flooding with the following. This is causing a huge problem because we keep returning 500's to our clients:

2012-10-14 14:28:38: (mod_fastcgi.c.3001) backend is overloaded; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 36 
2012-10-14 14:28:38: (mod_fastcgi.c.2764) fcgi-server re-enabled:  0 /tmp/php-7735.socket 
2012-10-14 14:28:39: (mod_fastcgi.c.2764) fcgi-server re-enabled:  0 /tmp/php-7735.socket 
2012-10-14 14:28:40: (mod_fastcgi.c.3001) backend is overloaded; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 37 
2012-10-14 14:28:40: (mod_fastcgi.c.2764) fcgi-server re-enabled:  0 /tmp/php-7735.socket 
2012-10-14 14:28:41: (mod_fastcgi.c.3001) backend is overloaded; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 57 
2012-10-14 14:28:41: (mod_fastcgi.c.3001) backend is overloaded; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 57 
2012-10-14 14:28:42: (mod_fastcgi.c.3597) all handlers for /index.php? on .php are down.

Does anyone have any clue as to what is going on? We restarted all php and lighttpd related processes and that didn't fix the problem. We ended up rebooting the whole box and now its gone away, although we fear it may come back later....

In general our deployment has been doing fine for a long time and this is the first time this has happened.

AbuZubair
  • 123
  • 1
  • 4

1 Answers1

2

This basically means that all the php processes are busy processing requests, and no free php processes are available to handle new incoming requests. This causes the 500 error.

Usually this means that the php code that you are running is slow, swamping the php processes. Try enabling your mysql slow log to see if slow queries are the culprit, try php opcode caching such as APC to speed things up a bit.

Wouter
  • 226
  • 1
  • 4
  • Thanks for the comment. I figured as much but we have very little db operations. I actually do have an auto complete implementation that makes an http request with every character typed. My auto complete implementation lives in ram so its very fast. However with an immense load I may actually hit the limit of handling incoming requests. Let me poke around and get back to you. – AbuZubair Oct 17 '12 at 03:45
  • That is indeed the problem. Our auto complete was getting overloaded which was burdening the web server with a barrage of http requests. I'm looking around for a better architecture to solve this problem. – AbuZubair Nov 12 '12 at 06:35