1

I get the follow error in the nginx log

[error] 17734#0: *6643 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: [cut], server: [cut], request: "GET /venues HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "[cut]"

I have a dedicated box with 8 gb ram, quad core chip. Good server. Nginx, php-fpm & mysql all latest versions running under ubuntu 10.04

I only get this when I stress test the server with siege. If I increase the number of concurrent connections to 100, I can get up to 20% of all requests to fail.

Furthermore, I don't get this on pages that have no mysql queries. And only a few failures on pages with moderate number of queries. Bit, I'm not sure if that's got to do anything with it.

I have a feeling this is something to do with php. But I can't figure it out.

Any suggestions of where to even start looking?

Update: and the php error log is silent. No record of anything going wrong

Ilya Biryukov
  • 153
  • 3
  • 6
  • I've seen many of these threads pop up in the php-highload-en group. One guys problem was hitting the mysql connection limit durring siege . Most of the time they recommend profiling the application and its sql queries. You could try posting there: http://groups.google.com/group/highload-php-en – iainlbc Aug 18 '11 at 23:24
  • What about the nginx error log? And the MySQL normal log as well as the slow query log? Is query caching in play at all? What is the nginx.conf like? What is the php config like? – JayMcTee Nov 11 '15 at 09:23

2 Answers2

1

Most probably you ran out of php-fpm workers. The log was silent because nothing was wrong in the code itself - workers just were busy processing your requests. If you did not get this on pages without MySQL queries, the bottleneck was the MySQL DB. You should identify long-running queries (using mytop or slow log feature or maybe some custom PHP logging around your SQL processing) and optimize them. Of course, "long queries" are actually quite short in high-load we world. Even 200ms query is long enough to put your server on its knees.

Alex
  • 7,789
  • 4
  • 36
  • 51
  • Does `php-fpm` seriously just silently drop connections when it's pool is fully occupied? That's horrendous. – womble Aug 18 '11 at 23:08
  • @womble When load comes in peaks it depends on a TCP backlog size I guess, but when load is constantly high the backlog will end sooner or later, so this behavior is quite predictable. – Alex Aug 18 '11 at 23:14
  • 1
    Sure, sometimes you can't handle something, and dropping a connection is possibly the best option. It's the *silence* I can't fathom. If you're going to do something out of the ordinary, you *have* to log it. No exceptions. – womble Aug 18 '11 at 23:42
-1

this may be resolved. i reported a similar issue with opening and reusing persistent tcp sockets under almost no load at all. there's now a patch landed for this in git:

https://groups.google.com/forum/#!topic/highload-php-en/qGu3Eaifj9s

leeoniya
  • 127
  • 5