1

I have an Apache server that's accessing source code over NFS (from a separate dedicated NFS file server). Periodically the Apache server stops accepting new requests (effectively bringing down the server), and when I look into it, it's filling up MaxClients. Upon further research, the site is not that busy... the reason the stack filled up was each Apache process is stuck in a "sending Reply" state (all W's via mod_status). Additionally in top, each apache process is in state "lockf", which makes me think it's an NFS issue. I can't find much information about it online, and I've been pulling my hair out this weekend trying to figure it out. Does anyone have any ideas on what might be going on, or how I can further diagnose the problem?

Thanks so much!

Harry
  • 221
  • 1
  • 5
  • 9
  • What are the NFS mount options you're using? – al. Oct 11 '10 at 14:34
  • What kernel version are you using? Please paste the output of `uname -a` – Tom O'Connor Oct 11 '10 at 15:02
  • It's a freeBSD system: $ uname -a FreeBSD -my-host-name-.com 7.2-RELEASE FreeBSD 7.2-RELEASE #0: Fri May 1 07:18:07 UTC 2009 root@driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 – Harry Oct 11 '10 at 17:29

3 Answers3

1

We've just had an issue with similar symptoms here, servers get to MaxClients then the number of apache processes doesn't drop and new clients can't connect.

It appears the issue was caused by deadlock from the interaction of our web applications and the MaxClients setting on Apache. I was pointed in the right direction by another question here on ServerFault: Apache reaching MaxClients and locking the server

edit: Incidentally since we can't fix our application at the moment we've had to settle for adding more web servers and un-capping the MaxClients setting.

Community
  • 1
James Yale
  • 5,042
  • 1
  • 16
  • 20
0

If your logs are being written to your nfs server, make sure you specify the Lockfile directive to be on a local disk.

karmawhore
  • 3,865
  • 17
  • 9
  • Logs are not being written to the NFS server (all logs are written locally, normal configuration). – Harry Oct 12 '10 at 16:37
0

James Yale is correct. You are currently suffering from the dreaded "Perpertual Bush Syndrome" (no one likes to be stuck with "W" forever). In short, you have to track down which script is "stuck in W" and find out what variables that's being passed to the script that results in the lock. here's a very good article that should help: Solving Httpd MaxClients and mod status "stuck in W"

user100055
  • 16