I appreciate that it doesn't always suit to have mod_status available and on all of the time, but it and apachetop are the best ways to diagnose these problems. However there are many ways to skin a cat.
This trick is useful in a number of circumstances and isn't just Apache specific. It does depend on a number of factors however, and you need to know what it's doing to know it's limitations.
for pid in `pgrep -u www-data`; do find /proc/${pid}/cwd -printf "%l\n" ; done
Let's break it down:
- pgrep -u www-data gives you the list of pids running under user www-data. That's the default on Debian / Ubuntu, change to suit your own system (RedHat based systems tend to use httpd, for example, as the user). For systems without pgrep, you can use ps axuwww | grep user | awk '{print $2}'
- the *for; do; ... done * loop means we loop over every entry running the command(s) within the do part of the loop.
- find /proc/${pid}/cwd -printf "%l\n" simply searches /proc for each of those PIDs and spits out the current working directory for that process. Apache will chdir() to the VirtualHost by default when serving files from that VirtualHost. /proc/PID/cwd is a symbolic link to the directory that apache process is running in. the printf "%l\n" prints the endpoint to that link. See find(1) for more info on that.
There are two major caveats to that trick:
1) If something running under the same context as the Apache process does a chdir()'s outside of the VirtualHost directory, you'd be hard pushed to find that out.
e.g. a PHP script running under mod_php (a CGI will be different as Apache fork's a separate process, but I'm presuming CGI's aren't a problem or you'd be able to track them easier).
2) If you have Apache instances which are very very quickly serving pages (e.g. a small static HTML page). This normally isn't a problem, but it may be possible. If you're getting a lot of "No such file or directory" errors, this is basically a manifestation of it. I would expect some, but not the majority unless they fit this particular case. Basically this is because the Apache processes you've scanned with ps have already exited by the time you've checked /proc. Obviously this means they are serving pages very very quickly.
Regarding memory bound Apache processes, I use ps_mem.py to calculate memory usage on my webservers. If you've got large Apache (in terms of resident memory size) processes and they are exiting quickly, that is roughly the equivalent of asking a big fat guy to keep running 100m sprints. If your webserver isn't a shared one, those "No such file or directory" errors are normally good candidates to move some content onto a smaller lightweight webserver (e.g. nginx / lighttpd) or start heavily caching content (e.g. varnish / squid).
