Lately Apache has been spawning workers even though there seems to be no reason for it. After ~10 minutes of a restart, it keeps spawning workers until it reaches the maximum number and new requests are never served.
After a service restart (service apache2 restart) I started monitoring the access log. There were only two rather simple requests before it started spawning workers like a madman:
#/var/log/apache2/access.log
66.249.64.215 - - [17/Feb/2016:00:30:18 +0000] "GET /wp-content/uploads/2015/08/colarinho_3.png HTTP/1.1" 304 179 "-" "Googlebot-Image/1.0"
201.80.103.196 - - [17/Feb/2016:00:33:00 +0000] "GET /wp-content/uploads/2015/09/circulo_cromatico.png HTTP/1.1" 200 40642 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0"
And here's the fullstatus before it completely stopped serving new requests.
$apache2ctl fullstatus
Apache Server Status for localhost (via ::1)
Server Version: Apache/2.4.7 (Ubuntu) PHP/5.5.9-1ubuntu4.12
Server MPM: prefork
Server Built: Jul 24 2015 17:25:11
__________________________________________________________________
Current Time: Wednesday, 17-Feb-2016 00:35:57 UTC
Restart Time: Wednesday, 17-Feb-2016 00:26:33 UTC
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 9 minutes 24 seconds
Server load: 0.00 0.06 0.18
Total accesses: 3 - Total Traffic: 48 kB
CPU Usage: u0 s0 cu0 cs0
.00532 requests/sec - 87 B/second - 16.0 kB/request
22 requests currently being processed, 4 idle workers
WWWWWWWWWWWWWWWWWWWW_WW___........................
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process
Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
0-0 6980 0/0/0 W 0.00 64 0 0.0 0.00 0.00 127.0.0.1
socialtailor.com.br:80 GET /lookup/191.185.144.200 HTTP/1.0
1-0 6981 0/1/1 W 0.00 55 0 0.0 0.00 0.00 127.0.0.1
socialtailor.com.br:80 GET /lookup/127.0.0.1 HTTP/1.0
2-0 6982 0/1/1 W 0.00 52 0 0.0 0.04 0.04 127.0.0.1
socialtailor.com.br:80 GET /lookup/127.0.0.1 HTTP/1.0
3-0 6983 0/0/0 W 0.00 65 0 0.0 0.00 0.00 191.185.144.200
socialtailor.com.br:80 GET / HTTP/1.1
4-0 6984 0/0/0 W 0.00 64 0 0.0 0.00 0.00 127.0.0.1
socialtailor.com.br:80 GET /lookup/127.0.0.1 HTTP/1.0
5-0 6989 0/0/0 W 0.00 55 0 0.0 0.00 0.00 127.0.0.1
socialtailor.com.br:80 GET /lookup/127.0.0.1 HTTP/1.0
6-0 6990 0/0/0 W 0.00 50 0 0.0 0.00 0.00 127.0.0.1
socialtailor.com.br:80 GET /lookup/127.0.0.1 HTTP/1.0
7-0 6991 0/0/0 W 0.00 41 0 0.0 0.00 0.00 127.0.0.1
socialtailor.com.br:80 GET /lookup/127.0.0.1 HTTP/1.0
8-0 6992 0/0/0 W 0.00 50 0 0.0 0.00 0.00 127.0.0.1
socialtailor.com.br:80 GET /lookup/127.0.0.1 HTTP/1.0
9-0 6993 0/0/0 W 0.00 40 0 0.0 0.00 0.00 127.0.0.1
socialtailor.com.br:80 GET /lookup/127.0.0.1 HTTP/1.0
10-0 6994 0/0/0 W 0.00 38 0 0.0 0.00 0.00 127.0.0.1
socialtailor.com.br:80 GET /lookup/127.0.0.1 HTTP/1.0
11-0 6995 0/0/0 W 0.00 35 0 0.0 0.00 0.00 127.0.0.1
socialtailor.com.br:80 GET /lookup/127.0.0.1 HTTP/1.0
12-0 6996 0/0/0 W 0.00 24 0 0.0 0.00 0.00 127.0.0.1
socialtailor.com.br:80 GET /lookup/127.0.0.1 HTTP/1.0
13-0 6997 0/0/0 W 0.00 32 0 0.0 0.00 0.00 127.0.0.1
socialtailor.com.br:80 GET /lookup/127.0.0.1 HTTP/1.0
14-0 6998 0/0/0 W 0.00 24 0 0.0 0.00 0.00 127.0.0.1
socialtailor.com.br:80 GET /lookup/127.0.0.1 HTTP/1.0
15-0 6999 0/0/0 W 0.00 24 0 0.0 0.00 0.00 127.0.0.1
socialtailor.com.br:80 GET /lookup/127.0.0.1 HTTP/1.0
16-0 7000 0/0/0 W 0.00 11 0 0.0 0.00 0.00 127.0.0.1
socialtailor.com.br:80 GET /lookup/127.0.0.1 HTTP/1.0
17-0 7001 0/0/0 W 0.00 7 0 0.0 0.00 0.00 127.0.0.1
socialtailor.com.br:80 GET /lookup/127.0.0.1 HTTP/1.0
18-0 7002 0/0/0 W 0.00 7 0 0.0 0.00 0.00 127.0.0.1
socialtailor.com.br:80 GET /lookup/127.0.0.1 HTTP/1.0
19-0 7003 0/0/0 W 0.00 4 0 0.0 0.00 0.00 127.0.0.1
socialtailor.com.br:80 GET /lookup/127.0.0.1 HTTP/1.0
20-0 7004 0/1/1 _ 0.00 7 0 0.0 0.01 0.01 ::1 socialtailor.com.br:80 GET
/server-status HTTP/1.0
21-0 7005 0/0/0 W 0.00 2 0 0.0 0.00 0.00 127.0.0.1
socialtailor.com.br:80 GET /lookup/127.0.0.1 HTTP/1.0
22-0 7006 0/0/0 W 0.00 0 0 0.0 0.00 0.00 ::1 socialtailor.com.br:80 GET
/server-status HTTP/1.0
__________________________________________________________________
Srv Child Server number - generation
PID OS process ID
Acc Number of accesses this connection / this child / this slot
M Mode of operation
CPU CPU usage, number of seconds
SS Seconds since beginning of most recent request
Req Milliseconds required to process most recent request
Conn Kilobytes transferred this connection
Child Megabytes transferred this child
Slot Total megabytes transferred this slot
__________________________________________________________________
Apache/2.4.7 (Ubuntu) Server at localhost Port 80
The site it is serving has a Wordpress installation. I updated it to the newest version and disabled cron jobs. Any idea why this is happening or how to debug this further?
Further strange thing: if I do a lynx http://localhost/ it keeps waiting for a response forever (even if Apache is not overloaded with workers) and then immediately Apache starts spawning a lot of workers, just like before.
I am using mpm_prefork with the following conf:
#/etc/apache2/mods-enabled/mpm_prefork.conf
<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxRequestWorkers 50
MaxConnectionsPerChild 1000
</IfModule>
