Torrents of apache processes bringing down the server

Question

We am running LAMP stack on CentOS 7. Have been fine for about a year.

But today we just saw loading averages hitting through the roof. We checked top didn't see any particular process with high load but there were a stream of very many apache processes running.

Tried to restart apache several times but didn't help.

Finally decided to restart MySQL. This briefly brought down the load but not for long.

Then decided to restart the server. This also just briefly brought down the load.

The logs don't show much of external connections:

$ sudo tail -n 500 /var/log/httpd/access_log | cut -d' ' -f1 | sort | uniq -c | sort -gr
395 ::1
101 127.0.0.1
3 66.102.6.91
1 66.102.6.89

What could be going on? How can we fix this?

Edit 1: More details

Load averages are ranging between 50 and 300. At some point we had this:

 $ uptime
     10:33:28 up 317 days, 22:36,  2 users,  load average: 81.21, 75.67, 75.62

Checking later:

$ uptime
12:41:50 up 50 min,  2 users,  load average: 227.67, 147.99, 75.37

Following Apache spawning hundreds of processes and pushing up load averages we checked config but didn't see ServerLimit or any of the other parameters:

$ less /etc/httpd/conf/httpd.conf | grep ServerLimit
$

I would read this [lengthy but concise explanation of how load averages work in UNIX /Linux systems](http://m.linuxjournal.com/article/9001). A high load is indicative that there is likely a performance issue somewhere but it's by no means a 'problem to solve'. You're gonna have to figure out whether you have a problem before you can try to fix it. You need to learn some basic diagnostic methods that really arent appropriate for the Q and A format we have here. You should be putting far more effort into solving your own problem before it would be appropriately on-topic for serverfault. — , Aug 27 '16 at 10:30
Also, please be exact as possible. Whatever your load averages that were 'hitting the roof' may be far lower than what I see on a regular basis. Without knowing how many cores your system has, whether it's a vps, or a dedicated server along with what the ACTUAL load average is, we can't even confirm that your load is higher than expected. — , Aug 27 '16 at 10:34
You need to find out what's going on. There are so many things you can do to debug apache or mysql problems, io performance, memory issues, cpu performance, etc... There are configuration options all over the place that could help/hurt you. Didn't you ever deal with this stuff up until now? It's not something that can be answered in a simple question on SF. — Ryan Babchishin, Aug 27 '16 at 13:58

score 2 · Answer 1 · edited Apr 13 '17 at 12:14

2

Your system is clearly overloaded. You should install monitoring, use it to gather data and then apply Scientific Method - it's the only way to be sure.

Torrents of apache processes bringing down the server

1 Answers1