High CPU + Memory, no visitors

Question

Current this is what I get when running top

Tasks: 151 total,   2 running, 149 sleeping,   0 stopped,   0 zombie
Cpu(s): 74.1%us,  1.3%sy,  0.0%ni, 43.6%id,  0.0%wa,  0.0%hi,  0.3%si,  3.6%st
Mem:   4045608k total,  3447088k used,   598520k free,    13588k buffers
Swap:   131068k total,        0k used,   131068k free,   387916k cached


3569 www-data  20   0  349m  72m  14m S   14  1.8   1:49.47 apache2
3572 www-data  20   0  349m  72m  14m S   14  1.8   1:47.16 apache2
3611 www-data  20   0  348m  70m  13m R   14  1.8   1:43.37 apache2
3565 www-data  20   0  349m  71m  13m S   13  1.8   1:45.07 apache2
3608 www-data  20   0  349m  73m  14m S   13  1.9   1:45.83 apache2
3550 www-data  20   0  349m  71m  13m S   13  1.8   1:49.43 apache2
3574 www-data  20   0  349m  72m  14m S   13  1.8   1:42.73 apache2
3602 www-data  20   0  349m  71m  13m S   13  1.8   1:40.62 apache2
3603 www-data  20   0  349m  71m  12m S   12  1.8   1:44.38 apache2
3561 www-data  20   0  349m  72m  14m S   12  1.8   1:45.13 apache2
3564 www-data  20   0  349m  72m  14m S   12  1.8   1:44.19 apache2
3531 www-data  20   0  349m  72m  14m R    6  1.8   1:43.68 apache2
3543 www-data  20   0  349m  72m  14m R    6  1.8   1:46.76 apache2
3604 www-data  20   0  349m  72m  14m S    2  1.8   1:44.09 apache2
3549 www-data  20   0  343m  66m  14m R    2  1.7   1:45.05 apache2
3052 mysql     20   0 2361m  83m  11m S    1  2.1   0:22.64 mysqld
1 root      20   0 24332 3224 2316 S    0  0.1   0:00.95 init

Looking through the log files it does seem I do hit maxClients, but I have no idea how this is happening because no one is viewing the site.. Current apache config below:

<IfModule mpm_prefork_module>
  StartServers          2
  MinSpareServers       6
  MaxSpareServers       12
  MaxClients            50
  MaxRequestsPerChild   3000
</IfModule>

This just started randomly happening, no updates, no changes.

Hi, kindly provide more information with your question. Is this an internally hosted server or accesible from the internet? How are you sure that no one is visiting the site? Can you check apache access logs to confirm that no pages were requested? — ngn, Jul 20 '15 at 14:14
@ngn currently the site is hosted using Linode.com, just checked the access logs and it's actually blank which is odd. Google Analytics reports 2 people currently — Starboy, Jul 20 '15 at 14:18
Can you run a `netstat -n | grep :80` on your box and post the output? — ngn, Jul 20 '15 at 14:22
@ngn updated orginal post (couldn't fit all of it) I'm thinking this is probably bad — Starboy, Jul 20 '15 at 14:29
It definitely looks like an attack. Saw a similar case in Stackoverflow a couple of days ago. Please refer to [this](http://serverfault.com/questions/134823/httpd-problem-suspect-an-attack-but-not-sure) and [this](http://serverfault.com/questions/706195/apache-unresponsive-and-fails-to-restart/706197#706197) for help. There are links to help pages and solutions in the second link. Please make sure you _completely remove_ or filter out the netstat output you posted here, it contains your actual IP address and will attract more trouble. EDIT: Have removed the netstat output from your post. — ngn, Jul 20 '15 at 14:38
Try this for MaxClients and see if it changes anything, MaxClients = (RAM - size_all_other_processes)/(size_apache_process) — zertux, Jul 21 '15 at 10:07

score 1 · Accepted Answer · edited Apr 13 '17 at 12:14

1

Although there are many ways to address this, I found an old question on ServerFault which suggests a simple iptables rule

iptables -I INPUT -p tcp --dport 80 \
         -m connlimit --connlimit-above 20 --connlimit-mask 40 -j DROP

edited Apr 13 '17 at 12:14

Community

1

answered Jul 21 '15 at 09:54

ngn

333
1
10

went ahead and put this in hopefully this locks it down. I wasn't under attack at the moment. Seems that Varnish would be a wise choice to implement as well. Thanks for the help again @ngn ! – Starboy Jul 21 '15 at 15:31

shodanshok · Answer 2 · 2015-07-20T18:11:29.967

0

You can use apache2ctl fullstatus to obtain a complete list of remote client and the URL they are viewing. EDIT: to be clear, I was talking about Apache's mod_status. After loading it, you can issue (in a terminal) the above command to have a detailed dump of Apache status.

If you detect a pattern (eg: too many connection opened from a single remote IP), you can use fail2ban to block the requesting client.

edited Jul 20 '15 at 18:11

answered Jul 20 '15 at 16:51

shodanshok

44,038
6
98
162

I went ahead and ran this command and all it did was give me a print out of the home page text? – Starboy Jul 20 '15 at 17:53
use [mod_status](http://httpd.apache.org/docs/2.4/mod/mod_status.html) to get the apache scoreboard @shodanshok was talking about in your browser. – fuero Jul 20 '15 at 18:03

High CPU + Memory, no visitors

2 Answers2