High load on X3220 Quad Core Linux Apache server

Question

I'm seriously in need of help. My sites are now nearly impossible to use because of massive loads on my server. I'm already a month late on my mortgage and this really isn't helping my situation. I've been working on fixing this intermittent load problem for months (never this bad).

I'm suspecting some kind of attack since I'm under DDOS attack a lot! I've been trying to figure out what is causing the load but I'm afraid I just don't have the experience or knowledge to understand all the data I've been looking at. I don't even know where to begin or how to test for the large array of attacks out there.

Here's some data you might find useful...

Server: Xeon X3220 Quad Core 2.4 GHz - Linux, FreeBSD 500 GB HD and 8 Gig of Ram. Runs Centos release 5.7 Server Version: Apache/2.2.21 (Unix) mod_ssl/2.2.21 OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 mod_qos/9.74

Warning: All sites are softcore adult sites - mostly fantasy art like elves and amazons.

1) Sites may run fine for weeks or just days at less than 10 load then start jumping to 40-80 load - no idea why. Same sites, same mods, same amount of traffic - just WHAM!

2) I get an email almost every day that says: "Large Number of Failed Login Attempts from IP (different each time)". My webhost (who almost never helps me) told me it was a udp flood or something.

3) I've changed the port for MySQL from the default. If I ever put it back to the default - I get Loads of over 100 from what must be a constant mysql port flood.

4) I've reconfigured MYSQL. Link: http://www.deadlyamazons.com/logs/mycnf.txt

5) I have 3 Joomla Jomsocial networks. I've spent a couple weeks turning all the mods/plugins off, waiting a day and then turning them back on the next day or later if there isn't any change (there hasn't been). For example, on Thursday I'll turn off videos, on Friday I'll turn off chat.. etc and nothing changes the load appreciably.

6) Joomla info: All SEF turned off - sh404sef completely disabled and removed. Components: Joomla 1.5.22, Jomsocial 2.0.5, Kunena 1/31/2011, HWDMediashare 11/22/2010 and JBolo Chat 2.7.3, Comet Chat or Envolve Chat. Page Compression is on, Cache is on 15 mins.

Please click on this forum to see links to all my reports: http://forum.joomla.org/viewtopic.php?f=433&t=706035&p=2777500#p2777500

4/9/12 - Added this part:

Hi guys I'm back with some more info about my poor server. The server is currently limping along with a load of between 20 and 60 averaging about 30.

I'll add an incentive to solving my problem: $100 via Paypal for an answer that solves the 'load' problem without the suggestion of buying 1 or more extra servers. Again, these sites worked fine with even higher traffic on a lower powered server.

I just recompiled apache 2.22 adding eaccelerator and zend-optimizer - no change. The other mod I included was QOS which keeps the # of connections at a lower level. I've had QOS working for awhile.

Suggestions and Requests:

Yes I did turn off the port to MySQL I should've mentioned that.

Traffic stats:

March bandwidth: 579.19G

KBytes Mar 2012: 3,194,134,948 | Dec 2011: 3,504,864,832

Visits Mar 2012: 920,619 | Dec 2011: 727,843

Pages Mar 2012: 10,231,430 | Dec 2011: 10,830,700

Files Mar 2012: 89,218,232 | Dec 2011: 102,862,958

Hits Mar 2012: 106,515,577 | Dec 2011: 120,884,007

Videos of Top -C during high load: Here are AVIs of 2 'Top -c's that I took when the server was running between 30 and 40 load.

Download 1.5 minute / 30M clip: http://www.mediafire.com/?yk3b5xota7l7s30

Download 30 second / 10M clip: http://www.mediafire.com/?4c2t37i8gmd189w

Videos of MySQL Processlist during high load Here are AVIs of 2 'Show Processlist' inside CPanel when the server was running between 60-30 load.

Download 2 minute / 40M clip: http://www.mediafire.com/?ymmfe8599bx11ho

Download 30 second / 10M clip: http://www.mediafire.com/?e675p3p1f0l65jt

DStat stats: 4 Sceeencaps taken in a row... Links:

http://www.deadlyamazons.com/logs/dstat01.jpg

http://www.deadlyamazons.com/logs/dstat02.jpg

http://www.deadlyamazons.com/logs/dstat03.jpg

http://www.deadlyamazons.com/logs/dstat04.jpg

Stats placed on Joomla Board that couldn't be seen earlier:

netstat -alntp | grep :80 | wc -l (1586)

netstat -n | grep :80 | grep SYN |wc -l (30)

netstat -anp |grep .tcp\|udp. | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n (nothing)

netstat -alntp | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n - Report: http://www.deadlyamazons.com/logs/netstat_alntp_awk_print_5_2.txt

netstat -alntp | grep :80 - http://www.deadlyamazons.com/logs/netstat_alntp_grep_80_2.txt

Top (command line) http://www.deadlyamazons.com/logs/top01_cli.jpg Top 2 (command line) http://www.deadlyamazons.com/logs/top02_cli.jpg Top (WHM) http://www.deadlyamazons.com/logs/top01_whm.jpg Top 2 (WHM) http://www.deadlyamazons.com/logs/top02_whm.jpg IOStat (command line) http://www.deadlyamazons.com/logs/iostat.jpg Daily Process Log (WHM) http://www.deadlyamazons.com/logs/daily_process_log.jpg Process Trace MYSQL (txt) (HUGE!) http://www.deadlyamazons.com/logs/trace_mysql.txt Process Trace MYSQL (rtf) http://www.deadlyamazons.com/logs/trace_mysql.rtf Process Trace sxyamzn (txt) http://www.deadlyamazons.com/logs/sexyamazonscom_indexphp.txt Process Trace sxyamzn (rtf) http://www.deadlyamazons.com/logs/sexyamazonscom_indexphp.rtf Process Trace sleepps (txt) http://www.deadlyamazons.com/logs/sleeppeepscom_indexphp.txt Process Trace sleepps (rtf) http://www.deadlyamazons.com/logs/sleeppeepscom_indexphp.rtf

Any help would be appreciated.

Answer 1

Here's the first thing I can see going wrong. Try to avoid having MySQL and Apache on the same server. Here's why.

Based on your provided configuration, which I squirted into a MySQL usage calculator

Session variables
max_allowed_packet 4.0 MB
sort_buffer_size 3.0 MB
net_buffer_length 16.0 KB
thread_stack 192.0 KB
read_rnd_buffer_size 8.0 MB
read_buffer_size 2.0 MB
join_buffer_size 96.0 MB
Total (per session)113.2 MB
Global variables
innodb_log_buffer_size 1.0 MB
query_cache_size 96.0 MB
innodb_buffer_pool_size 1.0 MB
innodb_additional_mem_pool_size 1.0 MB
key_buffer_size 384.0 MB
Total 483.0 MB
Total memory needed (for 300 connections): 33.6GB

You've got 8GB. So.. You're gonna be getting pretty swappy pretty soon. Your server will use up all its free RAM with MySQL, because you've told it to. Everything else will go into Swap space. On disk. Slow. IOWait will go through the roof, causing processes that need disk access to wait on it. That means processes will be blocked. That means that the Load Average will skyrocket too.

Joomla, in it's standard, untuned form is a bit of a beasty. How many sites have you got on this one server? It could be that it just can't cope. Do you have a layer like Memcached between the database and the webapplication? (That'd need more RAM alone, probably). Coppermine's also a bit of a CPU whore. Quite a lot of cycles are needed for a simple gallery load (or were, last time I used it).

Suggestions:

1. Try and grab a measure of IOWait when the load's spiking. Dstat is good for this.
2. Tune your MySQL limits down to what you actually need, rather than these enormous buffer sizes and caches.
3. Consider obtaining another server, or tuning your MySQL utilisation limits down. While it's not beyond the realms of possibility to have a 36GB+ Database server, it might not be required, as there's no indication of expected load.
4. Alternatively, consider just separating the Web and database servers, so that each can be more performant at doing it's own job, rather than both.
5. If you've got lots of static content, you might want to look at Varnish as a reverse caching proxy. 
6. Install APC or eAccelerator, both PHP Opcode caches which can massively reduce the amount of RAM used by PHP
7. Take a look at how you're running PHP. You might find that PHP-fpm gives you better memory performance. If you're really insane/crazy/suicidal, you could try precompiling the PHP with HipHop.

Answer 2

In addition to Tom's answer there might be a problem with php too, it is generating an incredible amount of load. Sometimes in Joomla the culprits are addons. Check if you installed new addons and use debug mode to see if there are problems with them. The main reason they cause load is because they query your database a lot. So please, as Tom O'Connor suggested, tune your DB.

Also have a look at eAccelerator for php. It caches your pages, so serving them becomes a lot quicker. This means more memory usage though.

And seriously, run a firewall, if it doesn't need to be externally accessible, then close it off (drop not reject).

You can try (and I'm not promising results) to run OSSEC to block brute force break in attempts. I use it and it works very well. If they are doing a TCP sync or UDP flood attack, the only thing you can do is to request the ISP to drop the traffic before it reaches you.