Apache tuning with very high-load LAMP server

Question

I have a LAMP server that can be serving ~400 requests per second at times, the server configuration is:

CPU: Intel Quad Core Xeon X3430 (4 x 2.40 GHz, 8MB Cache)
RAM: 16 GB REG ECC DDR3
HD: 500 GB Enterprise Grade SATA II
OS: CentOS 64 Bit (Latest Stable)

Apache2 prefork config:

StartServers      128
MinSpareServers   16
MaxSpareServers   64
ServerLimit     1500
MaxClients      1500
MaxRequestsPerChild  10000

At peak times the server load is very high, cpu utilization around 90% and load average around 130

I'm not sure it's server hardware limit or my prefork config have something wrong - What could be the problem?

Investigate a CDN, move the database onto a dedicated server, implement caching, etc. Lots of things you can do. — ceejayoz, Jan 13 '12 at 16:59
@Roy If MySQL is already on its own dedicated server, you don't have a LAMP server, you have a LAP server. — ceejayoz, Jan 13 '12 at 17:27
Yes, my server us runing MySQL also, do you mean I should separate the database server? but the server do not have many MySQL query cos many page is cached. — Roy, Jan 13 '12 at 23:53

score 3 · Answer 1 · answered Jan 14 '12 at 04:53

You are allowing Apache to spawn up to 1500 children to serve requests (ServerLimit / MaxClients) -- It's no wonder your server load (the number of processes waiting in the run queue) is getting enormous!

My first suggestions at 400 requests per second with the numbers you're quoting in your question would be "Move the MySQL server to its own box" or perhaps a better choice: "Add another web server and load balance your requests".
To help you figure out how to size the additional server follow the advice HDDP500 gave in their answer to figure the average size of an Apache process. Figure out how many apache processes you want to run on the new server and determine how much RAM you'll need to handle them. Remember to add a safety margin (a gig or two).

Figuring out CPU requirements (speed/number of cores) is a little more difficult - You will need to take into account how much "work" the server has to do to generate each page. A tool like XDebug can help you out here by telling you how long it takes to generate a page (you should perform this test on an unloaded server as well as one under load, but you should also not use a tool like XDebug on a production server in most cases.)

+1 for this. Move MySQL. Get 2+ Apache nodes. Use HAProxy/Varnish to load balance. — Tom O'Connor, Jan 14 '12 at 11:07
you could probably even get away with DNS round-robin for a little while, but a real load balancing solution (I'll throw pf or [pfsense](http://www.pfsense.org) into the ring) is definitely preferable — voretaq7, Jan 15 '12 at 21:04

HTTP500 · Answer 2 · 2012-01-13T17:47:53.290

2

What is the average size of your httpd processes?

Run this command when the server is under load:

ps -ylC httpd --sort:rss | awk '{sum+=$8; ++n} END {print "Tot="sum"("n")";print "Avg="sum"/"n"="sum/n/1024"MB"}'

That will tell you approximately the average size of an Apache process.

Your MaxClients is probably way too high.

edited Jan 13 '12 at 17:47

answered Jan 13 '12 at 17:39

HTTP500

4,827
4
22
31

score 0 · Answer 3 · answered Jan 14 '12 at 04:58

Take a look at your CPU interrupts and see where the bottleneck starts (via top); as mentioned before, It'd be a good idea to dial down your ServerLimit/MaxClients.

Depending on your consumption, this may (should) speed things up as you lessen the ability of the box to starve itself of CPU.

Apache tuning with very high-load LAMP server

3 Answers3