30

I'm running a traffic intense site plenty of dynamic content, mostly user-generated.

The server is a dedicated one and has a total of 4 Intel(R) Xeon(R) CPU X3210 @ 2.13GHz proccesors. I need to know to optimal values for ServerLimit and MaxClients apache's directives, considering that the server has 4GB of RAM and the MySQL database runs on a separate server. The panel is DirectAdmin with CentOS.

Below are my current directives, but during peak hours with over 5k users, an important lag is noticed - and it's not entirey MySQL's fault, because pages seem to be generated fast (I implemented a page generation time counter), but there is a long connection delay until the page starts responding and is sent to the browser.

<IfModule prefork.c>
    StartServers     800
    MinSpareServers   20
    MaxSpareServers   60
    ServerLimit      900
    MaxClients       900
    MaxRequestsPerChild  2000
</IfModule>
Timeout 90
KeepAlive On
KeepAliveTimeout 5

I should mention that monitoring the server using the top command, CPU usage never goes beyond 20% ~ 30% on peak hour. The MySQL server also has a 30~50% usage at that time, and I'm constantly working on fixing slow queries, but that's a different issue. I know it's not a DB bottleneck because static pages also take long to load on peak hours.

Any tips to optimize these values will be greatly appreciated, thanks.

Zoredache
  • 128,755
  • 40
  • 271
  • 413
andreszs
  • 709
  • 1
  • 6
  • 16

3 Answers3

25

Your MaxClients is WAY WAY WAY too high. What is the current size of your apache process? Multiply that x 900. Is that greater than 4GB? If so, the machine is likely going into swap. I usually start with MaxClients = 2x vCPUs in the box (grep -c processor /proc/cpuinfo). Which in this case would be about 8. Then make sure that MaxClients x apache process size isn't over 4GB.

You can up your MaxClients from there, depending on the type of connection that your clients have. (Dial-up users need to be spoonfed, etc.) But make sure you never get yourself into a swapping situation.

Then set your Min, Max, and Start servers to MaxClients. There's no real need to have them differ in a dedicated server environment.

Then do some testing with ab (as goose notes.)

toppledwagon
  • 4,215
  • 24
  • 15
  • For some reason it seems I determined the process size wrongly... now I see in the top command that the RESIDENT SIZE apache processes range from 10 to 15MB. I read somewhere that since shared libraries are included in this number, the "real" size is half that size. Considering this I should calculate that I can accomodate 570 processes of 7 MB size each, do you think this is correct? – andreszs Aug 25 '09 at 12:50
  • I suggest you use 15 MB for the calculations and then start to check the metrics: # http process vs. memory use. This will give you a better idea to the number of MaxClients – hdanniel Aug 25 '09 at 14:19
  • 1
    I lowered it to 400 and even before the peak hour the result had the opposite effect than desired: Any value lower than the original creates timeouts and long delays. In fact now I increased it to 1500 clients and memory usage is now on 3 GB while the average CPU usage is 8%. Of course there is more load on the SQL server now and I'll have to work in that. – andreszs Aug 26 '09 at 00:09
  • Here's my htop command now, 1500 apache process and nearly 100 system processes. This with 75% of RAM usage. http://a.imagehost.org/0011/htop.png Should you reconsider your formula? ;) – andreszs Aug 26 '09 at 00:31
  • 2
    This is why you need to do testing in your own environment. We have http accelerators in front of our apache's so they aren't spoonfeeding mobile users. Your application also appears to be very light. If you've moved the load to your DB that tells me that more of those apache processes are actually serving data vs sitting and waiting for a mysql connection. Which then leads me to ask how many connections are you allowing to your DB? Does that number exceed your MaxClients? Do you have 5,000 simultaneous connections? If so, you might want to look into something like perlbal in front. – toppledwagon Aug 26 '09 at 01:57
  • one thing that i'm also seeing in that htop graphic is that you're load is about 2-3x your available processors. If the majority ofyour processes are waiting on the database, then your load will queue up and you'll continue to see slow responses that are probably now more related to the database than the apache server. – goose Aug 26 '09 at 12:49
  • Max_connections is set to the same value of MaxClients now, 1500. Indeed now all database-intensive pages load quite slower. I'll ask a new question on how to deal with the MySQL server load and consider this one answered properly, thanks. – andreszs Aug 26 '09 at 23:56
  • How should the appropriate value of ServerLimit be determined? – Brad Koch Aug 09 '12 at 21:55
5

I recommend playing around with apache's benchmark (ab) tool. You can play around with the values to match them to your traffic flow and see what type of responses you're getting as far as average load time & the like. At that point you can play around with the settings that you're talking about to try and optimize them. You should be able with ab to get a handle on the optimial performance for each performance tweak.

It wouldn't really be prudent for me to speak about your settings, however you also need to take your RAM into account as well because it sounds like you're eating up a lot of RAM with those settings. Although that's just speculation without any data. htop gives you a good visual read of your resources.

Also your load average could say a lot. I doubt that your usage is much higher than your total amount of cores from 20-30% cpu, but it's another indicator of how hard your server is actually working.

goose
  • 151
  • 7
  • The problem is I'm not experienced enough in server administration so start playing around with settigns and monitor the results.. I never used the ab tool to be honest. Also changing the values requires restarting HTTDP, which causes inconveniences to my users so I prefer to avoid this. Imagine you are sending a message to another user and after you click "Send", you get a Server connection problem. On CPU usage, see the info from my previous comment: it dos not go beyond 15% on peak hour. I think that's quite acceptable considering I had 6000 online users yesterday. – andreszs Aug 25 '09 at 13:14
  • 1
    Well I definitely wouldn't want you to do this in a production environment. I recommend doing this at your lowest traffic time if another server (with very similar if not identical hardware) to test. ab is pretty easy to use, but I definitely think that toppledwagon gave a good instruction on calculating your MaxClients. Once you stop using swap space, you'll see a definite improvement. check http://httpd.apache.org/docs/2.0/programs/ab.html and http://www.cyberciti.biz/tips/howto-performance-benchmarks-a-web-server.html for AB – goose Aug 25 '09 at 17:23
5

You need to obtain the average size of your apache process. With this number and the total size of your RAM you can calculate the MaxClients directive. Remember this: "A webserver should never ever have to swap" (Apache Performance Tuning)

Monitoring with top or htop is ok but you need a better view of all stats of your servers (cpu, ram, disk i/o, apache requests, mysql slow queries, etc ...) with some monitoring tool like ganglia or munin to find possible bottlenecks.

hdanniel
  • 4,253
  • 22
  • 25
  • For the moment I only have the top and htop commands, and I can't get to understand all its info anyway. This is the activity yesterday on peak hour, it seems there is no swapping; please tell me if I'm wrong: Tasks: 1043 total, 2 running, 1041 sleeping, 0 stopped, 0 zombie Cpu(s): 13.8%us, 1.8%sy, 0.0%ni, 82.1%id, 0.8%wa, 0.0%hi, 1.5%si, 0.0%st Mem: 4138360k total, 3961276k used, 177084k free, 75016k buffers Swap: 2031608k total, 1484k used, 2030124k free, 1836600k cached – andreszs Aug 25 '09 at 12:54
  • 1
    Yes, your server is not swapping. I prefer real-life metrics but if you want you can use a stressing tool like ab or httperf to check how much your server can handle. For the tests take care of MaxClients and start with a low number (based on the 15MB assumption). – hdanniel Aug 25 '09 at 14:23