5

So I have a client with a very high bandwidth server. The hits its serving are pretty light in terms of demand - static jpeg and text.

SERVER SPECS:

  • Dual Quad-Core L5520 2.26gHz (3.2 Turbo & HT) 8MB cache
  • 72GB Ram
  • 2 300GB SSD's (1 dedicated to /usr to keep up with logging)
  • 100Mbps (unmetered) port at a very reputable datacenter

Ive currently got apache configured as follows:

  • Start Servers 512
  • Minimum Spare Servers 256
  • Maximum Spare Serves 512
  • Server Limit 7500
  • Max clients 7500
  • Max Requests per Child 100
  • Keep-Alive Timeout 5
  • Max Keep Alive requests 768
  • Timeout 30

I was seeing about 2000-3000 apache access's per second last week with no issues. That translated to 80-90Mbps up and a load around 1 with peaks around 5 and momentary extreme peaks of 30-50 when apache would restart and all the processes re-spawn. Memory usage was about 40GB. Traffic has increased and I'm now seeing 3000-4000 apache access's per second and the http is unstable. Although apache is occasionally restarting, I am getting periods of no http connectivity while apache is running and there are idle workers and open slots. Load is around 2 when stable and frequently spiking which no doubt is my connection issues, but I'm not sure why. RAM usage is 50-60GB. Throughput is about 130Mbps. (I'm not sure how I'm getting over 100Mbps when my package is a 100Mbps but I've got a ticket in with the DC now to inquire if perhaps they are dropping packets as a result of my usage, but that doesn't explain my stability issues unless ack's are dropping and leaving connections to time out... maybe... not sure... never thought about the theory behind that thought till now)

I'm looking for opinions on if I have reached the operational max of what apache can handle or is there some other system limit that I may be exceeding.

Some people have told me LiteSpeed is the solution but I have no experience with it, and my client has heard bad things so opinions there are appreciated as well.

Thanks All!

voretaq7
  • 79,345
  • 17
  • 128
  • 213

1 Answers1

13

First off, Apache, nginx, or LightSpeed aside - if you're running a server with 2000-3000 requests/second it's time to start thinking about dual servers and load balancing. Depending on what you're serving you can easily get more out of any of those servers, but at those rates you're serving something important (or at least high-traffic), so you want redundancy in addition to the ability to handle momentary load spikes.
Start seriously considering a load balancing infrastructure (there are plenty of questions/answers about HAProxy and the like here, and Google can help you as well).

That said, I don't believe you've hit the operational limits of Apache.
You can certainly consider other high-performance web servers (nginx is very popular), or you can consider tuning your Apache configuration for better performance.

Some Apache suggestions based on what I see in your question:

Before doing anything else, read the Apache performance tuning documentation.

  1. MaxRequestsPerChild is really only useful for containing resource leaks.
    100 (your current value) is absolutely insane. You're churning processes which kills performance.
    0 (Never kill a child) is certainly viable if all you're serving are static resources.
    10000 (ten thousand, the default) is fine in almost all circumstances. 50000 (fifty thousand) is what I use for pure static HTML sites.

  2. StartServers, MinSpareServers and MaxSpareServers can be tuned.
    I generally set StartServers and MinSpareServers to the same value.
    If there is a specific minimum number of spare servers you want to keep around, that is the number you should start with. A good value for this is your low-water-mark of simultaneous active connections.
    MaxSpareServers should be set to 75-80% of your high-water-mark of simultaneous active connections.

  3. ServerLimit and MaxClients can possibly be increased.
    If you have lots of free RAM and lots of free CPU, increase these numbers.
    If you're running close to resource saturation, leave them as-is.

  4. Use graceful restarts
    You say you are seeing "momentary extreme peaks" in your load when Apache restarts.
    This tells me you're probably not using graceful restarts.
    Whatever is causing Apache to restart, have it send SIGUSR1 to Apache rather than SIGHUP (or heaven forbid, actually stopping and starting the entire server). This is far less abusive and disruptive to the system than a regular restart of a full stop/start.

  5. Consider other MPMs
    You are almost certainly using the prefork MPM if you're on a Unix system.
    Consider the Worker MPM instead.
    Tuning for the Worker MPM is a little different

  6. Spend some cache
    Apache has caching modules which can be used to hold frequently accessed data in RAM. This avoids a round-trip to the disk (or at least the filesystem layer) for frequently accessed data.
    Configuring memory backed caching can give you a pretty big performance boost for a relatively small amount of memory.

voretaq7
  • 79,345
  • 17
  • 128
  • 213
  • 1
    Really good answer. I would add the suggestion to use Apache 2.4 with Event MPM, it will allow him to scale with lower resource consumption thatn Worker and is particularly well suited for high-performance static file serving – Daniel Lopez Nov 08 '13 at 22:37
  • @DanielLopez Yes, Apache 2.4 would be a good choice. There's a few roadblocks to adoption still, but for serving static content it should be a non-issue... – voretaq7 Nov 08 '13 at 23:03
  • @voretaq7 I have 15gb ram and 31eCPU server using apache in event mpm. Currently using 24 start server, 100 server limit and 3000 max spare threads. My analytics shows 2000 users in real time. What configurations should I use in event mpm? – lightsaber Jan 29 '16 at 18:45
  • @StarWars The best answer I can offer is ["Test it and see"](http://serverfault.com/questions/350454/how-do-you-do-load-testing-and-capacity-planning-for-web-sites) - It depends on what your environment is doing, but if you're not having any problems you can probably defer making changes until your analytics start pointing at something that needs to change… – voretaq7 Jan 29 '16 at 20:08