Tuning django based site deployed using apache wsgi for better performance

Question

I'm running django application using apache+mod_wsgi, I have setup 6 medium ubuntu servers on Amazon ec2 in which 2 for mongo and 4 for django and apache, using ebs for postgresql and configured a load balancer around all apache servers, each server is having 4 GB RAM, but during heavy load the system get slow, I saw using top command that apache is using 7 processes and eating approx 2.4 GB of RAM. Below is the apache MPM prefork module config

<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

and this is for wsgi daemon process

WSGIDaemonProcess example user=abc group=abc processes=2 threads=25

then I alter MaxClients to 12 by following this article http://fuscata.com/kb/set-maxclients-apache-prefork, but I'm getting "Request Header Read Timeout" error in apache error log. Please help me in tuning of apache for better performance.

Answer 1

The configuration you use really depends on the specifics of your particular Python web application. Without proper monitoring it will be very hard for you to optimally tune your Apache and mod_wsgi configuration.

In just what you have explained so far I can see various potential issues.

The first is that you are aiming to use mod_wsgi daemon mode, but are you? You don't give the full configuration and WSGIDaemonProcess directive alone doesn't mean you are actually using daemon mode. If you have not setup daemon mode properly, you will consume lots of memory very quickly with that configuration, as the web application could well be running in the Apache child worker processes and not in mod_wsgi daemon mode processes.

A second issue if you aren't using daemon mode properly is that you min/max settings will cause process churn. This will cause higher CPU load from constant reloading of your application as traffic volume goes up and down in short order.

A third is that even if you are using daemon mode properly, have you disabled Python from running in the Apache child worker processes to save on memory and CPU? Why are you even using prefork MPM and not worker MPM which is much better if you are using mod_wsgi daemon mode.

A fourth is that depending on balance between CPU bound and I/O bound tasks in your web application, a 2 processes and 25 threads could be quite bad. Using just 3 processes with 5 threads each may well be better, but not knowing what you application does, is really hard to say, monitoring is really needed.

Anyway, I could keep going on as that isn't all the potential issues I can see, but one is really guessing without real performance data to gauge what you should do.

All I can suggest you do is have a read/watch of the following:

• http://code.google.com/p/modwsgi/wiki/CheckingYourInstallation#Embedded_Or_Daemon_Mode
• http://lanyrd.com/2012/pycon/spcdg/
• http://lanyrd.com/2013/pycon/scdyzk/
• http://blog.dscpl.com.au/2009/11/save-on-memory-with-modwsgi-30.html
• http://blog.dscpl.com.au/2012/10/why-are-you-using-embedded-mode-of.html

Consider installing some monitoring and if you still don't know what to do, use the mod_wsgi mailing list to get help as explained in:

• http://code.google.com/p/modwsgi/wiki/WhereToGetHelp?tm=6#Asking_Your_Questions

The StackOverflow sites may be okay for simple answers pandering to the TLDR crowd, but they are a useless forum for a proper discussion of a topic like this as there is no simple answer one can give in two sentences.

Answer 2

We've had a similar situation, albeit in a simpler setup, and we've resolved the performance issues by using a higher number of processes in the WSGIDaemonProcess directive.

See "How many processes should I specify in a WSGIDaemonProcess while running Django through mod_wsgi?" for an in-depth discussion.