How many processes should I specify in a WSGIDaemonProcess while running Django through mod_wsgi?

Question

Let's say I have 2 sites(Superuser and Serverfault) running from their own Apache virtual host on one box. The 2 sites are powered by Django and are running on Apache with mod-wsgi. A typical configuration file for one of the site will look like the following:

WSGIDaemonProcess serverfault.com user=www-data group=www-data processes=5

The host is a linux machine with 4GB of RAM running Ubuntu. Can anyone suggest the number of processes I should specify above for my 2 sites? Let's assume they have the same traffic as the actual Superuser and Serverfault sites.

Answer 1

Well, how much traffic do the actual Superuser and Serverfault sites have? Hypotheticals aren't much use if they don't have enough info to make the answer easier...

Your worst-case process count should be the peak number of requests per second you want the site to be able to handle, divided by the number of requests per second that one process can handle if all those requests are made to your slowest action (so the reciprocal of the processing time of that action). Add whatever fudge factor you think is appropriate, based on the confidence interval of your req/sec and time measurements.

The average case count is the same, but you divide the req/sec by the weighted mean of your requests per second figure for each action (the weight is the percentage of requests you expect to hit that particular action). Again, fudge factors are useful.

The actual upper bound of how many processes you can run on the machine is dictated by the upper amount of memory each process takes; spool up one process, then run a variety of memory-hungry actions (ones that retrieve and process a lot of data, typically) against it with a realistic data set (if you just use a toy data set for testing, say 50 or 100 rows, then if one of your actions retrieves and manipulates every row in the table it won't be a good measurement for when that table grows to 10,000 rows) to see what the memory usage balloons out to. You can artificially constrain your per-process memory usage with a script that reaps workers that reach a certain memory usage threshold, at the risk of causing nasty problems if you set that threshold too low.

Once you've got your memory use figure, you deduct some amount of memory for system overhead (I like 512MB myself), deduct a pile more if you've got other processes running on the same machine (like a database), and then some more to make sure you don't run out of disk cache space (depends on your disk working set size, but again I'd go with no less than 512MB). That's the amount of memory that you divide by your per-process memory usage to get the ceiling.

If the number of processes you need to service your peak load is greater than the number of processes you can fit on the box, you need more machines (or to move the database to another machine, in the simplest case).

There you are, several years of experience scaling websites distilled into one small and simple SF post.

Answer 2

womble's answer is awesome, albeit a bit hard to understand and apply for the unexperienced. I'd like to give some empirical numbers, and "simple content" versus "e-commerce" application comparison.

There isn't much material around setting different use cases in relation to their appropriate configuration of mod_wsgi, so I hope it's okay to use a little prose here.

A) CMS Sites & Microsites

We run several customer websites, most of them mainly content sites or micro sites hosting django CMS, some custom forms, and sometimes Celery for scheduled background tasks. These sites aren't hungry for resources, several of them run happily in parallel on a single 4 Core Intel Xeon with 32 GB RAM. Here's the configuration we use for each of this kind of sites:

WSGIDaemonProcess example.com user=www-data processes=2 maximum-requests=100

I'm talking about roughly 40 sites on a single server, most of them with their Staging site running in standby. With 2 processes (having 15 threads each, by default) the sites are well-off, albeit limited in their capability of allocating server resources. Why this setup is sufficient can be justified with the simple nature of the (CMS) application: No request is ever expected to take more than a couple of milliseconds to complete. Apache will always stay relaxed, and so will be the CPU load.

B) E-Commerce Sites

More complex sites we do are characterized by still computationally inexpensive local operations but external dependencies (e.g. web services providing booking data) that are expensive in terms of transaction time. Operations with external requests occupy threads for much longer time, so you need more threads to cater the same number of users (compared to a simple CMS site from above). Even worse, threads are occasionally blocked when an external service can't answer a request immediately, sometimes for a couple of seconds. This can lead to the unpleasant side-effect that threads placing requests to the same service queue up, until all available mod_wsgi threads are used up and blocked waiting.

For those scenarios we have tried to use 6 processes without seeing much difference, and we ended up with 12 seeing an incomparable boost in performance and operational stability:

WSGIDaemonProcess example.com user=www-data processes=12 maximum-requests=100

Some simple load tests with 150, and 250 parallel users are easily handled by the site staying well responsive (while with 2 processes the site is unusable catering 50 users in parallel). The 2 CPU 6 Core Intel Xeon with 32 GB RAM runs well below 25% CPU usage under that load, RAM usage almost stays constant at less than 25%, too. Note that we use a dedicated machine just for a single site here, so we won't steal resources that other sites may need.

Conclusion

Using a higher number of processes is a trade-off between allowing Apache to make use of available system resources or not. If you want to keep a stable server system (not website!) under "attack" conditions keep the number low. If you want Apache to help you out using system resources (CPU, RAM) when needed choose a higher number. How high you can go calculates somewhat like outlined in the accepted answer above, and is ultimately constrained by the available CPU power and RAM.

(P.S.: I keep the ConfigurationDirectives section of the modwsgi project wiki under my pillow for Apache-like background reading. Also be sure to understand and monitor your Apache server's open connections.)