13

I have a django app and I want to to setup Varnish on a server in front of it. In another serverfault thread somebody suggested putting Nginx in front of Varnish.

Should I put Nginx in front of Varnish on the caching server? If so, should I use Nginx on the app server?

Community
  • 1
Enrico
  • 491
  • 1
  • 6
  • 15

4 Answers4

10

We are talking 1 - 3 frontend servers in total, not a large server farm with load balancing between the tiers?

Putting nginx in front of Vanish enables you to do HTTP compression on the fly. That is a performance best practice, but it could be dispensed with. (Content in Varnish is often kept uncompressed, so that ESI Includes work, and so you don't have to deal with multiple cached versions of the same object depending on Vary header / browser matching.)

Regarding nginx on the app server -- is Apache with mod_wsgi not the recommended and most common way to deploy new Django installations nowadays? I'm not aware of a compelling reason for using nginx/fastcgi over Apache/mod_wsgi for Django; but you should get advice from a Django expert.

Regarding Varnish having attractive load balancing features that nginx doesn't have, I don't see what they are? Varnish has random and round-robin balancing. nginx has round-robin, client IP and consistent hashing -- I don't see a significant benefit for Varnish? Is it VCL or Varnish' graceful config reload or something else?

For a small 1-3 server setup I guess I would just do

Varnish --> Apache / mod_wsgi / Django

or maybe

Squid --> Apache / mod_wsgi / Django

and ignore HTTP compression for simplicity, unless bandwith is expensive.

Update:

Graham Dumpleton has written a valuable comment below. He mentions a very common setup for for example a blog on a VPS, or a small web farm without caching:

nginx --> Apache / mod_wsgi / Django

This is a very good solution, for a couple of reasons:

  1. Simple setup
  2. nginx, which has high speed and minimal overhead, handles static file serving and browser connection keepalive.
  3. Django runs in Graham Dumpleton's excellent mod_wsgi, the recommended platform for Django.

The reason I didn't mention this initially is that OP seemed to require Varnish, a very high performance caching solution. The nginx / Apache / mod_wsgi combo can not do caching with a level of performance and flexibility that matches Varnish.

  • 2
    You could even use 'nginx --> Apache/mod_wsgi/Django' as many people do for various good reasons. – Graham Dumpleton Nov 05 '09 at 20:15
  • 4
    The other thing that nginx provides in that use case which many wouldn't realise is that nginx isolates Apache from slow clients. This is because request content up to certain size isn't streamed by nginx. Instead it will buffer up request headers and content and only proxy request when it has it all. This means it only hands it off to Apache when all information needed to handle request is available. Thus Apache/mod_wsgi will be busy for least time possible and processes/threads not tied up by slow client. A measure of buffering happens on reverse as well so Apache can finish quicker as well. – Graham Dumpleton Nov 06 '09 at 01:21
  • 2
    @Graham Dumpleton: It is **very** nice to have you here, I hope you will stay around. :-) . Regarding nginx HTTP multiplexing / smart connection handling, I completely agree. –  Nov 06 '09 at 03:47
  • First and foremost, thanks Jesper and Graham for taking the time to write such a comprehensive responses. I am planning on putting a load balancing server in front of two app servers. One server will handle most of the traffic with the second being used mostly for fail over and beta testing new features. – Enrico Nov 07 '09 at 11:28
  • Varnish is appealing because it gives me full control over which requests get sent to which backend. Varnish also health checks backend servers, failover is simple to setup and it handles slow/dead backends gracefully (if both servers die). – Enrico Nov 07 '09 at 11:29
  • I probably have enough bandwidth at the moment that I dont need nginx in front of Varnish. As for the app servers, it sounds like there wouldnt be any great advantage to having nginx so I'll stick with Apache/mod_wsgi. Thanks again for all the advice/thorough explanations. – Enrico Nov 07 '09 at 11:31
  • _The nginx / Apache / mod_wsgi combo can not do caching with a level of performance and flexibility that matches Varnish._ Yes, nginx can't, it's just performs better. http://todsul.com/nginx-varnish – VBart Sep 13 '12 at 00:12
4

You can use nginx without varnish to proxy and cache the content.

silent
  • 141
  • 2
4

I've being using Nginx, Varnish, and Apache/mod_wsgi/Django successfully. I started with the following config:

Nginx -> Apache/mod_wsgi/Django

Once I started seeing significant load on Apache, I added Varnish:

Nginx -> Varnish -> Apache/mod_wsgi/Django

I use Nginx as a kind of "URL router". Django admin requests are sent directly from Nginx to Apache. Client requests are sent from Nginx to Varnish which caches the requests from Apache and also serves "graced" items from the cache if the app servers are unavailable.

My Nginx server also serves certain static content directly (e.g. images, CSS, and javascript files).

In general, performance has been excellent. I have noticed a couple of caveats that I should mention:

  1. On a busy site, restarting Varnish can cause the load to spike on the app servers, so it's best to keep Varnish restarts to a minimum. (Varnish doesn't seem to have a "reload" like Nginx/Apache where it just rereads its VCL files). Conversely, reloading an Nginx configuration has minimal impact. For this reason, I do most of the URL rewriting and "routing" in Nginx.
  2. Varnish is easy to drop in between Nginx and Apache. If you start noticing high load on your app servers, adding varnish with even the default config can really make a difference.
  3. If you do use Varnish, you definitely need to think about how you're going to handle cache invalidation.
  4. My experience has been that Varnish handles failed backends a bit more gracefully than Nginx (as you pointed out earlier).
David Narayan
  • 211
  • 1
  • 1
  • I've never seen varnish behind something else. Varnish is usually the load balancer that does the url router. So you've got 2 webservers and 1 proxy or 2 proxies and 1 webserver?q – ioan Jan 11 '17 at 12:18
2

I'm using Nginx->Varnish->uWSGI->Django

MechanisM
  • 121
  • 2