5

Our site has a number of large PDF and MP3 files which we would like to cache in Varnish as static files. Currently we don't do much special - simply remove the cookies in vcl_recv and set resetp.ttl = 100w; in vcl_fetch.

The problem seems to be when one of these files is requested (maybe by older browsers) and it's not already in the Varnish cache. There is a delay while Varnish downloads the file from the backend. My understanding is that it doesn't start delivering to the client until the data is fully loaded. This may take 20 seconds or so and sometimes Adobe Acrobat or the MP3 plugin get confused.

Is there a way to both pass the content directly while download and save it in the cache for the next matching client request?

Nic Cottrell
  • 1,282
  • 16
  • 31

3 Answers3

1

Reading suggestion: https://www.varnish-software.com/blog/http-streaming-varnish

With vanilla Varnish 3.0 you could set beresp.do_stream to true in vcl_fetch. The file will be streamed while it is fetched from backend. The caveat: The object is set to busy while streaming, so other clients will be put on hold.

There is also a Varnish 3.0.2 release with baked in "full" streaming support available: http://repo.varnish-cache.org/test/3.0.2+streaming/ which people are using in production, so maybe you want to give it a try.

Or maybe you could "prewarm" your files, placing them in the Varnish memory cache before users start downloading it?

neovatar
  • 176
  • 3
  • How do you do prewarming? – drumfire Oct 06 '12 at 13:54
  • Normally it should be enough to just request the files (e.g. via a script). How long the files are cached in memory depends on your varnish/memory setup. In my old company on patch day we used a script to curl the most visited URLs to have them in cache before bringing the server online again. – neovatar Oct 10 '12 at 16:08
1

The 3.0.2+streaming branch is not up-to-date anymore and it is only supported for Varnish Plus subscribers, no community support on that.

So if you need streaming capabilities you can either:

  • Try Varnish Plus (only available to subscribers) which has built in support for it.
  • Test and see if the upcoming Varnish Cache 4.0 (release expected in Q2 2014) solves this issue for you (it should).
Michael Hampton
  • 237,123
  • 42
  • 477
  • 940
user214397
  • 11
  • 1
0

Varnish 5.1.2 does exactly what you want with the default config.

I tried it with server sent events where I need streaming first request and cached second request. It just works.

wiktor
  • 121
  • 2