How do I prevent lighttpd from caching static files, even when modified on disk?

Question

I am using lighttpd to serve static files. I have a bunch of images in a dir that I regularly update. This will change the file content (and filesize) as well as the modification date, but not their filename.

When I access the files through http, the updates are not taken into account and lighty serves the old file. I can manually rename the file to something different, then lighttpd will return a 404 error, and if I rename my file back, I will get the correct updated version. Seems like lightty is using some kind of cache mechanism of its own (which is fine) to return static files. Unfortunatly, it seems that this mechanism doesn't update itself when files are modified.

I checked through Wireshark, and my browser is really doing a request to the file, this is not a browser caching issue. It returns a 200 OK when requesting it from an empty cache, and a 304 Not Modified otherwise, as expected. But the file is returned with a wrong Last-Modified header that do not reflect the real last modification date.

Maybe there is some config directive that I am not aware of ?

I would like the files returned by lighty to reflect the changes made on disk directly, or at least being able to invalidate its cache.

Update for anyone following this question : I found a culprit. If I update a static file, Lighty does not return the new content, but does return the new Content-Length in its headers, resulting in garbage displayed. If I compress the file using mod_compress, the issue goes away as mod_compress uses its own caching system. Unfortunatly, I can't compress all the files (image files for example). So it is only a partial fix, but I'll get back to it later and will hoepfully find a solution.

score 8 · Accepted Answer · answered Dec 05 '11 at 14:50

I've finally found the issue. And it comes from VirtualBox.

When editing a file in the Host (Win), lighttpd in the guest (Linux) does not correctly update the file content (but does correctly update file size), thus returning cropped or garbled content.

Unmounting my shared drives and re-mounting them, or editing files directly in the guest, fixed the issue.

It took me 6 months to finally figure that out.

score 3 · Answer 2 · answered Jun 23 '11 at 04:33

3

You don't mention whether you have mod_cache installed or not? This module defaults to 'enabled' when installed.

I'm loathe to suggest it, but does turning on Etags help?

answered Jun 23 '11 at 04:33

agy

206
1
1

mod_cache isn't installed. ETags are enabled (but inode is not used to generate the ETag). I tried enabling inode, or disabling ETag, but to no avail. – Pixelastic Jun 23 '11 at 08:13

score 2 · Answer 3 · edited Nov 18 '11 at 06:50

2

Try setting stat engine caching to 'disabled':

server.stat-cache-engine = "disable'

edited Nov 18 '11 at 06:50

quanta

50,327
19
152
213

answered Jun 21 '11 at 15:10

Aleksey Korzun

276
1
4

Thanks, but this has no effect. However, I didn't know that directive, and it might come in handy later. – Pixelastic Jun 21 '11 at 15:19
There might be a middle man proxy between you and the server? Try restarting your server and accessing the same file. Are you using mod_compress? – Aleksey Korzun Jun 21 '11 at 15:24
I'm running a Ubuntu VM in a Windows 7 host. Lighty is in the VM. I don't think there might be a proxy issue here. I've restarted the server, but this does not clear lighty cache. I'm using mod_compress but not on those files. I'll try restarting the whole VM and disabling mod_compress to see if it changes anything. Thanks for the ideas. – Pixelastic Jun 21 '11 at 16:30
Hmm, I might have something here. If I change the file to a smaller one (but keeping the same name), I got only the top half of my file. Seems like the old file is displayed with the content-length of the current one. If I replace with a bigger file, the whole (old) file is displayed. Seems like the filesize changes are taken into account but not the file content. – Pixelastic Jun 21 '11 at 16:38
Sorry for spamming comments : disabling mod_compress does not change anything, nor does restarting the whole VM. – Pixelastic Jun 21 '11 at 16:42
Are you positive your browser is not caching the request? Unfortunately that's the only thing I can think of. – Aleksey Korzun Jun 21 '11 at 21:05
Yes, I tested with both Chromium and Firefox, and checked the network traffic through Wireshark : I can see outgoing requests. – Pixelastic Jun 22 '11 at 09:12
The files that you are serving are not mounted via NFS right ? – Aleksey Korzun Jun 23 '11 at 16:08
Well, the files are on the host machine (win7), but I access them in the guest /var/www/ through a mounted shared folder. I'm not sure this can cause the issue because I had the exact same issue with my production servers (where files are physically on the same drive). – Pixelastic Jun 24 '11 at 07:51

score 2 · Answer 4 · edited Apr 01 '13 at 08:50

2

This lighttpd option worked for me

server.network-backend = "writev"

edited Apr 01 '13 at 08:50

user9517

114,104
20
206
289

answered Apr 01 '13 at 08:47

danelips

121
1

Worked like a charm for me, on a Debian VM on a Debian desktop, thanks! – Yvan Jan 24 '14 at 20:35

score 1 · Answer 5 · answered Jul 02 '11 at 03:03

1

Try these etag entries and see if they help you any:

Cache control rules for lighthttpd

answered Jul 02 '11 at 03:03

Jodie C

733
6
9

How do I prevent lighttpd from caching static files, even when modified on disk?

5 Answers5