0

We wanted to switch all our servers from lighttpd to nginx but while the Web server migration went fine, the static files server migration was disastrous.

We ended up with up to 10 seconds of latency for loading pictures, using this config with nginx:

There are 4 sites, with about 120k visitors/day that consume massively streamed videos and picures.

A hint we had was that after rebooting nginx, loading picture was always really fast. The CPU or the memory consumption was always really minimal, so we though about the disk. We tried to tweak the file descriptors limit but it didn't change anything.

We switched back to lighttpd and we are back in business again, with nearly instantaneous picture loading.

We know we did something wrong, but what ? It would be nice to have only to maintain one server set up, and nginx is really easier to set up than lighttpd.

The configuration files you are seeing here are for nginx on a virtual machine dedicated to static files and video streaming (compile with the mp4 plugin).

womble
  • 95,029
  • 29
  • 173
  • 228
e-satis
  • 409
  • 5
  • 17
  • When you "thought about the disk", did you actually measure how hard the disks were working? Unless you're really lucky and someone's just happened to have this exact problem and recognises it, it's unlikely you'll get an answer here. You need to work the problem -- analyse what's going on under hood and determine the cause. strace, tcpdump, all the usual tools are your friends here. A 10 second latency shouldn't be hard to track down. – womble Jul 18 '11 at 12:21
  • Indeed. Since I don't have yet the skills to perfom such a debug, I was hoping that somebody with a similar problem could give me a hint. I'm well aware that I must perform myself an evaluation of the system, but I'm no sysadmin, and therefor, this is going to take a lot of time before I learn how to do so. In the meantime, it could happen to be a common problem and get an aswer here. – e-satis Jul 18 '11 at 12:46
  • It's not a common problem, I'm pretty confident of that. Time to hire you a sysadmin, this is what they do. – womble Jul 18 '11 at 12:59
  • Have you tuned the output_buffers in Nginx? You are asking Nginx to handle the file transfers so you should ensure it has the proper buffers to handle them efficiently. – Martin Fjordvald Jul 18 '11 at 16:52

1 Answers1

1

I had a a somewhat similar experience with nginx recently. We were serving up static files and over time the server would slow down and eventually crash. It turned out nginx was caching the downloads to disk, eventually filling it and causing the server to crash.

Not saying this is your problem and unfortunately I don't have the fix we applied at the moment but perhaps it might set you in the right direction.

  • Thanks. Do you know where this file is ? – e-satis Jul 18 '11 at 13:52
  • In our case we were proxying static files via nginx and setting 'proxy_max_temp_file_size 0;' in our nginx configuration resolved the issue. –  Jul 18 '11 at 16:12