0

I have the challenge of hosting about 200-1000 mp3 files, all in the 100MB+ size range.

Furthermore, there are some smaller RSS files, and some smaller JPG files.

All this is static content, no PHP, or any sort of scripting. There will be no HTML hosted either, nothing has to be HTTPS, no user data is stored on the server.

These files are non-copyrighted podcasts, which are produced by us and free to get on itunes and everywhere, discoverable via rss.

Until recently these files where located on a cheap hosting plan at godaddy, but because of the immense traffic we have no other option than to host these files elsewhere.

In the past I only used Apache for all my hosting needs, but I have the suspicion that apache would be not the ideal solution for these requirements, and because the server is a bit on the slower side, and does not have that much RAM, I am wondering if a different server would be better for these requirements.

Which server would you recommend? I was hoping there would be something which would understand that one file is in high demand, for example when a new episode comes out, and would put it in a RAM cache. Can NGINX be used in that way? Should I use Lighthttpd?

MikeyB
  • 38,725
  • 10
  • 102
  • 186
Carsten
  • 1
  • Recommendation questions are hard to definitively answer. It looks like you've got some research to do to skill up in hosting architecture. – Drew Khoury Jun 17 '14 at 10:26
  • Oh, I was hoping that someone would have been in a similar situation in the past. I have a lot of experience with apache configuration, and used lighttpd only for smaller side projects / testing in the past. I made the usual google fu with ngingx vs. apache, and read a bit about it. What I was hoping from this question was that someone would tell me, that my ambition, or underlying ideas are wrong. But I have the feeling that nginx is the way to go. I read a lot about memory issues with lighttpd, and this is quite a limitation I have. So is there anything that speaks against nginx? – Carsten Jun 17 '14 at 10:58
  • It's almost matter of taste choosing server for static files. With [sendfile (2)](http://linux.die.net/man/2/sendfile) available it's kernel do all the work. – Alexey Ten Jun 17 '14 at 12:16

3 Answers3

2

I wouldn't expect there to be much difference, if any, between Apache/Nginx/Lighttpd in this case. I would expect Apache to be slightly more memory heavy assuming you trimmed down all the modules that you don't need (which is likely most of them). My personal choice would be Lighttpd simply because I'm more familiar with it for serving static files and I've never any issues with it despite relatively high loads on it.

As for the RAM caching question...the OS should do that automatically assuming you have enough RAM available. You can test this by timing how long it takes to get N different files versus getting the same file N times though there may not be much difference. Seeing that your files in question are relatively large I would definitely not skimp on your server's RAM if you have a choice. Chances are in your case you will be limited by network bandwidth before anything else.

As with many questions of this type the best answer is test, test, and test some more. It is quite easy to get all three setup on the same server (under different ports) and then do some basic benchmarking with a tool like ApacheBench (ab, comes with Apache) or something similar. Check your server's load/memory usage, compare available request rates, and see which one works best for your situation.

uesp
  • 3,384
  • 1
  • 17
  • 16
2

This question reeks of premature optimization.

I was hoping there would be something which would understand that one file is in high demand, for example when a new episode comes out, and would put it in a RAM cache. Can NGINX be used in that way?

That's nothing you even need to worry about - it'll get cached by the OS and put into RAM the very first time it's requested. It'll stay in RAM since everybody will be asking for it.

Set up something simple and easy (apache/nginx) and let 'er rip.

If you need help serving the data (especially given that the server doesn't have much RAM for cache) put a CDN (i.e. Cloudflare) in front. Actually, since Cloudflare has a free tier, put that in up front!

MikeyB
  • 38,725
  • 10
  • 102
  • 186
0

Roughly the options you have are:

  1. Continue with what you were doing before but increase your own broadcasting capacity by upgrading to a bigger hosting plan.
  2. Get a server with a dedicated, unlimited 100/1000/10000 Mbit uplink allowing you to move from a generic hosting solution to something more customized to your use-case.
  3. Off-load your broadcast requirements to a content delivery network (CDN). This means you can probably stay on the current hosting plan because the majority of traffic will be served by the CDN and you don't have any changes in your work-flow either.

The first is often not very cost efficient, many generic hosting plans only work well with many small, relatively low traffic sites and large sites are not suitable for that business model, so often pricing is designed to drive you away once your reach a certain limit.

The second is not necessarily the cheapest, but allows you a lot of flexibility if you can afford to invest the time to acquire the needed skills and to test your proposed (software) configurations. Typically the fastest data comes from memory, then SSD, SAS and SATA disks lag behind but are the cheapest storage as well. Something that caches the most popular/current files in memory should be the fastest (e.g. Varnish), although Linux typically uses any unused memory as a disk cache anyway and any light weight webserver should suffice for static content.

Even when you don't use a free CDN like cloudflare a CDN might work out quite a lot cheaper then options 1 & 2.

HBruijn
  • 72,524
  • 21
  • 127
  • 192