I'm looking to build a server farm for a web business, I will need front end webservice servers and a common NAS file server backend.
Some companies choose to buy the fastest most reliable hardware which costs a premium. Other companies choose to buy lower spec cheap PC hardware but more of it. They build highly redundant systems, which they swap out when they fail. I believe Google takes that later approach.
Has anyone any real life specific experience and advice to offer me?
It depends on your application and tolerance for failures.
If you are running an oracle database for a financial business, you want expensive servers with hot-swappable parts and built-in redundancy (power supplies, disks, even cpu and memory).
If it's a web server or compute servers with NAS storage, go cheap (on the server, not the NAS) as long as you can tolerate the loss of a box without much impact.
Dont go so cheap that you are constantly replacing bad hardware.
The general rule of thumb for me has been to use raid to protect your important disk-based data, but buy cheap commodity hardware for compute and web farms. Get a good load balancer that can detect when a webserver is not responding and mark it offline.
Real life experiences:
Bad: Running oracle on commodity hardware was a cheap solution that we were able to put together very quickly, but a bad CPU fan caused a server crash which forced us to restore Oracle from tape (ugh!).
Good: We replaced 2 high-end heavily redundant machines with 70 commodity rackmount servers. We were able to drop maintenance on the 2 machines and started just buying $2500 'spares'. Over about 2 years, I think we only ever used about 6 of the 'spares' (the real challenge was avoiding deployment of spares for other purposes).
Personally I recommend to my customers that computers that need to be highly-reliable be name-brand computers with current, top-level support.
I call it "Packled-level support" after the alien race in Star Trek, because you should be able to call a support line day or night and say I am $NAME at $COMPANY. My support contract number is $X. My computer is a $TYPE, serial $NUMBER. It is broken, make it go and expect that some guy will show up 2-4 hours later to do exactly that.
Now expect to pay good money for that kind of support. But if you must be up, it's worth it.
For the most part I prefer pre-assembled computers with a warranty and support because it generally isn't worth the customer's money to pay me to spec, order, and assemble cheap computers.
For computers you can stand to be down for a few hours/days, cheaper hardware is probably OK. For example if I have five web servers, I'll want two of them to be highly available, and will tolerate the rest being cheaper. For compute farms I'll go with cheaper hardware.
There is this nice blogpost from Jeff on this topic, where he also looks at the energy and license fees.
From my experience, cheap hardware does cost a lot, too. I'm sure you do not need the high end model of a professional server to serve a few websites. But you don't want to miss
In addition, consider using a good loadbalancer, HTTP-cache (e.g. Squid), PHP-cache (e.g. ACP) and memcachd, if it fits your web application. This may obsolete some backend servers on a reasonably busy site.
Airflow, airflow, airflow.
Find a local vendor, build a reasonably priced machine, research the components so that you're not buying lemon parts, and ensure it is moving copious amounts of air throughout itself. When gear stays cool it lasts much longer. Redundancy is nice but depending on your organizations size it might be overkill. If you do want redundancy then perhaps you should look into purchasing less machines with much more fault tolerance (hdd, cpu, mem, psu, fans, etc) and running virtual servers. That way you would get fault tolerance in your server as well as the added advantage of running virtual machines. However, it's good practice to keep your DB servers to separate hardware or at the very least have them writing to a dedicated raid array.
Siting is definitely one of the most important factors that affect the energy consumption and thus the costs of computing. There are places with cooler climates than in the continental US and with more green electricity available. I would recommend anybody considering siting a datacenter to take a look at the Finnish website on these issues: http://www.fincloud.freehostingcloud.com/
Some of the most expensive hardware has some of the cheapest 4-hour response contracts.
The most expensive h/w I know of was the cheapest up front, but the manufacturer was unable/unwilling to fix it when it broke. They also had the most expensive service contract renewals (which turned out to be useless).
In the race to get to the 3 to 5 year lifetime of a system without major downtime, you don't want your entry to be marked Did Not Finish.
I think it entirely depends on your situation...it depends on if you want to have someone to blame or keep your maintenance in-house.
If you are more comfortable having a vendor to blame, go high-end with contract for support. If you want to save money and build a group of IT support people that are going to be worth a lot more money over time, do in-house hardware that's less expensive but more hardware for redundancy.
You probably should sit down and analyze disaster plans and single points of failure and compare it to how much money you can stand to lose. I don't know what load you're expecting on the servers but I can say that if your business absolutely need the IT services to work, you need to plan on every what-if you can...drive goes bad, offsite backups, fire in the server room (or power outage...how many ways do you get power in there?), controller failure, motherboard failure, etc. and how long can you go without services before it cripples the business, then how much is it worth to not go down (or channel to your support needs) vs. how much would you lose in the time it take to get another system in place?
If you have a good staff you could look at investing in identical servers and VMWare with an iSCSI or SAN solution with RAID plus a good backup server for the backend storage.
Analyze your needs, go over every failure scenario you can, and see what the cost of redundancy is and have a plan in place to deal with those emergencies then see if you want to trust a vendor with guaranteed service for these issues or if you want your in-house people to be prepared for it.
To be honest i is important to give this a reality check from a business perspective first. This is a technical question that is dependent on a business requirement.
So ask yourself a question. Going in to this, which is more important to you, system availability or is keeping the cost down? If having this system operate with no downtime is a requirement, then it is certainly in your interest to look at the premium hardware. If however you are concerned about being frugal with your budget, then by all means the second option to use lower end systems makes sense too. I have used workstations as servers to keep costs down and just make a habit of building them so if a server running on a workstation blows up, it can be quickly transitioned to another redundant server. While this won't win the speed demon award. It will bridge the gap between day one and a time when you have the proper cash flow to invest in the proper industry standard solutions.
Another thing to consider is what is it going to cost you to service hardware that is more likely to fail in manpower and materials? Sometimes it just makes sense to take the plunge if you know that supporting lower end hardware will put you behind the 8 ball.
Just my $0.02...Hope this gives you something to chew on.
