Optimal backup server strategy

Question

As I read the blog post by Jeff Our backup strategy describing in detail how they setup their backup environment, many people commented on how that is not a good approach.

My company currently does not have a backup solution setup yet, and I think this is something we really need to do soon.

But there are so many different approaches or solutions out there, that I don't know which one to do.

I have some possible ideas on what to do:

1. Create a small virtual machine on our current webserver, that backs up all the data to a NAS.

2. Setup another machine in-house, that has a lot of storage and backs all the data onto its harddrives.

3. Setup another small machine in-house, that backs up the data to a NAS.

4. Go with online backup services (which one can you recommend?)

Some information on our technical environment:

We have one big server running several VMs (database server, webserver, vpn server for our office, etc.). We need to backup all our Subversion data and of course E-Mails and daily database dumps.

I would appreciate any advice.

Answer 1

I don't think there's any such thing as a single optimal backup strategy. A lot of it is dependent on what you're backing up and how much of it there is. I'll second the point that it should be driven by your restore requirements, specifically how quickly you can get things back and how much you can afford to lose in the event of a failure. Finally you'll need to remember that there at at least 2 layers of restore you need to cover: restoration of individual data items and full server restores.

I always prefer tape to NAS, but for some reason tape has a bad rep in certain quarters. One major advantage however is that it's physically decoupled from any OS (or other logical) environment, so you greatly reduce concerns such as how do you backup the backup server and how do you implement a clean and easy offsite strategy.

My standard advice with backups is to keep them as simple and primitive as possible, so that your restores can also be simple, and so that there's fewer layers of sexiness to potentially go wrong. Backups should be predictable and boring.

Online backup is something that people can find tempting, but you should always remember that some time you may need to do a full server restore. Unless you have bandwidth to your provider that's comparable to your local bandwidth I would keep away.

Answer 2

You need to have backups in-house as well as off-site. The ones in-house are for quick recovery, and the ones off-site are in case your "in-house" blows up.

I'd recommend using RAID-6 as much as possible with a controller with battery backup installed on any backup machine.

I don't know which backup services you know of, but any of them which offer you unlimited space are a seriously bad idea.

Answer 3

You need to start with the business requirements. Backup strategies start by thinking about the recovery plan. Figure out what needs to be backed up on a daily (maybe even hourly) basis in order to get that particular business function up again in whatever timeframe is required. Backups might not be required to fulfill the functionality required. For example if I need a file server to be able to recover any file deleted within 30 mins I'd configure the volume shadow copy service in windows to take snaps every 15 mins.

Once you have the business requirements then you can determine what's technically required to implement that. You'll notice that in the comments the arguments were about cost/performance, reliability etc. All business issues.

Answer 4

Let me start by saying that any backup system is better than none at all but if you're only going to back up to another machine or storage device at the same location it's only slightly better than none. You need a way to have the backups off-site. My preference is and always has been to use tapes. Some will suggest an Internet based remote storage but that certainly isn't a globally viable solution. It also adds the risks associated with having your data stored at an unknown location and accessible by an unknown number of people. Regardless of what method you use, a backup is only as good as your ability to do a restore.

I currently work for a company which had to rebuild from the ground up after a fire totally destroyed their last premises, including everything in it. Due to adequate off-site backups they could do that. How would your company fare in the same circumstances? According to some statistics I've seen, the majority of businesses will fail within the first year after such a disaster and one of the primary reasons is the loss of critical data, which usually results in a loss of business.

Answer 5

First step when thinking about backup is: Put a price tag on your data, and, separately, downtime. Once these steps are taken, the rest more or less falls into place.

Answer 6

mh says wise things. Like him, I am a "traditionalist" and think that properly managed and maintained tape backup offers a lot of advantages (easy to take off-site and offline, low incremental cost to add capacity, "serious" tape technologies (like LTO and DLT) have media that is capable of long-term archival) to disk-based backup alone. Disk-to-disk-to-tape is a great way to shrink backup windows and provide convenient restore, and the cost is typically incremental to tape since a box filled with moderately fast disks (some large SATA disks in a RAID configuration, for example) isn't too costly.

None of your proposed methods mentions taking the backups offline. Backup is both off-site and offline. Keeping your backups online means that an attacker can trash your backups right after they've trashed your production data. It's awfully hard to remotely attack a physical media stored in an off-site, physically secure location.

If you decide to do disk-based backups, I'd strongly recommend regularly taking backups offline and off-site.

Whatever you do, test restoration is in order. We don't do backup to "have backups". We do backup in order to restore. You need to regularly test backups by doing restores. You need to know how to perform restores and you need to perform them on a recurring basis. Your attitude, re: restore, should be "Been there, done that". When a real catastrophe happens the last thing you want to be worring about is learning how to bring your systems back from the dead.

As I said in my comment to mh's answer, online backup is only as good as the restore mechanism (since, again, we do backup to restore). If it'll take 36 straight hours of downloading to bring your data back from the online backup provider and your business can't handle 36 hours of downtime then it's probably a bad deal. That having been said, if you can structure online backups to catch slow-changing "archival" data that's not critical for day-to-day business you can use it as an off-site storage strategy, thus decreasing the amount of business-critical data that needs to be on the physical media that you take off-site.

There are several good questions about this already on Server Fault. I'd go and read them. Here are a few that come to mind to me right off.

• Is our planned backup strategy adequate for my new server insfrastructure?

• Recommended Backup Media for Circa 2009?

• How do you do your backup?

Answer 7

Backup is the single most important things that Admins do. ....
Let me repeat.

Backup is the single most important things that Admins do.

A good backup strategy allows recovery from various kinds of data loss, including ...

1- hardware failure.
2- data/application errors and data corruption.
3- inadvertent or malicious file/folder/data deletion.
4- site or infrastructure problems (i.e. a fire in the server room).
5- regional disaster (i.e. Hurricane Katrina)

A good strategy balances the value of the data and the cost of downtime against the likelihood of the failure, the funds available and the cost of the various kinds of backup systems.

At minimum, you should have a daily backup to tape of all data, and some sort of scheme for keeping those backups off-site. A full backup over the weekend and nightly incremental backups are OK. I like a 4 week rotation of tapes. The backups should be tested (via restores and via test server rebuilds) regularly.

BTW -- IMHO our esteemed host needs to improve the off-site component of the Stack Overflow backup strategy.