Critique My Backup and Storage Plan

My current storage (RAID-1 off of a hardware RAID card) and backup (a spare drive) solutions for my home network are inadequate. I have too much data scattered on various one-off drives. It is time to evolve. Backups seem simple enough, at least: lots of big drives. However, I am bewildered by the number of choices for small home storage. The Drobo S looks appealing. So does the ReadyNAS. I am not looking for bunches of shiny features, I'm mostly interested in reliability. I am not interested in building Yet Another PC to create a file server or doing something in the cloud, or whatever. I'm stupid, so I am keeping it simple.

Requirements for Main Volume:

Starting working space roughly 2TB, with options for growth up to 5TB
RAID or something RAID-like with at least one parity drive
eSATA II for speed during backups
Ability to shut down gracefully when alerted of low power by a UPS
Optional but Desirable: Will take 2TB drives now with options for the larger 3TB drives coming in 2010-2011
Optional but Desirable: : RAID-6 or something similar, with two parity drives
Optional but Desirable: : Hot spare
Ethernet connection not required, as the volume will be shared via the same machines which runs my home print server

Backups:

Backup performed via ROBOCOPY in mirror mode to an external hard drive via a eSATA II connection.
Start with rotating between two external 2TB hard drives, will go up to six external 2TB drives.
Start with a weekly backup, move to a bi-weekly backup as more drives are added.
Move to 3TB drives as the size of my main volume increases.
Backup drives will be stored on an off-site location.

Hard drives:

I plan on buying all of the same model, but different batches from different vendors.
I found a "burn-in" utility with which I can pound away on the drives for a couple of weeks before adding them to the backup pool or the main volume.

I estimate that I am looking at roughly $1,500 to start, once I start throwing in two TB drives for backup and four for storage. So, are there any obvious flaws in my plan? What have I overlooked? Any suggestions for the storage device for my main volume that fits my requirements? Or do I just keep it simple, 2 drives in RAID-1, then perform due diligence with my backups, accepting that I will have to buy a whole new unit when my data grows past 2TB?

MetaHyperBolic

Posted 2010-03-18T17:50:57.117

Reputation:

What kind of hardware RAID card? Does it have a battery on it? – Broam – 2010-03-18T18:11:03.073

That's a good question. I set this up a long time ago. It's been very stable and functional, not a lick of trouble since I put it together. – None – 2010-03-18T18:30:20.843

@MetaHyperBolic: Just curious, what did you decide to do in the end? – rob – 2010-04-17T00:28:40.733

Answers

Main Volume

I know you said you don't want to build another PC fileserver, but most of the ready-to-go solutions don't have any safeguards against silent data corruption.

If you're looking for data integrity and reliability, you might want to consider running an OpenSolaris fileserver with a raidz2 or raidz3 configuration (2 or 3 parity drives, respectively) on ZFS.

With larger drives, the rebuild time will increase when a drive fails--which also increases the chances of a second-drive failure during the rebuild. But the main advantage of ZFS is that it protects you against silent data corruption, since the filesystem itself is checksummed.

You can also run ZFS on other operating systems, but OpenSolaris is always the most up-do-date version since it takes a while to port new features to the other platforms. If setting up an OpenSolaris box seems a little more work than what you want, FreeNAS seems to be the next best thing, in terms of ZFS support.

On the Linux side, ZFS is not supported in the kernel (only as a user-level driver), but there is also a new filesystem under development, called btrfs. Unfortunately, there is no stable release of btrfs, as of March 2010.

Backup

For your offsite backups, it might be more cost-effective to pay for a service like CrashPlan, Carbonite, or Mozy. It's very, very easy to configure any of these to automatically backup your files. Of the three, CrashPlan has the best backup and recovery features (and even allows you to backup to other remote computers for free), while Mozy's recovery methods are either expensive or very inconvenient (if you want to download a Mozy backup, you have to wait for your job to be queued up and bundled into a zip file). I haven't personally had any experience with Carbonite.

Note that you shouldn't depend solely on an offsite backup--if you backup to the cloud or some other offsite computer, you should also have a local backup.

The Drobo reviews I've seen noted poor write performance, but if you're just using it as a nightly backup drive, it might be sufficient.

Backup rotation

If you want to rotate backups between a local and off-site location, you need at least 3 backups to guarantee one is always local and one is always safe at the off-site location. The third is either in-transit or at one of the other two locations at any given point in time.

ROBOCOPY vs. CrashPlan

ROBOCOPY will cause more wear and tear on your hardware, since it has to read every file during every backup. It's not clear to me whether it only copies changed files or if it copies all files. If ROBOCOPY fails for some reason, it may not be apparent that it has failed, unless you have set something up to reliably report its backup status.

CrashPlan monitors your hard drive for changed files, and only copies the changed files. Since it actively monitors changes to disk, it does not need to read every file in your backup source. CrashPlan automatically e-mails you to notify you how long it has been since the last backup, and how much data was transferred during the last backup.

That said, keep in mind that CrashPlan doesn't have to replace your ROBOCOPY backup scheme. You can use CrashPlan to supplement whichever other backup scheme you happen to choose.

rob

Posted 2010-03-18T17:50:57.117

Reputation: 13 188

I do not want any kind of cloud, online, whatever backups. Absolutely not. I do not want to create Yet Another PC. Especially one where I will have to learn a new operating system for. No. I did not mention those things I wished to avoid out of whimsy. – None – 2010-03-18T18:33:15.550

I noted your objections to building another PC, but I thought it was important to point out a huge weak point in your plan: even with backups, you're not protected against data corruption. ZFS addresses this problem, and is not very difficult to set up. As far as cloud storage, whatever plan you come up with for offsite backup, it's going to be less convenient and will require more maintenance than using something like CrashPlan. If you just don't trust the cloud, CrashPlan doesn't require you to backup to the cloud--you can store an encrypted backup on your cousin's computer if you want. – rob – 2010-03-18T19:34:30.063

Offsite backup is pretty easy. I have a storage locker I visit at least once a week.

Data corruption is tough. A directory is gone -- did I mean to delete it or did something else do it? How would a program know? I have struggled with that issue for a while. – None – 2010-03-18T19:35:26.443

Just as scary as having a directory disappearing is having silent data corruption. The files are all there, but you won't know if they've been corrupted until you try to use them. You could be backing up the same corrupted file for weeks, months, or years before you discover it has been rendered useless. – rob – 2010-03-18T20:01:05.303

1@MetaHyperBolic: I feel I should point out that you specifically "asked" for a critique. The comments/concerns above are valid, and I would suggest you put your guard down a bit. You comments sound very defensive when this is exactly what you wanted – a second opinion. – Paulo – 2010-03-18T20:57:52.260

I understand that. However, I'm looking for a critique of the particulars (more backup drives? less? something besides ROBOCOPY? problems with the Drobo?), not to go off in a wholly new direction. There's a difference between a critique and assuming that, if I don't want something, I'm just wrong about that. – None – 2010-03-18T21:32:56.137

After giving it some more thought, I would suggest doing ROBOCOPY backups very infrequently, and using CrashPlan as your main backup program. I added a very brief ROBOCOPY vs. CrashPlan discussion explaining why. – rob – 2010-03-18T22:35:53.410

I think it's also worth mentioning that any manual steps are susceptible to human error and/or neglect. For example, you might mix up the drives (if you haven't properly labeled them) or forget to transport the backup drive (if you're in a hurry to leave). The backup drive could also be damaged or misplaced in transit--for example, you could slip on ice or trip on your way to the car, and drop the drive. – rob – 2010-03-18T22:58:39.363

"CrashPlan monitors your hard drive for changed files, and only copies the changed files" this unfortunately has a bug if you have any applications which use the old 8.3 DOS style naming convention (file~1.txt). We have found that if a file in a sub-directory of a directory you have excluded from the backup is accessed it can't differentiate between the 'long' and 8.3 name. We ended up having to turn off real time file monitoring as it kept uploading a huge temporary directory and stopping the backup completing. – Richard Lucas – 2011-09-08T07:31:29.743

Your question is specifically called "Critique My Backup and Storage Plan". So at the risk of getting down voted because I'm not going to tell you what you want to hear, here's my opinion of your plan. It's overly expensive and overly complex.

I am not interested in building Yet Another PC to create a file server or doing something in the cloud, or whatever. I'm stupid, so I am keeping it simple.

Seriously? Having to setup backup scripts with Robocopy (which can't actually backup in-use files)? Manually transporting drives off-site? Hardware raid? Buying drives from multiple vendors and running "burn-in" utilities on them? All of that is the opposite of simple.

For all the hype there is around "cloud" services, home data backup really is one of the killer uses for it. I personally use CrashPlan because the software is well behaved and dead simple to setup. But they're all roughly the same in terms of price. Expect to pay around $3-$5 bucks a month total for unlimited storage from all of your machines. The "win" with cloud based backups is that they take all the hassle out of getting your data off-site. But no matter what service you use, restorals are a pain since downloading gigs of data will take a while. So you should augment it with a local backup solution as well. And since you've already got the cloud based backup working, your local backup solution doesn't need to be bullet proof in terms of reliability to be useful.

Here's the setup I have for my house. I built a small low-powered Atom based "home server" with four 1TB disks in a software raid 5 array with no hot spares. To be clear, the raid here is not for reliability. It's for the simplicity of creating a single backup drive. The server is running Windows and a copy of CrashPlan which is not configured to backup anywhere. The rest of the machines in the house are also running CrashPlan and configured to backup to both the CrashPlan Central cloud and my home server. That's it. Everything happens automatically and I get backup status reports via email from CrashPlan to make sure everything is running smoothly.

If any client machine in the house dies, it can be restored from the home server very quickly since it's all local. If the home server dies, there's nothing to restore since it was just a local repository for everyone else's backups. I just re-build it and re-point all the clients to the new/fixed box. If my whole house is vaporized from orbit, I've still got the cloud backup to restore from as a last resort.

Ryan Bolger

Posted 2010-03-18T17:50:57.117

Reputation: 3 351

Getting my data offsite is not an issue. I leave the house daily and hit my storage unit often. Pushing my stuff through the thin pipes available here, though, would be an issue, and I can't think of any vendors I am willing to trust with my personal data. It looks slick, but given how oversold bandwidth is here, it's not appealing.

As to software RAID, I haven't yet run across one which was simpler than hardware RAID. Maybe things have changed, but they still require Yet Another PC. – None – 2010-03-18T21:41:20.783

@MetaHyperBolic: It would help to know your upload speed, but keep in mind that most good network backup applications will only copy the changes (or, at worst, the changed files) over the wire. The initial backup will take a long time unless you seed the offsite backup (which is possible with CrashPlan), but subsequent backups will be much faster unless you're frequently changing your files. Also, network backups typically run more frequently (e.g., whenever your computer is idle), making each backup job even smaller. – rob – 2010-03-18T21:57:49.907

Only the one backup? That's a mistake :) Especially putting so many resources into it, it'd be a pain if, for example, a natural disaster knocked that backup, and your own, out.

Phoshi

Posted 2010-03-18T17:50:57.117

Reputation: 22 001

Definitely not "only the one backup." Note: "Start with rotating between two external 2TB hard drives, will go up to six external 2TB drives."

Six backups seems ... sufficient. – None – 2010-03-18T19:33:15.290

@Meta; So long as they aren't all in the same location, I agree with you! – Phoshi – 2010-03-18T19:49:43.120