How to test your backups against medium degeneration?

Question

Because most storage media degenerate over time (CD, DVD, harddisk, tape), it is very important to test your backups. Is there a way to monitor the condition of your backups?

I am looking for both hardware and software solutions, as well as strategies you can use. For example:

• Hardware solution: using the S.M.A.R.T. features of harddisks; ...
• Software solution: using a backup tool that calculates MD5 checksums; using a tool for checking the Reed-Solomon ECC on CDs; ...
• Sample strategies: keeping multiple copies of your backup and doing a file comparison every month; scanning every now and then the MD5 checksums calculated by your backup tool; ...

Answer 1

The only way you can be sure of your backups is to restore them. Checksums may verify the content of your medium but they won't tell you if a restore is possible: what if you're not backing up everything you need to?

The complexity of this depends on whether you're backing up for bare-metal recovery (which has its own issues) or just carrying out data backups.

For data backups, one option is to build a virtual machine and periodically fire it up and test a restore. This is less than valid for bare-metal because the VM will undoubtedly require a completely different set of e.g. drivers, etc. I guess (depending on your OS and tool-set), if you were really keen, you could probably script the process of carrying out the restore and do it for every backup.

Answer 2

You should be verifying your backups before storage to begin with. A lot of attention is brought to how hard it is to do backups, but no one ever thinks to check them before safely bringing them off site.

I generally only wait for 1 bad write to a CD-R to toss it, for how cheap they are, and focus on S.M.A.R.T. results for HDD media to tell me when to replace them.

Best of luck to you.

Answer 3

With today's large amounts of data and relatively small optical disc sizes, backup to removable HDD's seems best to me. Also, HDD's are easier to maintain both physically and in terms of filling data onto them.

I regularly run GRC's SpinRite on my disks. SpinRite tests the written data, and optionally refreshes or even recovers it.

Answer 4

Well, you could use par or QuickPar for Windows or run an MD5/SHA1 checksum on the iso.

Answer 5

testing your backups is an interesting point. however, it is time consuming and questionable in my opinion. because if a medium is damaged, your backup may be a total and partial loss, thus testing is rather moot.

i prefer multiple backups. as for optical media (not my preferred choice), if you create a new copy every, say, 5 years you should be on the safe side, if you have two copies (kept in different locations to reduce the risk of data loss in case the house burns down :), the chance that at least one has survived the aging process is very high. of course the risk is decreasing reciprocally to the number of backup sets.

Answer 6

Sounds like you need a checksum for your backup.

Answer 7

All of the answers have been good, but I would like to add one more suggestion. Replace your media on a regular schedule. We replace the back up media every six months. It is fairly inexpensive, especially compared to needing a backup and find that it is bad.

Answer 8

The method I've used in the past is to take a backup server and periodically try to image it from my backup medium.

If it works - great. If not, I go get new media.

clarification for justin re:comments
My imaging process always includes starting the server from the restoration and verifying applications start as expected - otherwise it hasn't been tested, it's merely been written-to.

Answer 9

With WinRAR you can specify the use of a recovery volume which will allow an entire CD/DVD from a recovery set to be destroyed and still be able to recover from the remaining disks. For example, I use WinRAR to backup a set of files/directories by compressing them into a RAR archive, using the span options to set it to span to the size of a DVD(or whatever media you are spanning, the size of a tape backup, etc.) and also set a Recover Volume of 1 or more (found under Advanced tab in the Volumes section, and requires that you have set the Split To Volumes on the General tab, this is different from a Recovery Record so don't confuse them).

With a Recovery Volume an extra Rar file will be created in the set. Some sort of parity is used such that any one of the rar files in a set can be lost and the remaining files can still be used to recover all of the data. You could use Recovery Record as well to allow for a certain amount of damage in all of the rar files.

You can then test the integrity of the data periodically by performing a test extraction. I don't know if you'll get any feedback about the damage of the files. You could easily test this by purposely damaging a file in a set. With a Recovery Volume the data should still be extracted, but hopefully WinRAR will display some feedback in it's console about the damaged archive.

Answer 10

CFV seems to do the job (http://cfv.sourceforge.net/cfv.1.html).

Personally, I use the following 2 commands:

Creating:

cfv -rr -C -tsha1 -fc:\users\zian\desktop\out.txt

Verifying:

cfv -rr -tsha1 -fc:\users\zian\desktop\out.txt

Answer 11

I found a related question here on SuperUser: Which file system automatically stores hashes of files?.

Answer 12

You could use ExactFile to generate a file containing the MD5 checksums of a fresh backup. Before the next backup, you can then compare the MD5 file against the current situation. If there are differences, this means that the backup has gone corrupt.

Answer 13

Making sure that anything I want to keep exists not only in multiple places, but on multiple mediums. Medium degeneration shouldn't just include the physical media themselves, but also the reading of them. I have some old text files backed up on 3 (THREE!) floppy disks, but if I needed them, well, I don't have a floppy drive anymore. How long until the CD drive is deprecated - we're heading that way with netbooks, and 'cloud computing'! If I'd stored my files on a variety of media, I'd be less likely to be in this situation - far more likely I'd still be able to read them!