Verification of files on backup media

I have a system in place where I back up my work files daily to a portable hard drive. I actually have two portable hard drives - one is stored off-site and I swap them regularly.

I also keep my family photos and other historical files backed up, but I only back the photos up occasionally (ie when I have new photos).

The backup media is for backup only, and it is unlikely I will ever read the files from the backup media unless a disaster occurs and I lose the master.

It worries me that my backed up files could become corrupt without me knowing it. It is also feasible that my master files could become corrupt, and eventually the corrupt files would be replicated to the backup media.

I'm currently using Cobian Backup, but I'm open to alternatives.

Is there a tool I can use to confirm that the backed up files are identical to the files that were first copied? I know it would be possible to generate a checksum and periodically validate the backup files against the original checksum, but I'm looking for a tool which will do this automatically.

Update: Of course I can (and will) manually test the backups by occasionally doing a test restore, but there is a lot of data involved (10,000 photos, 10,000+ emails over 5 pst files, 1,000+ genealogy records and a bunch of other stuff). A test restore will be quite time consuming so it's not realistic that it will be done very often (definitely not as part of the daily procedure), and it's still possible that I could miss problems reviewing the data manually. I'm looking for a supplement so I can test the data more regularly and pick up problems earlier (even if it's not 100% guaranteed), as well as validate the manual review.

Greg Sansom

Posted 2011-03-13T23:41:12.620

Reputation: 827

There is no substitute for test restores. If you can't do a restore, then you haven't done a backup. Checksums and file verification are nice but they're no substitute. – Paused until further notice. – 2011-03-14T00:30:05.887

Greg: By your question you must be interested in genealogy. Can you help us start a Genealogy Q&A site at: http://bit.ly/U3vnDX

– lkessler – 2012-09-23T01:47:39.063

Answers

How automatic does it need to be? This tool, for example, can generate (and check, of course) MD5 checksums in a pretty slick/quick manner.

Craig H

Posted 2011-03-13T23:41:12.620

Reputation: 1 172

That looks like a useful tool to perform checksums, with decent (and logical) integration into the GUI. Thanks for sharing that link! – Randolf Richardson – 2011-03-14T01:35:16.343

+1 for the link. I'll try and incorporate this into my backup procedure and if it works well I'll accept. – Greg Sansom – 2011-03-14T01:40:05.997

Disclaimer: I am the author of this library.

If you backup on hard drives, I made the pyFileFixity, a self-contained Python 2 library, containing the tool "rfigc.py".

If you plan to backup on optical discs, there is the DVDisaster project which is awesome (it has a very good protection scheme, which includes meta-data like directory trees).

gaborous

Posted 2011-03-13T23:41:12.620

Reputation: 1 412

At the very least, select a few file at random (e.g., a few JPeG images, a PDF or two, some text files), then restore and try to load them. If they work, then chances are that your backup is probably good.

Now, to take it one step further, I advise you to insert a different backup media instead of using the most recent one you just backed up to. The reason for this is that many years ago I had a tape backup drive fail on me, but the failure was really strange -- I could restore from the current tape, but after ejecting the tape and inserting any other tape, the restore would fail (and reverting to the most recent tape resulted in a failure as well).

Randolf Richardson

Posted 2011-03-13T23:41:12.620

Reputation: 14 002

@Randolf: Thanks for the suggestion but I think it's tangential to the question being asked. – Greg Sansom – 2011-03-14T01:41:02.440

Verification of files on backup media - after the backup

Answers