Is there a way I can create ECC data for hard disk files?

I've got about 500 GB of data that I'd like to archive onto a hard disk. However, I've found disks to occasionally flip a bit or two over that 500 GB of data over a span of about 5 years or so. Large disk systems mitigate this problem by using multiple drives, and "scrubbing" the data regularly.

However, I'd like to yank this drive out of my machine, place it at someone else's (a relative, or something like that) house, and leave it there for long term offsite archival of that data. Because the drive isn't plugged in, I can't use anything like the data scrubbing typically used to mitigate this problem.

I've had good results for optical media with the DVDisaster tool. Does such a tool exist for general hard disk files?

Billy ONeal

Posted 2010-06-27T21:18:46.847

Reputation: 7 021

Answers

Yes, these are called parity files. There are various utilities to create these kinds of files, such as Parchive and PAR2.

These utilities will generally let you choose how much parity data you wish to generate. The more parity data, the greater your ability to recover files. Disk space is the trade-off.

Mike Fitzpatrick

Posted 2010-06-27T21:18:46.847

Reputation: 15 062

Most of these "error correction" algorithms are implemented using Reed-Solomon Error Codes. Some other software options are MultiPar and ICE ECC. QuickPar is also popular, but hasn't been updated in years.

– afrazier – 2010-06-28T01:03:20.277

This sounds kind of like ZFS's checksumming and RAID-Z. I don't think anything exactly like this exists at the drive level, but there are tools (mentioned above) at the file level.

Slartibartfast

Posted 2010-06-27T21:18:46.847

Reputation: 6 899

With a single disk, you can still enable ZFS ditto blocks (set copies=2) to overcome silent disk corruption. Of course, this is more costly than ECC as the storage space required will double. – jlliagre – 2010-06-28T12:50:15.213

-1

I seriously doubt a bit will get flipped while the disk is disconnected and stored.

If you are looking to scrub the drive once you decide to use it years later, I suggest one of these 2 tools.

Spinrite 6 http://www.grc.com/intro.htm not the cheapest, but the best
HddRegenerator http://www.dposoft.net/#b_hddhid

Moab

Posted 2010-06-27T21:18:46.847

Reputation: 54 203

1I seriously have had file corruption on disks which were disconnected and stored. Hard drive archival life when the disk is not in use is typically under 10 years. I'd like that to be better and it can be if sufficient error correction is available. – Billy ONeal – 2010-06-28T02:03:09.550

After reading the posts here, I don't think there is a perfect solution, if you store parity data on another hard drive, what is to prevent that parity data from having a flipped bit.

Maybe this if you can afford it http://www.cranberry.com/

– Moab – 2010-06-28T13:08:49.383

1Good parity software (such as that in the accepted answer) is able to deal with flipped bits. Look up the dvdisaster tool I referenced for an example using optical media. – Billy ONeal – 2010-06-29T03:30:08.997

I understand what parity data is, what is to prevent parity data from being corrupt? Like I said there is no "perfect" solution, but Mike's looks to be the best so far. – Moab – 2010-06-29T12:44:47.353