4

I've been thinking about data integrity, I currently backup about 2tb of data and always have one backup of data from a year ago.

My concern is if a file became corrupt on our production file server no one would notice because some files aren't accessed for many years and if corruption occurred I'd be backing up a corrupted file.

I'm not sure how I should handle this problem, is there a way to detect data corruption? Or is the only solution to store older backups in case something becomes corrupted and isn't noticed?

Sean Bannister
  • 741
  • 8
  • 19

2 Answers2

8

In my experience, each file type needs its own checks to determine if something is indeed corrupt. Data is just dots and dashes at its heart, and what determines "corruption" is entirely file dependent. You will need to determine what file types are most important, and then determine if it is reasonably possible to create automation that checks for the file type's consistency. That will be a daunting task as file type specifications change over time and as you encounter proprietary formats that have no easy way to programmatically determing corruption.

Furthermore, data corruption is only a part of the problem. Sometimes files can be wrong from a human perspective, but consistent from a data structure perspective. If someone mistakenly edits a file - the data is fine from a corruption standpoint.

Ultimately you need to sit down with the leadership of the business and determine what the most important data assets are for the company. Then determine how long those need to be retained and with what level of recall. Do they want fine-graned point-in-time recovery to four years in the past? Maybe only for certain files but not for others?

Considering that you only have 2TB to backup, a GFS tape backup scheme using LTO4 cartridges can allow you to reach back many years with relatively few tapes. This is, of course, entirely depdendent on data churn. If you have a lot of busy bits, then you'll have more tapes. Still, 2TB is a relative speck that LTO4 or even commodity disk storage would yawn at to keep a few years of data around.

It's not an easy task to protect digital assets. Keep the Tums handy.

Wesley
  • 32,320
  • 9
  • 80
  • 116
3

This is one of the reasons why ZFS is so great, no more data corruption. Sharing out ZFS filesystems over CIFS is pretty straightforward.

churnd
  • 3,977
  • 5
  • 33
  • 41
  • This seems like the solution to my problem and that wiki was a great read. Shame we're currently on Windows and won't be able to move for a while. Also the term "silent corruption" has given methe term I need to Google for to learn more. The only bit that confused me was "ZFS cannot fully protect the user's data when using a hardware RAID controller". So I presume this means it only works with software RAID as I just read a little about its support for Raid-Z? – Sean Bannister Feb 20 '12 at 14:24
  • In my search I just read about ReFS coming to Windows 8 Server, don't know much about it yet but for anyone searching it might be interesting to look into. – Sean Bannister Feb 20 '12 at 14:33
  • ZFS is "software raid", yes, but is superior to hardware raid controllers in features & functionality. The server itself becomes the hardware RAID controller. The only downside is you need a big, powerful server for ZFS to take advantage of some of it's features. For data protection (no corruption) alone, you can get by with relatively little. I've seen white papers on ReFS & it's a good thing. ZFS will still be superior. – churnd Feb 20 '12 at 15:59
  • 2
    You cannot assume that data is never corrupted above the filesystem layer. The most obvious example is someone simply writing incorrect data into the file. – wfaulk Feb 20 '12 at 18:51
  • 1
    What if the application smashes a stack and puts out a bad data stream? It will have consistent writes to the filesystem, but the data structure of the file itself will be bad. I thought the OP was referring to checking the consistency of individual file types, and not just if there had been a consistent write to the filesystem. – Wesley Feb 20 '12 at 18:55
  • @WesleyDavid and wfaulk: In my particular case, if an application or user writes incorrect data it would mean that data was being regularly accessed at that point in time and therefore we could recover from backups when someone notices it. My main concern was our archival data that isn't accessed for many years but still needs to be accessible and accurate, my concern was what I now know is called silent corruption. – Sean Bannister Feb 20 '12 at 19:13
  • @SeanBannister there are three types of silent corruption though: Physical layer (your media goes rotten & passes a few bad bits that escape all the error checking), Filesystem layer (what ZFS protects you against) and Application layer (where the app doesn't strictly open the files as "read-only" so if it blows up it can trash the file). You really need to safeguard against all 3 - that's why we have RAID, Backups and Software Testing (respectively) :-) – voretaq7 Feb 20 '12 at 19:18
  • @SeanBannister In that case, ZFS will not protect you against all forms of silent corruption. It would only help to protect against the degradation of physical drives if your archive was on disk. I think the better option would be tape backup. The issue of file-type corruption will only be solved by specific, individual checks of each file of a certain type using special methods, likely provided by the software vendor, to see if the data structure of the file itself is correct according to the application's specifications. – Wesley Feb 20 '12 at 19:20
  • To add onto voretaq7's comment, I think ZFS does more than any other file system to protect against (detect) the physical layer going bad. In response to WesleyDavid, the problem with tapes is they are not a live filesystem & are not immune to data corruption either. To @Sean, check out Nexenta's NexentaStor. It's a ZFS "filer" with commercial support & I think they still offer a "WORM" plugin, which stands for Write Once, Read Many, which is basically for long term archiving of data. Coupled with ZFS to eliminate bit rot, throw in some tape backups & you've got a pretty good solution. – churnd Feb 21 '12 at 11:54
  • @churnd Thanks I'll check that out, I was also thinking about using FreeNAS in the future. – Sean Bannister Feb 21 '12 at 14:19
  • So far I've decided down the track we'll probably be moving to ZFS or keeping an eye on ReFS. And keeping a yearly (possibly 6 month) backup going back 5 years. – Sean Bannister Feb 21 '12 at 14:24
  • @SeanBannister, FreeNAS is based on FreeBSD, which has an "ok" implementation of ZFS. It's not as up to date (and arguably not as good) as a genuine Solaris distribution's version. NexentaStor has a free community edition that lets you use up to 18TB & you can convert to enterprise at any time. – churnd Feb 21 '12 at 14:48