What lasts longer: Data stored on non-volatile flash RAM, optical media, or magnetic disk?

6

3

What lasts longer: Data stored on non-volatile flash RAM (USB stick or SD cards?), optical media (CD, DVD, or Blu-Ray?), or magnetic disk (floppies, hard drives?) My gut tells me optical media, but I'm not sure.

Furthermore, which of those digital media would be most suitable for long-term data storage where environmental issues are unknown, such as low/high temperature or humidity?

For example, what digital media could be stored in a basement, attic, or time capsule, and be expected to survive a reasonably long time? e.g. a lifetime, and then some.

Update: Looks like optical media and magnetic tape each have one vote below. Does anybody else have an opinion or know of a study comparing the two?

Chris W. Rea

Posted 2009-07-16T01:14:48.803

Reputation: 10 282

2Acid-free paper. – Daniel R Hicks – 2012-06-26T03:25:25.790

Answers

11

See Recommended Backup Media for Circa 2009? on Server Fault.

As for optical media:

DAX Archiving provides some whitepapers, which I have not read. But there's one written by Verbatim, claiming More than 100 years projected lifetime for DVD-R General.

Recently Adrian Wong of Tech ARP by accident found out that "CD-Rs that were just 7-9 years old were failing at a significant rate". So: cross your fingers.

Arjan

Posted 2009-07-16T01:14:48.803

Reputation: 29 084

3Yes, pressed CDs are completely different from CD-Rs. Pressed CDs should last ages, it's basically just plastic, Whereas CD-Rs use chemicals which change color when written to, and these chemicals could deteriorate. – davr – 2010-03-03T00:34:13.613

Thanks for the link to the collection of whitepapers. +1 – Chris W. Rea – 2009-07-26T17:21:28.633

Also ... interesting about those CD-R's failing so soon. I've got some audio CDs purchased more than 20 years ago that still sound great. I suppose pressed CDs and CD-Rs are considerably different in terms of durability? – Chris W. Rea – 2009-07-26T17:22:59.067

Yes, that's for sure. Burning at a lower speed apparently even increases the lifetime, for what I've read. – Arjan – 2009-07-26T17:49:12.700

1Agreed, Low-Quality CDs/DVDs are simply Rubbish. The silver stuff starts to flake off in a matter of years (not 10s of years). They scratch incredibly easy, and are prone to getting dirty.

My vote for long term goes to Magnetic media. – dubRun – 2009-09-04T01:27:27.077

5

Most CDRs sold are of the low-quality variety. Race to the bottom and all that. The 100+ year predictions have turned out to be garbage in most use cases.

The only really long-term solution is custodianship of your data. Every 5-10 years you need to copy your data to a new device, verify the integrity, and make sure you still have the tools to read whatever format it is in. If the tools look like they might not be around or convenient in another 5 years, convert your data into a format which looks like it has good future potential, and keep both copies. Rinse and repeat.

This strategy requires an easy-to-read media, so tapes are probably out. DVDs are convenient these days, but aren't particularly high density. Hard drives are high density and convenient to read, but have moving parts that can go bad over time. Your most reliable bet is going to be spreading verified copies of your data over multiple mediums, rotating these as specified above.

Don't forget to verify the integrity of your data on both read and write. Bitrot does happen, and it's far more common at modern data densities than you imagine.

Paul McMillan

Posted 2009-07-16T01:14:48.803

Reputation: 826

Also it would be good to be using a checksum that incorporates ECC so after you know your data has gone bad you have a chance of fixing it after the originals are long gone. – Scott Chamberlain – 2012-06-26T02:35:09.613

1Good answer - I like the "custodianship" concept. So, does "verify the integrity" necessarily mean that files must be preserved with MD5 or SHA1 checksums or something similar, for later verification? – Chris W. Rea – 2009-10-11T00:40:08.173

Yeah, you need checksums, or preferably a checksumming filesystem like ZFS if you can manage it. Checksumming a tarball would be easier than keeping one for each file. – Paul McMillan – 2009-10-20T10:24:57.210

1Ultimately though, your checksumming solution may need to migrate about as often as you modify formats and transition to new storage media. – Paul McMillan – 2009-10-20T10:27:24.103

2

Optical media is good because it's cheap, but I would use magnetic tape.

CDs and DVDs suppose to be readable for 25-50 years but it's very far from the truth. Even if there's no structural damage some badly written DVDs can loose their content in 2 years.

Flash memories do not discharge for many years. However, it's a relatively new technology, so they might not be tested well enough in this respect.

In contrast, magnetic tapes can be read back after 20-30 years and are designed for backup. Very high temperature (e.g. fire) will erase them, but given normal conditions they will remain intact.

KovBal

Posted 2009-07-16T01:14:48.803

Reputation: 1 250

When I was young, I had a summer job as "tape drive operator" at a large company. A console would indicate which jobs needed which tapes on which drives. These were the reel-to-reel 6250 BPI tapes, and the company had just started installing new cartridge-style drives. Anyway, I recall we frequently rotated tapes. For that reason, I'm skeptical about tape's long term viability. Mind you, might that have been the mechanical stress of frequent use? Perhaps not an issue with archival tape? – Chris W. Rea – 2009-07-26T13:42:45.803

@cwrea : I thought you talked about archiving for many years. (E.g. you record something and you want it to readable for decades.) – KovBal – 2009-07-27T10:25:48.560

Yes, I am interested in archiving for many years. In my comment, I just want to be clear that the issues I've experienced with magnetic tape were due to their frequent use, not because tape is inherently prone to fail. – Chris W. Rea – 2009-07-30T01:51:42.990

@Basically: yeah i did that gig too. mostly those weren't archived tapes, that was production data. so they get rotated to reduce mechanical stress. – quack quixote – 2009-10-10T17:59:13.643

1

This is obviously late, but might be of interest as new discoveries/technologies are made.

All mentioned medium have their specific physical disadvantages. Filesystem, file formats, etc. are common problems to all of these. A couple new ideas:

  1. Cloud (internet, data center, etc.) storage. You can encrypt the whole lot, or in parts, and write the passkey/password/etc. onto something reliable, like stone tablets.

  2. Similar to cloud storage is something called "live/rolling storage". Basically, you store the data on a live system which you replace with newer, more modern hardware, every so many years and move the data to the new system.

  3. Microfiche (data is encoded on it). Or some new medium based on nickel (can't remember much about this medium). Microfiche preserves well. Data density might be a problem, but everything has a disadvantage. This isn't as outdated as it might appear as of the time of this writing, just somewhat reserved for larger endeavors.

Ioan

Posted 2009-07-16T01:14:48.803

Reputation: 451

1

The bottleneck of data storage for future reading is mentioned ( from device to read the data upto the file format of data itself). Assuming you have always upgraded your data archive to latest mass media; Why would anyone spent time to retrieve your old sock with data?

I'm starting to get more and more convinced of creating an index of the most important files including specification of software used to save the file. This index can be printed. A copy of index can also be placed on several cloud systems. Since companies have an average same life span as CD's (home brewn and official audio cd ;-) I would not trust my archive on cloud computing as final archive.

Mathilde

Posted 2009-07-16T01:14:48.803

Reputation: 11

1

Backup - tapes only. CDS are good for installing OS, flash is 100% unsecure.

Adam RichardSon

Posted 2009-07-16T01:14:48.803

Reputation: 352

1

I'd go with the Optical Media. It should survive the varying temps and humidity the best for long-term storage. As long as it isn't scratched or broken during that time...

JFV

Posted 2009-07-16T01:14:48.803

Reputation: 1 017

1

Optical is going to be your best bet. It is the most resilient against long term conditions (such as the ones you speak of).

Magnetic will eventually lose it's data. RAM can be wiped by accident, as well as suffers from humidity and temperature.

With optical the only thing you have to worry about is structural damage, really.

Joseph

Posted 2009-07-16T01:14:48.803

Reputation: 852

1

This is when I wish that MiniDisc had caught on, it should out-last any current archival media.

Basically how it works is to write any data, you first have to heat up the media with a laser, and give it a magnetic field. Which means you can't really even use a degausser to erase the data.

It also doesn't have to read the data through a polycarbonate substrate, like CDs, or DVDs or BluRay disks. Which means that you can leave it in the sun without worrying about it becoming unreadable, due to discoloring.

It's a shame that Sony keeps trying to force it's will onto the consumer. The only time they managed to come out on top, is with BluRay, which they only accomplished through making deals with movie makers.

Brad Gilbert

Posted 2009-07-16T01:14:48.803

Reputation: 771

1

A flash disk.

I vote 'flash drive' because I know nothing of it, but it appears to have its own USB bus in hardware. Finding hardware in 100 years that will read old data is the problem neglected here.

Magnetic media, when on tape, we were told to 'refresh' (copy over itself, remagnetizing the surface) every three years. Since, I've found floppy disks to be readable decades later. These disks were stored in a well-insulated environment, so the plastic (mylar) stratum didn't expand or contract. Each has its use.

It's the environment that affects optical media. You may notice that CDs & DVDs are actually translucent. Remember, it's light that causes the dye to react. (DVD-RW disks are written by melting xtals and chilling them to a glass: this glass will recrystallize within months.) Further, the stratum is not perfectly elastic, expanding & contracting back with the heat of the day: it is slightly plastic, never returning exactly to the same shape. Avoid temperature variations. I use Taiyo-Yuden lacquer-coated or Verbatim (with LightScribe) for archiving. I may now switch to a flash drive.

This because I'm unclear of the usefulness of a 300 year old DVD, when DVD players have a life from 6 months to 3 years. I can read only mini-floppy disks now because I saved a good drive in a safe place. If you're going to archive with DVDs, save a $40 DVD player. Or, use a flash drive, which comes with its own player.

All the above is irrelevant if you cannot read the data and move it in the future to a new medium and possibly new format. Use a lossless, ISO standard format: newer ones will replace compressed, 16-bit TIFF for photos. Keep track of new, popular software formats and hardware media; and transfer your family photos as these become available. As with magnetic media, you need choose your file system on a flash drive: the source code for Linux ones are published. The flash drive is unique in having a player with no moving parts.

Consider that a programmer can use source code published in paper libraries or on internet archives to re-write old file systems and read file formats. But it would take a very clever engineer to build a device for reading the medium; unless its just electronics to convert USB in the distant future to 'Microsoft Protocol' protocol, & software to convert ext3 to 'Microsoft Format' format.

Although the Wikipedia gives newer EEPROMs a 10-year life, this quote is interesting:

'Channel Five's Gadget Show cooked a flash drive with propane, froze it with dry ice, submerged it in various acidic liquids, ran over it with a jeep and fired it against a wall with a mortar. A company specializing in recovering lost data from computer drives managed to recover all the data on the drive.[22] All data on the other removable storage devices tested, using optical or magnetic technologies, were destroyed.'

http://en.wikipedia.org/wiki/USB_flash_drive#File_system

Because a simple USB dongle can now hold as much as 55 DVDs, and the USB bus will likely outlast DVD players or tape drives, a flash disk is the clear choice in archiving data. (Get another with a write switch for holding a bootable repair & malware extraction kit; get another that stores encrypted incremental backups & take it home each night.)

Bruce Bathurst PhD

PS. To support my research in petrology, I took jobs in all aspects of computing. For each client, I chose the best tools for that particular business. Thus, I don't feel biased in my conclusion. In fact, I've never even used a flash drive! :-)

Bruce Bathurst

Posted 2009-07-16T01:14:48.803

Reputation:

3Flash works by pushing electrons across an insulated barrier. Over time, they will slowly leak out across the insulation. However, I'm unable to find reliable figures on whether this happens over 10 years or 100 years. – pjc50 – 2012-02-16T13:35:18.510

1

There are a number of issues - first is the stability of the media itself. Floppy disks here in Hawaii rot due to high humidity/mold/mildew issues. I've seen so much media destroyed that I'd say if you can't guarantee low humidity, mag media is out.

The second thing is the media format - who can read 5-1/4" floppy disks? 8" floppy disks? Even the Zip 100 disk drives are getting few on the ground much less the older tape formats.

The third thing - what encoding are you using for the data? If not a standard and hopefully open standard format, then you may not be able to even get it understood. Even Microsoft Office has problems reading Microsoft Works on occasion, much less something like more obscure formats.

Blackbeagle

Posted 2009-07-16T01:14:48.803

Reputation: 6 424