How to store and preserve lots of data?

Question

I have many DVDs with movies and games, they are getting old and dusty, some of them are damaged. And I have more than 300 GB of general files in my PC. I want to store and preserve all of my data and keep them for a long long time.

I want to know what is the best and more reliable storage media that I should use and... I can buy.
I will copy all of my DVDs to a new media and throw all of my DVDs away. It will take a very smaller physical space.

I think that everything that I need to store is currently 800 GB.

I do not like optical discs, because they are fragile and I have lost many CDs and DVDs.
Hard drives would be good, but they can suddenly stop working, and it is very expensive to recover data from a broken hard drive.

Other questions:
How often will I have to copy the files to a new storage media?
Is it good idea to have another copy of each file stored in another storage media?

Edit:
If I put all of the movies and games in a HDD, if the HDD stops working, I will lose everything.
This is how valuable they are for me: If I lose 10% of them, I will not cry, because I can get new ones, but if I lose 90%, it will be a disaster.

Edit 2:
I do not think that that online storage is reliable even if I pay for it, because they may be closed or may get bankrupt. They are not from my country and it would be difficult to get the files back.

Answer 1

The simple answer is: multiple copies. Whatever else you do, don't trust any single media, location or service.

Personally, I currently use external (USB-connected) hard disks for backup purposes. A 2-3 TB drive can be sourced quite cheaply and will provide you with plenty of storage capacity not just for the time being but also account for any reasonable future needs. At the physical size of roughly a large paperback book, it will easily hold the content of 200-300 movie DVDs without further compression. Buy two, or three if you are paranoid, preferably one of which from a completely different manufacturer (might want to mix Seagate and Western Digital, for example, since they are unlikely to use disks with the exact same design or manufacturing defects), and keep at least one of them in a physically separate location - a bank safety deposit box is a relatively cheap alternative that will give you physical security as well, but even just keeping one copy at work or at a friend's home will almost always work just as well. If you can arrange to be able to refresh an off-site copy without bringing it to your own location, that is even better. If any of the content is privacy-sensitive, keep that in mind when planning how to handle off-site copies.

Also keep in mind that the amount of data you are talking about (300 GB counting as more or less "irrecoverable", another 500 GB "nice to keep" but which in a pinch you could probably get from other sources such as second-hand movie DVDs) is not really all that much. I currently have a grand total of about 100 GB of digital photos alone, and it's not hard for me to add during a single day some 10-15 GB to that - and I have done that on a few occasions going to events where I have had reason to take lots of photos. Many of those photos are of in various ways questionable quality, many are mundane (nice to have, but in a pinch there's nothing truly special about them), but some of them actually are irreplacable from a content point of view as well as actually of good quality. For backup purposes, though, I treat them all the same way: multiple copies. I've had a few hard drives fail on me and while a few times I've lost data I would really have liked to keep, overall this strategy has meant that I can restore the most recent backup to a new drive and be on my merry way. If the live copy fails restore the backup to a replacement primary drive; if the backup drive fails, get a replacement backup drive and make a new backup.

If you do go the multiple storage media route, too, remember to keep checking each for signs of degredation. It's fairly quick and easy to do a SHA1 hash run over all the files on a drive and compare the results, as well as storing the list of hashes itself in multiple locations. That way, even if you get read errors at some point, you can determine which copy is "good".

Answer 2

You have to consider how valuable the data you're backing up is to you. I would split it into at least 2 categories:

1. Irreplaceable
2. Would like to keep but won't be heartbroken if the data is lost.

Strategy

For category 1: I would suggest one of the popular online storage systems (Amazon S3, Dropbox, etc). Here you're paying for someone to help you manage the backup, and ensure longer term access and easier access. Otherwise follow the suggestions for category 2 and ensure correct redundancy of data exists and care is taken. Assuming it's a smaller percentage of the 800 Gb total.

For category 2: it's your decision on how much to invest storage fees and time to upload data online. For that reason I would even suggest you use a large HDD to backup all the data, and store that drive disconnected from a PC, and just be aware it has a lifetime linked to a PC that supports current tech eg SATA. Then you can port the data to a new drive / new tech in the future. A 1 or 2TB HDD is reasonably well priced and will cover your data requirements now and into the short term.

Redundancy

Multiple drives with the same data would be your redundancy, which again can even be stored 'off site' if you're truly concerned about data safety.

Security

As an added feature if you're trying to protect the data from unauthorized access encrypt it locally before uploading online, and/or storing to HDD. Something like TrueCrypt will be ideal

Answer 3

First, think about the ways you might lose your data, and decide which you want to protect against. Some examples:

• accidentally delete something
• hard disk dies
• a software or hardware bug
• malware
• theft
• fire
• natural disaster (fire, flood, quake, volcano, lightning, etc.)
• government seizure

In my life, I've lost plenty of data, but only to the first two causes on that list.

Hard disks are compact, hold a lot of data, get bigger all the time, readily available, don't require special equipment to use, cheap, and getting cheaper (flooding in Thailand not withstanding).

I keep all my data one one drive; a second USB drive holds regular automated backups; a third identical USB drive sits offsite (at work, or in a safe deposit box is good). Monthly I carry the current backup drive to the offsite location, and bring the other drive back.

All storage media decay; the only way to be sure that your data is good is to use it. As part of that monthly routine, I pick an arbitrary file and restore if from my backup.

Answer 4

I had the same problem as you and looked hard and long for a solution. Many NAS drives have 2 hdd bays.. i wanted more.. but that always had a big price tag on it..

but then I found this! ^{demonstrated model has been superseded sine original post}

• 4 SATA-2 BAYS (not hot swap) max of 8TB (modding can provide more space)
• 1 SATA ontop for CDROM/SYSTEM HDD running on DMA33
• ESATA on the BACK
• 4 USB on the FRONT! 2 ON THE BACK
• USB INSIDE (where the door is- so you boot of USB drive and not have it sticking out)
• 8GB MAX DDR3!
• Hardware Embedded SATA RAID 0/1 ^{But you should use ZFS instead}

Product Spec Sheet

I bought mine for £100 (there was a HP deal on) but normal price is £200 - Still half price of many other NAS.. AND! Its a full fledged processor! 1.6Ghz Dual Core AMD (special type that uses low power)

I could not believe my freaking eye balls!

Then what? This is what

I went over to FreeNAS and installed freenas7 - I know FreeNAS 8 is out.. but it seems to chew down on memory.. but the ZFS driver is allot more stable and faster on it.. So the choice is yours.

I installed 4x1TB and used RAIDZ similar to RAID-5 on ZFS (the HP server does have hardware raid(0,1) too.. but the driver does not work on freenas7 :( -- but does work on Windows Essentials! )

so I got 3TB of storage with 1 redundant hard drive(RAID-5)

Did i forgot to mention it has a full PCI-E slot in it? (low profile) Here is a sneak peak at the MB and the nice usb plug in directly on mobo

MODDING

• You can fit up to 6 3.5" drives in here. Use a slim optical drive/3.5" drive adapter for the DVD bay, and then fit another behind the HP light cone with a little dremel work.

• OK and if that is not enough.. then how about 19TB?

• And i got my self so fired up i started writting my own dashboard and hopefully some tutorial on how to set up a uber freenas server over here.

-EDIT AUGUST 2012

I found this post that has a section how to hack the BIOS and enable that extra CD-ROM port to run at 1.5GBs SATA! - YES! I did it and works well.

-EDIT OCT 2017

We have since had the Gen9 and Gen10 of the microservers - I am still using the original configuration from the time of this post.. yea.. still going strong and so happy with it.

It is high time to probably invest in a newer MicroServer and get 4 3TB drives in there using ZFS RaidZ-2 (2 Redundant drives using 4 drives total) Currently just just using RaidZ-1 which is not fully optimised for 4 drives but it has been fine for me for over 5 years :D

Answer 5

Well, if you are serious about saving your data, I would suggest you build two NAS units from inexpensive computer parts. The hardware does not have to be bleeding edge, just capable of accessing Terrabyte SATA drives. You could use one of the many UNIX/Linux flavours or an old Windows version as an operating system. Then load your DVDs and disk files onto this unit and then copy it to the other. You will have to keep the two systems synced with each other in some fashion once a day, week or month depending on your paranoia level.

This is better than tapes or Blu-ray as the data is readily available and you will have two copies floating around. Once in a while, you will have to scan the NAS hard disks and decide if you need to replace them as hardware does fail.

Answer 6

I have gone over the same idea of what to do, many times, and the end results are always the same.

CD-Rs never failed me over time, a good CD even going bad, the data is mostly there after ages. Low density. The DVD-Rs though not so, even very good burns, the density of the data is high, and the ability of it to fail is higher because of that. less quality optical media is plastic (like) and all plastic gets more brittle, and whatever stuff they hold them together with eventually fails. Even great quality opticals begin to show signs of delamination after much time. . . . CD would pull it off, but it is too dang small, and DVD is too pickey (in time), for me it is not an option. Think about putting your most valuable pictures on 2X on some very good quality CDs with a very slow good burn. They will last longer than your InkJet prints :-)

Blu-ray-R, Burned blu-rays :-) I don't even have to consider, the data density is way higher, some store bought "Glass mastered" blu-rays have had failures in the short time of their existence. A mere single fingerprint. Burned, many users found that their blu-ray burns, didn't last time at all, let alone 10 years.

Tape, got a whole lot of tape, stored properly the stuff is indeed good for 10-20 years. If anyone would like to show me where I can get the Frilling tape Unit that these tapes were for, I might even be able to access their data, assuming it used the same read patterns as 15 years ago. Tape changes, tape size change, and tape machines change. The media I stored on it didn't :-) If I had continued to painstakingly move this data to anything it would be useful, right now it is as useful as a wire recording :-)

The Cloud? Not an option for me. While they will redundant my redundant. It is not their data, make them care :-) Often you also agree that they now have the data, how far you agree they have your data can include they own it. Today my data is there, tomorrow it is bought out by someone or a dead link :-) Just keeping up with the changes to the TOU, would be harder than transferring it myself. Add bandwidth caps, quantity of data, and lack of speed. Not going to happen for me.

Hard Drive.

1) Nothing is forever, if I have to keep messing with it to keep it alive, and I don't pay myself by the hour :-) to deal with it. Then it would be nice if it moves 50MB/s instead of 3.

2) Working drive I myself have had very good luck with hard drives, what else has been bit x bit accurate through so much transfer and quantity and use. As long as it has not been spun-up for 4 years, or treated poorly.

3) Storing a drive, I myself have had zero problems properly storing drives the same as tape and other media. I can wake it up after having been parked for 10 years, and it doesn't act any different.

But people who store way more drives than I do, and might not store properly, can get a hard lock on the motors. The motor won't spin from being parked for so long. Humidity and oxidation could be large factors. Say a person lived in a rainforest next to a salty ocean, the drive is not 100% sealed. Minor dehumidification, or major sealing with silica and even O2 removers, would help. Moisture, road films, smoke, kitchen grease, and all the other things that can slip into tiny holes, should be considered.

I have the best luck, speed, and compatibility and time with the hard drive, even as the other stuff failed. It has its own built-in reader :-) . It is just a matter of having (at least) 2.

My theory of 2 HD: (which didn't work for 2 DVDs)

A) One working, one that stays in the system, is allowed to live 3-4 years, is monitored, subjected to possible virus, is "known" it is watched and used, motor can't lock-up, but it can croak over time. It is maintained, updated, I know if it starts to fail. I can only hope it doesn't fail hard, because they do. I try not to let them rot on the vine.

B) One in a box, one that stays spun-down most of the time, it doesn't suffer from spin-up time, or wear, but it isn't parked for years either. This second hard drive, can be brought out of storage, updated, spun-up a bit so it doesn't lock up, and checked to see how it is doing. Be that some convenient external box, or very well stored resealed in the static bag, with adsorbers, for longer terms.

Every once in a while, be that 1 week or 1 year, the working drive and the storage drive can be updated/synced.

That just leaves C) storing it ANY other way also. And only small parts of my system have that kind of value.

That is my plan so far, and without any plan (before), that has been what has worked, when the others were failing. If my other data was SOOO valuable on the other medias, I would have used more methods for it, and kept shifting it around.

Whatever method, the one thing that is true in all cases, if you love it, it's going to be a labor of love to keep it forever. When they come out with crystal media storage, make sure you sling a copy onto it too :-)

Answer 7

It's been said a couple of times already: nothing really exists if it's not in three copies in two different formats and one of the copies should be off site.

In this way you can be sure to have at least one readable copy. Don't rely on hardware based NAS etc if they have proprietary software (most do). When the hardware crashes you probably will not find the same model hardware again...

I suggest you go with simple USB disks, but several of them. Take a copy to work and be protected against theft or fire at home (don't all eggs in the same basket). If paranoid take double copies off site.

Let the disks spin up and check them every three to six months. You want to know if they stop working. Normally disks are not good at warning in advance before they go bad. S.M.A.R.T really doesn't work that well.

I would also suggest using md5sum or some other program that can compute checksums for each and every file you have. Check these hash sums every now and then. If a file has been corrupted the checksums will not match. Then you know and can act accordingly. Without checksums the files will get corrupt without you knowing about it. You think the files are OK, but they are not.

If really paranoid don't rely on only NTFS, FAT32, HFS+ or other filesystem format. Use several. Who knows maybe you will no longer have a Windows PC when you want to find old files in the future. In this case you should also think of having your movies in several formats.

Redundancy, redundancy, redundancy and detect when errors happen!

Answer 8

Besides the question of physical storage media, there is a body of research on which digital formats are preferable for long, long storage. The US Library of Congress Sustainability of Digital Formats page is a good place to start.

You may also want to review the thinking that underlies the Archivematica technologies.

Answer 9

This question asks for opinions, so mine is that one should constantly use the best, fastest available media, make multiple backups and continue to upgrade storage as is necessary. Every medium, including stone tablet can be suddenly damaged or destroyed.

Answer 10

I don't have too much to add to what's already been said. One thing though - I would not trash your optical disks unless you really have to; I'd put them in a box and store them in a closet or something. After all, it's a form of backup already, and if you make copies from it, then until something goes wrong with them, they are a known good backup. The only optical disks I've had go bad were when I used to make labels for them. For some reason, the labels seemed to screw them up.

Personally, I prefer multiple hard drives for backup. They are quick, easy, and reliable (assuming you have more than one copy). I don't like tapes for person use. Businesses use tapes because they often store multi generations of all their backups in a safe deposit box, and it's pretty hard to fit a lot of hard drives in one. But for personal use, I've found tapes to be a lot of trouble and not always reliable.

I'd consider storing really important, irreplaceable stuff on either USB thumb drives and/or optical. I mean things like photos, not music or video (unless it's irreplaceable).

Answer 11

Seriously, consider paper for really critical stuff. Consider that an inexpensive laser printer can print at 1200 DPI, and inexpensive scanners can scan at 2-4 times that resolution. If you scale down to 600x600 resolution for reliability you can put about 2.8MB on a sheet of paper (with substantial margins).

Use good quality acid-free paper and laser (vs inkjet) printing and the medium is good for easily 200 years if reasonable care is taken in storage.

And extracting the data is so simple that one could write a scanner program from scratch, if somehow the original got lost. So long as there are computers and scanners/digital cameras the data would be accessible.

Answer 12

Well, Google uses tape to back up data (and so do others), so I think it's probably a safe bet.

Answer 13

This is hard to answer because it depends how important your data is to you and how much you are willing to spend. I'll tell you my setup, which for me works well.

First thing I would do is get a NAS. I currently own a Synology NAS and the software is great on it. With the software it currently has on there, you can backup to other devices as well.

Second thing is you can store data online in the cloud. You have to be careful about what you store though. I personally only store stuff I want to keep, but isn't anything too personal. Problem with this too, is upload speed. It took me about 30 days to upload about 230GB of stuff. You can always send a hard drive for them to upload your data though for a cost of course. I currently use Crashplan, which is pretty cheap and exactly what I need.

Third, as is stated before, you can get a Synology NAS and attached an external device on some of the models they sell. This really wouldn't matter unless you take that hard drive off site, to say a safety deposit box. Maybe backup a months worth of data and just keep switching out the hard drives some where off site.

As for my current setup, I have a Synology NAS with crashplan installed on there and it automatically backs up the data with the folders I selected to crashplan. This is a perfect setup for me, but it can be costly. The NAS itself is RAID 5, providing with redundancy.

I paid about $1200 or so for the NAS and hard drives. But that's cheap compared to trying to recover data. I find that CDs/DVDs are kind of unreliable and shouldn't be used for long term backup. A flash drive would probably be better than a CD or DVD.

If you do all of this, you should be ok in case of hardware failure or some kind of natural disaster like a fire.

Answer 14

My recommendation is to put everything onto a hard drive, then back that drive up using cloud software. My personal recommendation would be CrashPlanbecause that's what I use at home, it's affordable, lets you back up any file you want, anywhere you want, and even lets you backup to local media too. It does periodic checks to verify the integrity of your backup and has some other nice features. You can set it up with a local encryption key, so even if CrashPlan received (and complied with) a subpoena, no one would be able to read your data. I have nearly 3TB backed up with them and so far have loved paying the $5 a month charge for this rather nice service. I in no way benefit from "selling" them here, I'm just a very satisfied customer. Maybe you should take a look at their service.

Answer 15

I store my files in a HD and in the cloud. I'm currently using Google Docs because is cheaper than Amazon and DropBox.