Which RAID/LVM configuration for new 2x8TB file server is safest?

Question

I just built a totally new PC to be a fileserver. It has one 60gb SSD for OS (ubuntu 12.04), and 8 2tb data drives in it. I am looking for the safest, most stable way to set up a single giant storage pool with redundancy.

Considerations:

• My TOP priority is protection of the data. Availability, reliability, resiliency, recoverability, and risk management. All of those words which mean not losing my files.
• I'm not merely concerned with the theoretical risks implied by the nature of the raid types, but also practical considerations of simplicity, and minimum points of failure. Especially considering I am not an expert in RAID or FS technology and my ability to troubleshoot is limited to what I can google up.
• This is not a write-heavy, like database servers with transactions day, it's mainly to hold media for various streaming, as well as all of my users files for network access.
• My worst nightmare is to find that due to some chaos-theory alpha particle shot out of the sun landing in a capacitor, to have some error in my HDDs/metadata/motherboard/software/whatever that renders the entire array unrecoverable. DOOM&H8!
• Because I will be streaming data, read speed advantages are appreciated (but secondary)
• Eventually I might want to add drives, with LVM pooling RAID1 pairs, this is easy. Would RAID6 or RAID10 arrays have to be rebuilt from scratch with blank drives?
• If this is relevant, I have my 8 drives plugged into the 8 onboard SATA slots on my ASRock Z77 Extreme4 motherboard, 4 of which are SATA2 and 4 are SATA3. The OS is on a SSD hooked to a PCIe SATA controller. 8gb of RAM.
• See #1 - My TOP priority is protection of the data.

I have read that RAID6 is actually less risky since any 2 disks can fail instead of only one per mirror-pair, and the main drawback of RAID6 is write speed, which I am not to concerned about. I have accepted the loss of half my drives to mirroring, so I dont want to sound greedy for even considering RAID6, but if its actually SAFER, faster for reads, AND lets me keep 12 gb instead of just 8, maybe it would be stupid not to. Or maybe I am missing a bigger picture

I guess what I am asking for is the drawbacks of, or which one is best suited for data protection, and to a lesser extent read speed, in an 8-12 drive array.

• LVM(4x2tb) + nightly cronjob backups of independant disks
• LVM(4xRAID1)
• (RAID10)
• (RAID6)

Thanks for your help!

Answer 1

Safest? Two-axle RAID60. Or better, ZFS. Don't forget your backups!

Answer 2

I really didn't appreciate your last consideration -- If I lose my files, I will take a bath with the toaster. That will put people off from responding, even though it is supposed to be funny (or is it?).

Moving along, data resilient storage is a myth and everything can fail. Be prepared for it. From your requirements, a couple of observations,

[1] Your single SSD drive can fail. You are not an expert on RAID or FS, how will you recover your data? Is it an enterprise class SSD? How will SSD help you here?

[2] I assume you are using "desktop" class drives. They are not designed for 24x7 operation nor they are built to last longer. Typical desktop drives come with 1 year or 2 year warranty while an enterprise class drive comes with 3 to 5 year warranty.

[3] You are using onboard motherboard RAID. While this may work, it is hard to setup and manage. A true hardware RAID controller gives you better protection, performance and easier to work with. If you can afford it, go for it. Ensure the firmware on the RAID controller is up-to-date though.

[4] Your motherboard, RAM, power supply, power source all can fail. Are you prepared to recover data from any of these since failure could potentially cause data corruption/loss. You may lose some data in transit, if you do not turn off caching on the HDD drives (set it to write-through mode).

These are just a few pointers,

• Among 8 drives, set 2 drives dedicated as hot spare. You will need them.
• Buy enterprise class drives if you can
• Use LVM or zfs as your filesystem. Check FreeNAS.
• Take regular backups and snapshots of your data including your configuration files especially for your RAID and store them elsewhere.
• Avoid all single point of failures -- boot drive, power supply, network card (bonding/teaming will help).
• Buy motherboard with longer warranty. Some are available with 5 year warranty and are military/industrial grade built.
• Use RAID 6
• Ensure sufficient cooling and airflow for the server
• Run complete periodic monitoring to be proactive. Check for CPU/HDD temperatures. Check CPU/Cooling Fan speed. Check SMART for possible disk failures and replace them in advance. Check for filesystem consistency.

If you can afford it, and data is very critical, build another storage (with different manufacturer components) and have redundancy using DRBD (or anything else) for replication across both storage servers. At work, we keep replication factor of 3 with GlusterFS on enterprise grade hardware.

Hope this helps.

Answer 3

If your data is truly that important that losing it means bathing with the electrical appliances, then there's no way around needing a second server on which to mirror the devices, and having true backups. Anything you do on just one server has at least one single point of failure. (At a minimum, the server itself - for the most basic example, what if there's a fire? Or a the PSU goes out and takes everything else with it? Etc.)

And you'll need true backups with some kind of retention for the same reason. Not having any backup means that data corruption could take out the whole thing as well, because if you're replicating the one system to the other, you're likely to replicate any corruption that occurs as well.

And since no one's said it here yet, RAID6 and big SATA disks is not a good combination. It does provide more redundancy than a RAID5 setup, but not by enough. On such large disks, you're almost certain to have a read error on one of your parity stripes, which really means that on one of the disks, somewhere (but you won't know where), you're effectively running in RAID5. Which is all well and good until a disk fails. Then you're running on failed array RAID, which is too big a risk with critical data.

And on the topic of disk failures, the issue RAID6 has is the same as RAID5 - parity bit calculations are complex, and take a lot of time, resources and I/O to do, meaning that you're actually at a bigger risk of array failure during a rebuild than you'd think, and I've actually seen the nightmare scenario of a RAID6 array (with a hot-spare, even) failing during rebuild - not terribly shocking considering all the I/O a rebuild takes, but terribly devastating if it happens to you.

I'd recommend running it in 2xRAID10, for what it's worth - you get performance benefits, as well as some measure of redundancy. Rebuilding a failed disk in a mirror RAID is a lot faster, and therefore safer, than you'll find in any parity-based system. If it matters, I have a 12 disk RAID6 array at home (12x2TB near-line SAS), so I definitely do like RAID6, but it's just not safe enough for critical data.

As a final note, you don't want to kill yourself by electrocution. It's quite excruciating, so you'd probably be better served by a different method of suicide.

Answer 4

The simpler is the better. LVM on LinuxSoftRAID-6 looks like quite reasonable. Another important thing left to consider is FS on top. In Linux world up to this moment only Btrfs has built-in data verification mechanisms. But I doubt its developers would share your attitude regarding bath-n-toaster in case. ;)

Anyways, just before using RAID6 take drills for loosing 1, and 2 disks. Go even further — take out 3 or 4, then make it working back. With LSR it's quite possible. And piece of advice — use its bitmaps, it would save hell-a-lot of the time for you.

Answer 5

If you're going to go the RAID 6 route, then do yourself a favor: Visit eBay and pick up a couple of PERC 6/i cards (one as a spare) and two SFF-8484 to 4xSFF-8482 cables to hook up the drives. You'll thank me later.

If you decide to go with ZFS, then go whole hog and set up NexentaStor. That's probably about the safest you'll get with your data in its intended, uh, home.

And think very carefully about a backup strategy; neither RAID nor ZFS are backup. Backup of terabytes of data is not easy or cheap; it's not usually something you can do over the Internet, and terabytes of "cloud" storage would set you back a lot more than the drives anyway.

Answer 6

There are three kinds of service available... Fast, Good and Cheap. Pick two.

There is no "best" just "better given xYz tradeoffs."

Given some of the trade offs you listed ditching DASD (Direct Attached Storage Device) might be a good idea. You may find looking at something like GlusterFS (Red Hat Storage) or CEPH more useful. Gluster for instance replicates data over a cluster of file servers, where that cluster can lose one or more disks, and potentially one more more nodes and maintain data integrity.

Overall your list lends itself to two or more solutions. You may need to use fast DASD or SAN for a RDBMS, in which case you will want to spend time and money investing in mirroring and so forth. Other use cases may be very well suited by something like Gluster which can allow for easier scaling and can abstract you from needing to know the intricacies of each RAID type.

Bottom line, you need to do more data analysis on your use case.

Answer 7

Please read The Tao of Backup.

You don't need the best possible RAID — simple RAID5 would suffice. RAID is a high-availability solution, not a data protection solution. What you really need is a very good backup solution — one that will backup automatically, often, keep several old backups and backup periodically offsite (this is not automatical — it can be a for example a set of disks in a deposit box in a bank).