3

In the spirit of the question LVM mirroring VS RAID1 I thought I'd push a bit further.

Suppose I have 4 identical hard drives (e.g. 1TB) that I want in a RAID setup for a backup/media server. LVM over the top seems to be current best practice for resizing, but what about the RAID layout? My options seem to be:

  • 2x 1TB RAID1 sets with LVM concatenation
  • 2x 1TB RAID1 sets with LVM striping
  • 1x 2TB RAID10 (striped 2x 1TB) set with LVM on top
  • 1x 2TB RAID6 (striped 2x1TB with 2x 1TB parity) with LVM on top

RAID-on-LVM sounds too crazy to be worth investigating, but I'd be interested if someone can make a good case for using it.

How does each of these perform in the scenario of

  • The I/O load will be a combination of low but constant (e.g. video streaming) and high but bursty (e.g. a backup system)
  • It is very likely that a drive will die unexpectedly
  • I want to upgrade (increase the capacity of) the drives 1 or 2 at a time (no longer relevant)
  • At some point in the future I might want to add another 2x HDDs in RAID1 to the LVM

What are you recommendations/experiences with any of these RAID configs?

Community
  • 1
Andrew
  • 7,772
  • 3
  • 34
  • 43
  • I assume 'upgrade the drives 1 or 2 at a time' means a size upgrade? Also, so that the layering is clear, there are no options listed where LVM is the lowest (closer to the disk) layer; a RAID array is the on-disk format in each case, with LVM storing its data on the RAID array(s). Is that accurate? – Slartibartfast Aug 31 '11 at 04:54
  • Indeed. I've heard that RAID on LVM is madness or something. – Andrew Aug 31 '11 at 06:01
  • This borders on being a non-constructive question, 'poll' questions are generally discouraged. I think your question would have been better phrased as 'Which RAID setup should I use with 4 drives in scenario X'. – Dana the Sane Sep 01 '11 at 04:36
  • @Dana Hopefully it looks a bit better now. – Andrew Sep 01 '11 at 06:03
  • Have a look at this to ensure write caches and 4K sector alignment work properly with LVM, including RAID: http://serverfault.com/questions/279571/lvm-dangers-and-caveats/279577#279577 – RichVel Sep 01 '11 at 06:23

1 Answers1

4

I would not mix md (software RAID) and md (LVM) RAID features. In the spirit of KISS, I'd go with pure md with LVM on top for snapshots / resizes.

With 4 disks going RAID6 is a Bad Idea (TM). It gives you exactly as much space as RAID 10, but with much, much worse performance (you have to calculate two parities and face read-modify-write penalty for writes smaller than stripe size).

RAID 6 gives you marginally beter resilience (any 2 disks can fail, while in RAID 10 one disk from mirror-pair can fail) at a high cost. Not worth it.

RAID 10 gives you best performance possible in this setup.

  • 2x 1TB RAID1 sets with LVM concatenation -- concurrent I/O performance depends on whether you hit the same, or different disk pairs. Solved by RAID 10, which spreads your I/O over all disks.

  • 2x 1TB RAID1 sets with LVM striping -- should give similar performance to md RAID 10, but you have more complicated setup. I love simplicity.

  • 1x 2TB RAID6 (striped 2x1TB with 2x 1TB parity) with LVM on top -- worse performance for writes, bad performance with a lost drive.

Performance characteristics I'm going to assume RAID 10, because in my opinion it has all advantages and no downsides in your scenario. You are going to be limited by a performance equivalent of a pair of HDDs. In other words you are going to be able to serve/write data at about 2x what a single HDD can do. For streaming you should be able to saturate 1Gbps link with no effort (reading or writing). For bursty data you are stuck with ~150 IOps (assuming 7.2krpm SATA drives). RAID 10 will guarantee that the load is spread among all drives for all I/O (unless you are unlucky enough to have application access data with a stride matching your RAID chunk size), and RAID 10 "far" layout should give you similar performance no matter which region of filesystem you are accessing.

Lost drive means marginal loss in read access time (you loose the "far" layout benefit for the affected mirror pair).

If you expand the storage with another pair of mirrored drives md will not be able to reshape the data to re-stripe it over the new space. Effectively you have a RAID10 + RAID1 setup, unless you backup, re-create the array and restore.

Paweł Brodacki
  • 6,451
  • 19
  • 23
  • 1
    RAID 6 is not a bad idea. With RAID 6 you can loose 2 disks of your 4 disk array without any problem. If you loose the wrong 2 disks of a 4 disk RAID10 array then your volume is dead. – ceving Aug 31 '11 at 09:19
  • 2
    RAID 6 with 6-8 1TB disks would be a good idea. With 4 disks, I don't think so. Yes, it gives you a tiny bit extra reliability over RAID 10, but it costs a lot performance-wise. – Paweł Brodacki Aug 31 '11 at 09:37