54

I currently have LVM on software RAID, but I'd like to ask you what you think it is better solution, maybe some pros and cons?

Edit: It is about software raid on lvm or lvm on software raid. I know than hardware raid is better if we are thinking about performance.

Ency
  • 1,201
  • 1
  • 19
  • 26
  • 9
    lvm on raid. raid on lvm is the devil's work. -- no reasoning. – Sirex Dec 31 '10 at 13:12
  • well, hw raid is not necessary better that sw raid. It depends on several factors. If it not a high-end one it is usally worse than linux sw raid (aka mdadm). Regarding management, stability and performance. – cstamas Feb 24 '11 at 01:34
  • @cstamas: I agree, because cheap ones are usually actually software raids. – Ency Feb 24 '11 at 06:46

6 Answers6

60

Your current setup is like this: 

| / | /var | /usr | /home  |
 --------------------------
|       LVM Volume         |
 --------------------------
|       RAID Volume        |
 --------------------------
| Disk 1 | Disk 2 | Disk 3 |

It's a much simpler setup with more flexibility. You can use all of the disks in the RAID volume and slice and dice them whatever way you like with LVM. The other way isn't even worth thinking about - it's ridiculously complicated and you lose the benefits of LVM at the filesystem level.

If you tried to RAID LVM volumes, you would be left with a normal device without any of the LVM volume benefits (e.g. growing filesystems etc.)

Perleone
  • 135
  • 8
Philip Reynolds
  • 9,751
  • 1
  • 32
  • 33
11

This is an old question, technology has advanced and the recommended setup is to use built-in RAID support of LVM (see here for setup https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Logical_Volume_Manager_Administration/raid_volumes.html ), especially if you are using SSD. Red Hat does not recommend the use of RAID 1/5/6/10 with SSD since mdadm will write the complete partition to ensure proper functioning of checksums. This can lead to faster degradation of the SSD, as stated here https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-single/Storage_Administration_Guide/index.html#ssddeploy

Pozzo-Balbi
  • 382
  • 2
  • 5
  • 2
    USER irwinr suggested: Edited Feb 4th 2018: I am unable to find any reference to RedHat recommending LVM RAID over mdadm RAID nor any caution against using RAID with SSDs. In fact the #ssddeploy tag does not even appear in the HTML for the 2nd linked page. ---- Thanks irwinr, it looks like RedHat updated their documentation. Still mdadm syncs whole partition, while "smart" LVM does not, hence less unnecessary disk writes, hence longer life for your SSD. – Pozzo-Balbi Jun 30 '18 at 00:58
  • Just a thought: although you have the basic idea in a comment you might want to edit it into the answer maybe even with a disclaimer; that is to say that you should have a warning of some kind that says under what circumstance(s) you're suggesting your method. – Pryftan Dec 18 '18 at 14:08
  • 1
    I'm guessing LVM is just using the mdam option `--assume-clean` behind the scene i.e. still no advantage of using LVM-RAID instead of mdadm (at leat not in that regard). If your're happy with no full resync when create array with mdadm just use that optino yourself. Today, 4 years later, resync the whole SSD disk make no difference in regards to wear-out as the endurance-levels on newer SSD are so high. – MrCalvin Oct 22 '20 at 04:12
  • There is a huge benefit in having LV based RAID. I do have a LV with RAID1 for more hard to reproduce data and normal LVs for the rest, this gives me maximum flexibility for size vs. redundancy considerations. – wiebel Sep 20 '22 at 10:21
4

Your current setup is fine. This is the recommended way to do it.

Raid deals with keeping the bits secure/redundant/fast/whatever and LVM helps you present them in a esasy to use way.

cstamas
  • 6,607
  • 24
  • 42
2

have hardware raid and you can have lvm on top - best combination.

silviud
  • 2,677
  • 2
  • 16
  • 19
  • 3
    Good enterprise solution, but an awful suggestion for homelab! You need a service contract with the hardware supplier or you're in deep trouble if your RAID hardware has a fault - you need to find compatible proprietary hardware to replace it. – Gareth Davidson Mar 15 '19 at 10:57
0

I am assuming you mean Hardware RAID with LVM on Top, vs. LVM and Software RAID on top of LVM. If so, I always advise to opt for hardware based RAID first. Software RAID is just that, while overhead is small, hardware RAID performance will be better 9 out of 10 times. Of course, the methodology will very much depend on your final goal. What are you trying to achieve (performance, protection, etc., etc.)

slashdot
  • 651
  • 5
  • 7
-3

I think it makes sens to use RAID over LVM if you want to split your disks between a RAID 0 volume and a RAID 1 volume.

With this you can't realocate space between RAID0 and RAID1

| / | /var | /usr | /home  |
 --------------------------
|       LVM Volume 2       |
 --------------------------
|    RAID 0   |   Raid 1   |
 --------------------------
| Disk 1 | Disk 2 | Disk 3 |

With this you can

| / | /var | /usr | /home  |
 --------------------------
|       LVM Volume 2       |
 --------------------------
|    RAID 0   |   Raid 1   |
 --------------------------
|       LVM Volume 1       |
 --------------------------
| Disk 1 | Disk 2 | Disk 3 |

the good thing is that you can also still move the LVM Volume 2 logical volumes between raid 0 et and raid 1 volume, by using pvmove command

the bad thing is that the setup is complex It would be better if LVM had a better integration of software raid features.

mirak
  • 1
  • 6
    I don't know enough storage goodies to say *why* this is a bad idea, but it leaves me quite unsettled and a distinct feeling of dread. – Scott Pack Nov 07 '12 at 16:15
  • 2
    If "RAID" here was RAID 1 / 5 / some other form of redundant RAID, you would not get any redundancy. If any disks under the LVM Volume 1 failed, you would lose both RAID volumes, as the LVM volume would be dead (OK, you could force it online still assuming that it wasn't striping, but a large amount of the data would now be gone, and the RAID ontop of the failing LVM volume probably won't be able to recover...). I don't know why you would do this, but don't. – BSchlinker Nov 07 '15 at 00:16
  • 3
    Scott, it seems that your dread was justified. I used mdadm RAID over LVM2 and got myself into trouble - http://serverfault.com/questions/826479/ubuntu-16-04-frozen-mdadm-array - let my experience be warning to others. – Ghostrider Jan 16 '17 at 06:58