3

The following is qouted from: Intro to Nested Raid

Linux md allows you to combine “near” and “far” RAID configurations. For example, in a m=2 and f=2 configuration with four drives (the minimum allowed) the data layout looks the following (omitted). Notice that there are four copies of each data chunk! Also notice that we can lose two drives without losing access to the data. However, in contrast to the normal RAID-10 we can lose any two drives and still retain access to all of the data. This is much better than RAID-10 where we can lose any single drive, but only specific combinations of two drives. The capacity of this particular four-drive “near” and “far” RAID configuration is the following. | Capacity = (n/2) * capacity of single disk | So we don’t gain any extra capacity relative to RAID-10 but we do gain the ability to lose any drives (a strong plus in my book).

I don't know if there is a mistake and I can't really find any other mention of the huge (in my opinion) benefit of the MD-RAID10 n2 f2 layout - the ability to loose ANY TWO (out of a 4 disk array) drives out of the array, as opposed to standard RAID10, in which you can only loose certain pairs of disks. Is this true for absolutely any MD RAID10 n2 f2 layout? Also, what is the capacity of a 4 disk md raid 10 n2 f2 layout? The article is wrong and some of the comments disagree.

Thanks!

Diego
  • 135
  • 7
  • possible duplicate of http://serverfault.com/questions/139022/explain-mds-raid10-f2 – Niko S P Feb 29 '12 at 00:33
  • That question is asking about f2, and they just answered with man pages - I'm asking specifically about n2 f2 and whether it actually allows you to loose **any** two drives out of a 4 drive mdraid10 array – Diego Feb 29 '12 at 00:42
  • I don't understand why it would be limited to only loosing two drives. Since n2 f2 has 4 copies of all the data, then it seems like you should be able to get away with loosing 3 drives in a 4 disk array. – Zoredache Feb 29 '12 at 00:54
  • so with 4 drives at n2 f2, your capicity would be (4 disks / 4 copies )*capacity of a drive, and you could loose any three drives? – Diego Feb 29 '12 at 01:38

1 Answers1

6

They are correct, such an arrangement would allow an any-two failure to be survivable.

The 'N=2' setup ensures that first two of the replicated blocks are on equivalent sectors of different disks.

The 'F=2' setup ensures that the second two of the replicated blocks do not share disks with the N=2 replicated blocks, so long as the array meets or exceeds N+F drives in size.

As for the capacity math, the article has it incorrect (see below). Let's take a look at a few examples:

A 4-drive 2x2 array

  • There are a total of 4 drives in the array
  • There are 2 'near' replicas
  • There are 2 'far' replicas
  • The 'stripe width' is therefore 4 blocks.

Each block is therefore replicated four times. Capacity is the size of a single drive.

A 5-drive 2x2 array

  • There are a total of 5 drives in the array
  • There are 2 'near' replicas
  • There are 2 'far' replicas
  • The 'stripe width' is therefore 4 blocks.

As with the 4 drive array, each block is replicated four times. However, the extra drive provides another full-drives worth of blocks to expand into. Capacity is the size of two drives.

Put it a different way:

Given:

  • N = Number of Drives
  • R = Number of Replicated blocks per stripe
  • S = Size of Drives

Capacity = S * ( N - (R-1))

4-Drive Array:
N = 4
R = 4
Capacity = S ( 4 - (4-1)) = S * (4-3) = S * 1

5-Drive Array:
N = 5
R = 4
Capacity = S * ( 5 - (4-1)) = S * (5 - 3) = S * 2

The 'any two can fail' condition only exists if N ≧ R. In fact, with R=4, any three can fail. Again, only if N ≧ R.

I must point out a math error in the article itself. To quote:

The capacity of this particular four-drive “near” and “far” RAID configuration is the following.

Capacity = (n/2) * capacity of single disk

This is incorrect. The 2 in that formula is supposed to be the number of replicated blocks. Which in a 2x2 setup is 4. This is clearly demonstrated in the diagram, where the "A1" block shows up four times. The author gets it right for the 3-drive examples, as those formula show division by 3.

[Example is for a two-replica RAID on three disks]

Capacity = 2/3 * capacity of single disk

This is further supported by the man-page for md:

Finally, it is possible to have an array with both 'near' and 'far' copies. If an array is configured with 2 near copies and 2 far copies, then there will be a total of 4 copies of each block, each on a different drive. This is an artifact of the implementation and is unlikely to be of real value.

So a 2-by-2 RAID setup will have four copies. Therefore, a four drive implementation of a 2-by-2 RAID will have the capacity of a single drive.

The author's thesis that a near/far RAID setup will provide additional protection above and beyond normal R10 is weak. The protection does not come from the near/far setup, the protection come from there being more than 2 replications of the data.

RAID configs where data is replicated R times can tolerate up to R-1 disk failures. More failures can be tolerated so long as the extra failed devices are in already-failed replication-sets. This is why a mirror pair (R=2) of RAID0 devices can tolerate a single drive failure. If R equals the number of drives (N) you can have all but one fail and still maintain service.

Diego
  • 135
  • 7
sysadmin1138
  • 131,083
  • 18
  • 173
  • 296
  • beautiful answer - so it seems to me that this layout seems like overkill. Ideally, I would want something like RAID10, except I can loose ANY TWO drives, while keeping (2 * size of one drive) out of a 4 disk array - any suggestions? – Diego Feb 29 '12 at 02:54
  • @Diego That would require 3 for R, and I'm unclear on how that works with MDRAID. N and F each have to be 2 at minimum, which suggests R's lower bound is 4. – sysadmin1138 Feb 29 '12 at 03:34
  • hmm.. So it seems to me I have to decide whether to go for standard MDRAID10, meaning I keep 50% capacity of the 4 drives, but can only loose certain pairs of drives, OR I can go with MDRAID10 n2f2 and keep 25% capacity, but loose any three... I'm purchasing consumer grade drives because I can't afford enterprise right now, but I don't think they will be under such heavy load with only about 10 users working off it to read/write documents/research files. – Diego Feb 29 '12 at 18:14
  • The article says that the n2f2 setup has a capacity of (n/2)*capacity of single drive where n is the number of drives. If n=4 then our capacity is 2*capacity of single. You're saying that under n2f2 capacity of the array is 1*capacity of single. So the article is wrong? – Diego Apr 27 '12 at 22:13
  • @Diego Looks like I was doing the math for a RAID11, not 10. Aaaaand `man 4 md` says the R1 driver doesn't do offsets like the R10 driver. I'll update the answer later. – sysadmin1138 Apr 28 '12 at 12:18
  • Hey @sysadmin1138 just wondering if you got a change to update the answer? – Diego Jun 25 '12 at 22:37
  • hey @sysadmin1138 any chance you can update the answer? I often use this as a reference, and in case other people do as well. – Diego Jul 23 '12 at 20:01