Why is RAID 1+6 not a more common layout?

Question

Why are the nested RAID levels 1+5 or 1+6 almost unheard of? The nested RAID levels Wikipedia article is currently missing their sections. I don't understand why they are not more common than RAID 1+0, especially when compared to RAID 1+0 triple mirroring.

It is apparent that rebuilding time is becoming increasingly problematic as drive capacities are increasing faster than their performance or reliability. I'm told that RAID 1 rebuilds quicker and that a RAID 0 array of RAID 1 pairs avoids the issue, but surely so would a RAID 5 or 6 array of RAID 1 pairs. I'd at least expect them to be a common alternative to RAID 1+0.

For 16 of 1TB drives, here are my calculations of the naïve probability of resorting to backup, i.e. with the simplifying assumption that the drives are independent with even probability:

RAID | storage | cumulative probabilities of resorting to backup /m
 1+0 |     8TB | 0, 67, 200, 385, 590, 776, 910, 980, 1000, 1000, 1000
 1+5 |     7TB | 0,  0,   0,  15,  77, 217, 441, 702,  910, 1000, 1000
 1+6 |     6TB | 0,  0,   0,   0,   0,   7,  49, 179,  441,  776, 1000
(m = 0.001, i.e. milli.)

If this is correct then it's quite clear that RAID 1+6 is exceptionally more reliable than RAID 1+0 for only a 25% reduction in storage capacity. As is the case in general, the theoretical write throughput (not counting seek times) is storage capacity / array size × number of drives × write throughput of the slowest drive in the array (RAID levels with redundancy have a higher write amplification for writes that don't fill a stripe but this depends on chunk size), and the theoretical read throughput is the sum of the read throughputs of the drives in the array (except that RAID 0, RAID 5, and RAID 6 can still be theoretically limited by the slowest, 2nd slowest, and 3rd slowest drive read throughputs respectively). I.e., assuming identical drives, that would be respectively 8×, 7×, or 6× maximum write throughput and 16× maximum read throughput.

Furthermore, consider a RAID 0 quadruple of RAID 1 triples, i.e. RAID 1+0 triple mirroring of 12 drives, and a RAID 6 sextuple of RAID 1 pairs, i.e. RAID 1+6 of 12 drives. Again, these are identical 1TB drives. Both layouts have the same number of drives (12), the same amount of storage capacity (4TB), the same proportion of redundancy (2/3), the same maximum write throughput (4×), and the same maximum read throughput (12×). Here are my calculations (so far):

RAID      | cumulative probabilities of resorting to backup /m
1+0 (4×3) | 0, 0, 18,  ?,   ?,   ?,   ?,   ?, 1000
1+6 (6×2) | 0, 0,  0,  0,   0,  22, 152, 515, 1000

Yes, this may look like overkill, but where triple mirroring is used to split-off a clone for backup, RAID 1+6 can just as well be used, simply by freezing and removing 1 of each drive of all but 2 of the RAID 1 pairs, and while doing so, it still has far better reliability when degraded than the degraded RAID 1+0 array. Here are my calculations for 12 drives degraded by 4 in this manner:

RAID      | cumulative probabilities of resorting to backup /m
1+0 (4×3) | (0, 0, 0, 0), 0, 143, 429, 771, 1000
1+6 (6×2) | (0, 0, 0, 0), 0,   0,  71, 414, 1000

Read throughput, however, could be degraded down to 6× during this time for RAID 1+6, whereas RAID 1+0 is only reduced to 8×. Nevertheless, if a drive fails while the array is in this degraded state, the RAID 1+6 array would have a 50–50 chance of staying at about 6× or being limited further to 5×, whereas the RAID 1+0 array would be limited down to a 4× bottleneck. Write throughput should be pretty unaffected (it may even increase if the drives taken for backup were the limiting slowest drives).

In fact, both can be seen of as ‘triple mirroring’ because the degraded RAID 1+6 array is capable of splitting-off an additional RAID 6 group of 4 drives. In other words, this 12-drive RAID 1+6 layout can be divided into 3 degraded (but functional) RAID 6 arrays!

So is it just that most people haven't gone into the maths in detail? Will we be seeing more RAID 1+6 in the future?

Answer 1

Generally I'd say RAID 1+0 will tend to be more widely used than 1+5 or 1+6 because RAID 1+0 is reliable enough and provides marginally better performance and more usable storage.

I think most people would take the failure of a full RAID 1 pair within the RAID 1+0 group as a pretty incredibly rare event that's worth breaking out the backups for - and probably aren't too enthusiastic about getting under 50% of their physical disk as usable space.

If you need better reliability than RAID 1+0, then go for it! ..but most people probably don't need that.

Answer 2

The practical answer lies somewhere at the intersection of hardware RAID controller specifications, average disk sizes, drive form-factors and server design.

Most hardware RAID controllers are limited in the RAID levels they support. Here are the RAID options for an HP ProLiant Smart Array controller:

[raid=0|1|1adm|1+0|1+0adm|5|50|6|60]

note: the "adm" is just triple-mirroring

LSI RAID controllers support: 0, 1, 5, 6, 10, 50, and 60

So these controllers are only capable of RAID 50 and 60 as nested levels. LSI (née Dell PERC) and HP comprise most of the enterprise server storage adapter market. That's the major reason you don't see something like RAID 1+6, or RAID 61 in the field.

Beyond that consideration, nested RAID levels beyond RAID 10 require a relatively large number of disks. Given the increasing drive capacities available today (with 3.5" nearline SAS and SATA drives), coupled with the fact that many server chassis are designed around 8 x 2.5" drive cages, there isn't much of an opportunity to physically configure RAID 1+6, or RAID 61.

The areas where you may see something like RAID 1+6 would be large chassis software RAID solutions. Linux MD RAID or ZFS are definitely capable of it. But by that time, drive failure can be mitigated by hot or cold-spare disks. RAID reliability isn't much of an issue these days, provided you avoid toxic RAID level and hardware combinations (e.g. RAID 5 and 6TB disks). In addition, read and write performance would be abstracted by tiering and caching layers. Average storage workloads typically benefit from one or the other.

So in the end, it seems as though the need/demand just isn't there.

Answer 3

• You have diminishing returns on reliability. RAID 6 is pretty unlikely to compound failure even on nasty SATA drives with a 1 in 10^14 UBER rate. On FC/SAS drives your UBER is 1 in 10^16 and you get considerably more performance too.

• RAID group reliability doesn't protect you against accidental deletion. (so you need the backups anyway)

• beyond certain levels of RAIDing, your odds of a compound failure on disks becomes lower than compound failure of supporting infrastructure (power, network, aircon leak, etc.)

• Write penalty. Each incoming write on your RAID 61 will trigger 12 IO operations (naively done). RAID 6 is already painful in 'low tier' scenarios in terms of IOPs per TB random write. (and in higher tier, your failure rate is 100x better anyway)

• it's not '25% reduction' it's a further 25% reduction. Your 16TB is turning into 6TB. So you're getting 37.5% usable storage. You need 3x as many disks per capacity, and 3x as much datacentre space. You would probably get more reliability by simply making smaller RAID6 sets. I haven't done the number crunching, but try - for example the sums of RAID 6 in 3x 3+2 sets (15 drives, less storage overhead than your RAID10). Or doing 3 way mirrors instead.

Having said that - it's more common than you think to do it for multi-site DR. I run replicated storage arrays where I've got RAID5/6/DP RAID groups asynchronously or synchronously to a DR site. (Don't do sync if you can possibly avoid it - it looks good, it's actually horrible).

With my NetApps, that's a metrocluster with some mirrored aggregates. With my VMAXes we've Symmetrix Remote Data Facility (SRDF). And my 3PARs do remote copy.

It's expensive, but provides 'data centre catching fire' levels of DR.

Regarding triple mirrors - I've used them, but not as direct RAID resilience measures, but rather as full clones as part of a backup strategy. Sync a third mirror, split it, mount it on a separate server and back that up using entirely different infrastructure. And sometimes rotate the third mirror as a recovery option.

The point I'm trying to make is that in my direct experience as a storage admin - in a ~40,000 spindle estate (yes, we're replacing tens of drives daily) - we've had to go to backups for a variety of reasons in the last 5 years, but none of them have been RAID group failure. We do debate the relative merits and acceptable recovery time, recovery point and outage windows. And underpinning all of this is ALWAYS the cost of the extra resilience.

Our array all media scrub and failure predict, and aggressively spare and test drives.

Even if there were a suitable RAID implementation, cost-benefit just isn't there. The money spent on the storage space would be better invested in a longer retention or more frequent backup cycle. Or faster comms. Or just generally faster spindles, because even with identical resilience numbers, faster rebuilding of spares improves your compound failure probability.

So I think I would therefore offer the answer to your question:

You do not see RAID 1+6 and 1+5 very often, because the cost benefit simply doesn't stack up. Given a finite amount of money, and given a need to implement a backup solution in the first place, all you're doing is spending money to reduce your outage frequency. There are better ways to spend that money.

Answer 4

Modern and advanced systems don't implement shapes like that because they're excessively complicated, completely unnecessary, and contrary to any semblance of efficiency.

As others have pointed out, the ratio of raw space to usable space is essentially 3:1. That is essentially three copies (two redundant copies). Because of the calculation cost of "raid6" (twice over, if mirrored), and the resulting loss of IOPS, this is very inefficient. In ZFS, which is very well designed and tuned, the equivalent solution, capacity-wise would be to create a stripe of 3-way mirrors.

As an example, instead of a mirror of 6-way raid6/raidz2 shapes (12 drives total), which would be very inefficient (also not something ZFS has any mechanism to implement), you would have 4x 3-way mirrors (also 12 drives). And instead of 1 drive worth of IOPS, you would have 4 drives worth of IOPS. Especially with virtual machines, that is a vast difference. The total bandwidth for the two shapes may be very similar in sequential reads/writes, but the stripe of 3-way mirrors would definitely be more responsive with random read/write.

To sum up: raid1+6 is just generally impractical, inefficient, and unsurprisingly not anything anyone serious about storage would consider developing.

To clarify the IOPS disparity: With a mirror of raid6/raidz2 shapes, with each write, all 12 drives must act as one. There is no ability for the total shape to split the activity up into multiple actions that multiple shapes can perform independently. With a stripe of 3-way mirrors, each write may be something that only one of the 4 mirrors must deal with, so another write that comes in doesn't have to wait for the whole omnibus shape to deal with before looking at further actions.

Answer 5

Since noone said it directly enough: Raid6 write performance is not marginally worse. It is horrible beyond description if put under load.

Sequential writing is OK and as long as caching, write merging etc. is able to cover it up, it looks ok. Under high load, things look bad and this is the main reason a 1+5/6 setup is almost never used.

Answer 6

Seek times

The issue is that, the write seek amplification behaves very differently to the write throughput amplification. The minimal write throughput amplification with parity occurs when an entire stripe is written at once (let's call this adjective ‘full-stripe’) yet the minimal write seek amplification occurs, conversely, when the entire write following a seek in the virtual device fits into a single chunk. Before going into detail, the relationships are much easier to convey in tabulated form:

RAID | write throughput amplification factor | write seek amplification factor
     | full-stripe (e.g.) | single-chunk     | full-stripe  | single-chunk
   0 | 1           ;  1   | 1           ;  1 | n       ; 12 | 1           ;  1
   1 | n           ; 12   | n           ; 12 | n       ; 12 | n           ; 12
   5 | n/(n - 1)   ; ~1.1 | min [3, n]  ;  3 | n       ; 12 | min [3, n]  ;  3
   6 | n/(n - 2)   ;  1.2 | min [5, n]  ;  5 | n       ; 12 | min [5, n]  ;  5
*1+0 | n₁          ;  3   | n₁          ;  3 | n       ; 12 | n₁          ;  3*
 1+5 | n/(n₅ - 1)  ;  2.4 | expr₁       ;  5 | n       ; 12 | expr₁       ;  5
*1+6 | n/(n₆ - 2)  ;  3   | expr₂       ;  8 | n       ; 12 | expr₂       ;  8*
expr₁ = 2n₁ + min [1, n₅ - 2]
expr₂ = 3n₁ + min [2, n₆ - 3]

where n is the total number of drives, n₁ is the number of drives in the RAID 1 groups, and n₅ and n₆ are the number of groups in the RAID 5 or RAID 6 arrays respectively. Examples relate to the 12-drive example in the question (relevant rows are ‘*bolded*’); examples for RAID levels 1+0, 1+5, 1+6 are 4×3, 6×2, 6×2 respectively.

Note that only the full-stripe write throughput amplification factor is directly related to the proportion of redundancy. The single-chunk cases are more complicated for those with parity. They arise because writing a single chunk requires reading whichever is easiest of the parity chunks or the other data chunks, before writing the parity chunks along with the new data chunk. (They are not directly multiplicative because the induced reads must instead be multiplied by the respective read throughput/seek amplification factor for RAID 1, both being 1; see below.)

Unfortunately, choosing a chunk size that minimises this extra write throughput amplification has the side-effect of actually maximising the write seek amplification. For tiny writes with a negligible write time compared to the seek time, the write performance of striping with a very small chunk size (to be full-stripe) is only 1×, like mirroring, as it requires all of the drives to seek to the chunks for each write and the gained throughput from mobilising all of these drives is irrelevant. It has divided the ratio of write time to seek time by the number of drives in the array, but for tiny writes this was already negligible. It wouldn't make sense to use a chunk size so small as to make even tiny writes be full-stripe. For writes small enough to feel the effects of seeking, it is best that they fit within a single chunk.

RAID | large contiguous write throughput    | concurrent tiny writes throughput
     | full-stripe    | single-chunk        | full-stripe | single-chunk
   0 | n×       ; 12× | n×          ; 12×   | 1×     ; 1× | n×          ; 12×
   1 | 1×       ;  1× | 1×          ;  1×   | 1×     ; 1× | 1×          ;  1×
   5 | (n - 1)× ; 11× | max[n/3, 1]×;  4×   | 1×     ; 1× | max[n/3, 1]×;  4×
   6 | (n - 2)× ; 10× | max[n/5, 1]×;  2.4× | 1×     ; 1× | max[n/5, 1]×;  2.4×
*1+0 | n₀×      ;  4× | n₀×         ;  4×   | 1×     ; 1× | n₀×         ;  4×  *
 1+5 | (n₅ - 1)×;  5× | expr₃×      ;  2.4× | 1×     ; 1× | expr₃×      ;  2.4×
*1+6 | (n₆ - 2)×;  4× | expr₄×      ;  1.5× | 1×     ; 1× | expr₄×      ;  1.5×*
expr₃ = n/(2n₁ + min [1, n₅ - 2]) = max [n/(2n₁ + 1), n/(2n₁ + n₅ - 2)]
expr₄ = n/(3n₁ + min [2, n₆ - 3]) = max [n/(3n₁ + 2), n/(3n₁ + n₆ - 3)]

Note: The middle 2 throughput columns can be ignored given a sensible chunk size that is bigger than the writes for which the seek time is significant, but small enough such that large writes are full-stripe. The large chunk size of the 2nd throughput column is more akin to spanned drives. A ‘tiny’ write is where the effect of throughput is negligible.

Having an inappropriately small chunk size also increases the effect of seek amplification for reads, though not as much and only in the full-stripe case.

RAID | read throughput amplification factor | read seek amplification factor
     | full-stripe      | single-chunk      | full-stripe (e.g.) | single-chunk
   0 | 1                | 1                 | n      to n;    12 | 1
   1 | 1                | 1                 | 1      to n;  1–12 | 1
   5 | 1                | 1                 | n - 1  to n; 11–12 | 1
   6 | 1                | 1                 | n - 2  to n; 10–12 | 1
*1+0 | 1                | 1                 | n₀     to n;  4–12 | 1           *
 1+5 | 1                | 1                 | n₅ - 1 to n;  5–12 | 1
*1+6 | 1                | 1                 | n₆ - 2 to n;  4–12 | 1           *

Note: The ‘to n’ is because when there is only one read happening concurrently, it is theoretically possible to mobilise all drives to seek to appropriate places and collectively read the data for maximum large contiguous read throughput.

RAID | large contiguous read throughput | concurrent tiny reads throughput
     | full-stripe (e.g.)| single-chunk | full-stripe         | single-chunk
   0 | n×          ; 12× | n×     ; 12× | 1×          ;  1×   | n×     ; 12×
   1 | n×          ; 12× | n×     ; 12× | n×          ; 12×   | n×     ; 12×
   5 | n×          ; 12× | n×     ; 12× | n/(n - 1)×  ; ~1.1× | n×     ; 12×
   6 | n×          ; 12× | n×     ; 12× | n/(n - 2)×  ;  1.2× | n×     ; 12×
*1+0 | n×          ; 12× | n×     ; 12× | n₁×         ;  3×   | n×     ; 12×*
 1+5 | n×          ; 12× | n×     ; 12× | n/(n₅ - 1)× ;  2.4× | n×     ; 12×
*1+6 | n×          ; 12× | n×     ; 12× | n/(n₆ - 2)× ;  3×   | n×     ; 12×*

Note: Again, the middle 2 throughput columns can be ignored given a sensible chunk size. The 3rd throughput column is again closely connected to the proportion of redundancy.

However, a large enough chunk size means that tiny reads are never full-stripe. So given an efficient implementation and appropriate chunk size, read performance should be proportional to the number of identical drives when not degraded.

So really, ‘amplification factor’ is a lot more complicated than the formula in the question, where only full-stripe throughput amplification had been considered. In particular, the write performance of 6×2 RAID 1+6 for concurrent writes that are small enough to be seek-bound will be worse than that of 4×3 RAID 1+0. And for tiny writes, which are all seek, the performance can only be about a 3rd of that of 4×3 RAID 1+0 at absolute best (i.e. given a perfect implementation).

Having cleared that issue up, the 12-drive comparison does not have an outright winner:

                                  | 4×3 RAID 1+0 | 6×2 RAID 1+6
   number of identical 1TB drives | 12           | 12
                 storage capacity | 4TB          | 4TB
            redundancy proportion | 2/3          | 2/3
large contiguous write throughput | 4×           | 4×
 large contiguous read throughput | 12×          | 12×
concurrent tiny writes throughput |*4×           | 1.5×
 concurrent tiny reads throughput | 12×          | 12×
safe number of random drive loses | 2            |*5
    12 - 1 large write throughput | 4×           | 4×
     12 - 1 large read throughput | 8×           |*11×
    12 - 1 tiny writes throughput |*4×           | ~1.42×
     12 - 1 tiny reads throughput | 8×           |*~9.33×
  can split-off a copy for backup | yes[1]       | yes[1]
                  2-site failover | yes          | yes
    2-copy large write throughput | 4×           | 4×
     2-copy large read throughput |*8×           | 6×
    2-copy tiny writes throughput |*4×           | ~1.28×
     2-copy tiny reads throughput |*8×           | 6×
   2-copy safe random drive loses | 1            |*2
2-copy - 1 large write throughput | 4×           | 4×
 2-copy - 1 large read throughput | 4×           |*5× or 6×[2]
2-copy - 1 tiny writes throughput |*4×           | ~1.46× or 1.2×[2]
 2-copy - 1 tiny reads throughput | 4×           |*3.6x or 6×[2]
can be divided into 3 full copies | yes          | yes
                  3-site failover | yes          | yes
    1-copy large write throughput | 4×           | 4×
     1-copy large read throughput | 4×           | 4×
    1-copy tiny writes throughput |*4×           | ~0.85×
     1-copy tiny reads throughput |*4×           | 2×
   1-copy safe random drive loses | 0            | 0
                       complexity |*simple       | more complex

Note 1: A full copy of the stored data is respectively a RAID 0 quadruple or a 4/6 degraded RAID 6 array. Note 2: There's an even chance as to whether the drive failure offlines one of the 4 degraded RAID 1 pairs or degrades one of the 2 normal pairs.

Nevertheless, it would have double the read performance of a RAID 6 array of 6 drives and the tiny writes throughput should be 25% better (1.5/1.2) due to the required reads being divided between the RAID 1 pairs, and RAID 6 obviously does have suitable applications, so in high-availability applications that have larger writes or that are more concerned about read performance than write performance, maybe there is a niche for RAID 1+6 afterall. But that's not all…

Complexity

This is still just in theory so far (mostly combinatorics), in practice the complexity will mean that implementations of RAID 1+6 may have deficiencies that miss opportunities and don't achieve the theoretical outcomes. RAID 6 is already more complex, and nesting adds a bit more complexity atop of this.

For example, it's not immediately obvious that 6×2 RAID 1+6 can be abstracted as having 3 independent virtual read heads capable of simultaneously reading 3 contiguous large reads at 4× throughput each, just like 4×3 RAID 1+0. Simply nesting 6 RAID 1 pairs in a RAID 6 array using a software RAID may not be so elegant; the implementation may be stupid and thrash (I haven't tested this hypothesis yet though).

The complexity also presents an increased cost of development of implementations and tools. Even though there may be applications that could benefit from such a nesting, the improvements may not be worth the development costs.