RAID 1E with ZFS

Question

IBM's storage systems as well as Linux' mdraid do support RAID level 1E to be able to use an odd number of disks for a 50% redundancy:

RAID 1E

Would I achieve the same effect creating a zpool with an odd number of single-disk vdevs while setting copies=2? How does this work from a management perspective - would I still be able to do a zpool replace to replace a disk? How would the pool behave upon a disk failure in such a setup?

@ewwhite nope - it is a [striping technique](http://en.wikipedia.org/wiki/Non-standard_RAID_levels#RAID_1E) where the stripes are written to disks in a round-robin fashion - creating redundancy with 50% of the capacity. — the-wabbit, Jun 16 '13 at 06:45
If anyone has any questions about RAID 1E, read [What are the different widely used RAID levels and when should I consider them?](http://serverfault.com/questions/339128/what-are-the-different-widely-used-raid-levels-and-when-should-i-consider-them) first. — Chris S, Jul 25 '13 at 19:20
Yes! With 27 physical disk, you may build 9 raid 1E using HW-Raid controlers, than you could build 3 Linux MD Raid-1e, useable as 3 physical devices for ZFS, from where you may use `copies=2` to be able to use 1.5x size of 1 disk... data would be redundants! — F. Hauri - Give Up GitHub, Jul 25 '13 at 19:20
@F.Hauri I don't quite understand what you're getting at. From the performance and redundancy characteristics, RAID1E is basically what RAID10 is for an even number of disks, so with 27 physical disks you would create a *single* RAID1E device with 13.5x of the single disk's capacity. If you have an odd number of disks and a requirement for these characteristics, you can either choose to give up one disk and use RAID10 or use RAID1E with all available disks. — the-wabbit, Jul 26 '13 at 08:13
I was wrong: following this schema, 27 -> 9 -> 3 -> =4.5 x 1 disk. It was a joke as this is clearly *overkill* (with this, each block of data would exist on 6 different physical disk): There are redundant/concurrent technologies: hw raid, soft raid and Zfs... — F. Hauri - Give Up GitHub, Jul 27 '13 at 05:04

score 2 · Accepted Answer · answered Jul 25 '13 at 18:49

As far as I know, no.

'zpool copies' creates redundant bits as you know, and as I recall, it is supposed to try to push those redundant bits as far away geographically as possible, but I don't believe it is a hard & fast requirement like it is for mirrored bits in a mirror vdev; space constraints, other moving pieces could I believe lead to a scenario where copy #2 is still on the same disk. If that happens even once, then losing a drive from such a config would be a problem, data retention wise.

Nor is it something that the pool administrative commands are designed to treat in the same way they'd treat mirrored vdevs. Nor is it something the ZFS workflow is designed to treat the same way it does mirrored or parity vdevs -- if you lost a disk out of a pool that had multiple disks as top-level vdevs, even if you had copies=2 or higher set right from day 1, I would expect ZFS to complain mightily, and possibly begin returning errors for data access.

I would not expect copies=2 (or more) to act like RAID 1E. However, if you created multiple partitions on your disks, and set them up in the proper mirrored vdevs, you could probably replicate the idea of RAID 1E with ZFS. If you had 3 drives, each with 2 partitions, and you set up the mirror vdevs so that each pair of partitions are not from the same drive, then the resulting pool could survive a single disk loss. The performance characteristics of such a pool, however, have never been explored to my knowledge (I'd surmise they would be bad).

Your assumptions seem to be backed by other people's findings: http://serverfault.com/questions/377892/zfs-how-do-you-restore-the-correct-number-of-copies-after-losing-a-drive, `copies` are not guaranteed to be on different vdevs and the pool will go to a FAILED state after losing a vdev, no matter the number of `copies`. — the-wabbit, Jul 26 '13 at 07:48
Partitioning the device to mimic RAID1E seems to do the trick and balance I/O evenly if in an array with N disks you create 2 partitions on each disk and create a zpool of mirrored vdevs pairing up D(1)P(1) - D(2)P(1) ... D(N)P(1) - D(1)P(2) ...D(N-1)P(2) - D(N)P(2) . It obviously would break sequential writes, though. — the-wabbit, Jul 26 '13 at 08:15

score 1 · Answer 2 · answered Jul 25 '13 at 19:23

ZFS does not support RAID 1E because RAID-Z does not suffer from a RAID 5 write hole. RAID-Z offers better space utilization, no performance difference, and the same fault tolerance.

Setting copies=2 does not force those copies to be on different physical disks. So it offers no fault protection for physical disk faults. Setting the number to more than 1, the default, makes copies called ditto blocks. These are useful for recovering from URE or power loss data corruption (as there will be another copy of the data, hopefully with a valid checksum). ZFS offers inherent data corruption detection, not protection; this is protection against certain types of corruption.

There *is* a significant difference in performance characteristics for the *small reads* workload. As RAIDZ would read all stripes of a set and recalculate the parity for validation, small reads would keep the array busy quickly. — the-wabbit, Jul 26 '13 at 06:39

RAID 1E with ZFS

2 Answers2