29

I'm planning to setup a set of 3x 2TB 7200rpm drives as a LUKS-encrypted Z-RAID pool in Linux (for a NAS solution).

My understanding of the problem at hand is that the only way to achieve this is to luksFormat every physical device and then assemble a zpool out of unlocked LUKS containers.

I have the following concerns with this:

  • Wouldn't it significantly impede write performance? In this setup redundant data is encrypted several times because LUKS is not "aware" of Z-RAID. In LUKS-on-mdadm solution data is encrypted once and merely written to disks multiple times. My CPU supports Intel AES-NI.

  • Will ZFS be aware of disk failures when operating on device-mapper LUKS containers as opposed to physical devices? How about deduplication and other ZFS features?

MasterM
  • 1,031
  • 2
  • 11
  • 18
  • 4
    I wouldn't do it. Sounds failure-prone. – ewwhite Apr 05 '14 at 23:01
  • Why not leave the devices unencrypted and encrypt the file system instead? – MadHatter Apr 09 '14 at 13:53
  • 3
    @MadHatter Because it's ZFS. You can't do that. – Michael Hampton Apr 09 '14 at 13:55
  • 1
    Fine (I'll take your word for it). Make a single large ZFS containing a single large file, loopback mount it and encrypt *that*. – MadHatter Apr 09 '14 at 13:56
  • A modern Intel or AMD processor with AES-NI instructions won't have problems keeping up. – Brian Apr 09 '14 at 14:11
  • There are "plenty" of such setup for FreeBSD using `geli` ... even official ones (`geli` is FreeBSD `LUKS` counterpart). – Ouki Apr 09 '14 at 14:49
  • Why are you pursuing this? It really sounds like an incompatible combination of technologies. – ewwhite Apr 09 '14 at 15:55
  • 1
    @eewhite I'm merely figuring out options for using encryption with ZFS on Linux until the ZFS kernel module implements encryption itself. But I have to agree - LUKS and ZFS don't seem to be getting along nicely. – MasterM Apr 09 '14 at 17:03
  • 4
    Never, ever, create a large file and loopback it with ZFS. You're gone screw up your pool to an unusable speed when CoW run out of space for it's operations. – Vinícius Ferrão Apr 10 '14 at 05:14
  • So full disc encryption with LUKS is not obtainable? – XTL May 02 '15 at 16:37
  • 1
    I've been running ZFS in FreeBSD on a virtual block device backed by a LUKS device on top of DMRAID for years. Many other weird combinations too. No problems, ever. – ruief Nov 14 '15 at 07:26

2 Answers2

35

One of the servers that I administrate runs the type of configuration that you describe. It has six 1TB hard drives with a LUKS-encrypted RAIDZ pool on it. I also have two 3TB hard drives in a LUKS-encrypted ZFS mirror that are swapped out every week to be taken off-site. The server has been using this configuration for about three years, and I've never had a problem with it.

If you have a need for ZFS with encryption on Linux then I recommend this setup. I'm using ZFS-Fuse, not ZFS on Linux. However, I believe that would have no bearing on the result other than ZFS on Linux will probably have better performance than the setup that I am using.

In this setup redundant data is encrypted several times because LUKS is not "aware" of Z-RAID. In LUKS-on-mdadm solution data is encrypted once and merely written to disks multiple times.

Keep in mind that LUKS isn't aware of RAID. It only knows that it's sitting on top of a block device. If you use mdadm to create a RAID device and then luksformat it, it is mdadm that is replicating the encrypted data to the underlying storage devices, not LUKS.

Question 2.8 of the LUKS FAQ addresses whether encryption should be on top of RAID or the other way around. It provides the following diagram.

Filesystem     <- top
|
Encryption
|
RAID
|
Raw partitions
|
Raw disks      <- bottom

Because ZFS combines the RAID and filesystem functionality, your solution will need to look like the following.

RAID-Z and ZFS Filesystem  <-top
|
Encryption
|
Raw partitions (optional)
|
Raw disks                  <- bottom

I've listed the raw partitions as optional as ZFS expects that it will use raw block storage rather than a partition. While you could create your zpool using partitions, it's not recommended because it'll add a useless level of management, and it will need to be taken into account when calculating what your offset will be for partition block alignment.

Wouldn't it significantly impede write performance? [...] My CPU supports Intel AES-NI.

There shouldn't be a performance problem as long as you choose an encryption method that's supported by your AES-NI driver. If you have cryptsetup 1.6.0 or newer you can run cryptsetup benchmark and see which algorithm will provide the best performance.

This question on recommended options for LUKS may also be of value.

Given that you have hardware encryption support, you are more likely to face performance issues due to partition misalignment.

ZFS on Linux has added the ashift property to the zfs command to allow you to specify the sector size for your hard drives. According to the linked FAQ, ashift=12 would tell it that you are using drives with a 4K block size.

The LUKS FAQ states that a LUKS partition has an alignment of 1 MB. Questions 6.12 and 6.13 discuss this in detail and also provide advice on how to make the LUKS partition header larger. However, I'm not sure it's possible to make it large enough to ensure that your ZFS filesystem will be created on a 4K boundary. I'd be interested in hearing how this works out for you if this is a problem you need to solve. Since you are using 2TB drives, you might not face this problem.

Will ZFS be aware of disk failures when operating on device-mapper LUKS containers as opposed to physical devices?

ZFS will be aware of disk failures insofar as it can read and write to them without problems. ZFS requires block storage and doesn't care or know about the specifics of that storage and where it comes from. It only keeps track of any read, write or checksum errors that it encounters. It's up to you to monitor the health of the underlying storage devices.

The ZFS documentation has a section on troubleshooting which is worth reading. The section on replacing or repairing a damaged device describes what you might encounter during a failure scenario and how you might resolve it. You'd do the same thing here that you would for devices that don't have ZFS. Check the syslog for messages from your SCSI driver, HBA or HD controller, and/or SMART monitoring software and then act accordingly.

How about deduplication and other ZFS features?

All of the ZFS features will work the same regardless of whether the underlying block storage is encrypted or not.

Summary

  1. ZFS on LUKS-encrypted devices works well.
  2. If you have hardware encryption, you won't see a performance hit as long as you use an encryption method that's supported by your hardware. Use cryptsetup benchmark to see what will work best on your hardware.
  3. Think of ZFS as RAID and filesystem combined into a single entity. See the ASCII diagram above for where it fits into the storage stack.
  4. You'll need to unlock each LUKS-encrypted block device that the ZFS filesystem uses.
  5. Monitor the health of the storage hardware the same way you do now.
  6. Be mindful of the filesystem's block alignment if you are using drives with 4K blocks. You may need to experiment with luksformat options or other settings to get the alignment you need for acceptable speed.

February 2020 Update

It's been six years since I wrote this answer. ZFS on Linux v0.8.0 supports native encryption, which you should consider if you don't have a specific need for LUKS.

Starfish
  • 2,716
  • 24
  • 28
  • 3
    +1 For finding a way to make this work *with* examples. – ewwhite Apr 10 '14 at 14:22
  • Personally I will never use ZFS this way. Ok, it can works, but do you guarantee that this setup will not end with problems? I'm not saying that it will not work, but many ZFS experts doesn't recommend layering things on bottom of ZFS. I will leave this link here: http://forums.freenas.org/index.php?threads/slideshow-explaining-vdev-zpool-zil-and-l2arc-for-noobs.7775 There's a lengthy explanation of basic rules of ZFS, the thread is from guys that are in touch with ZFS devs, so it's a worth reading documentation. – Vinícius Ferrão Apr 10 '14 at 14:28
  • 2
    1 MiB is already evenly divisible by 4KiB, so you should be correctly aligned all the way up to ashift=20 (which I don't think will be necessary within my career) provided you used raw disks. – Michael Hampton Apr 10 '14 at 14:36
  • 1
    Just one more thing: I'm voting up your answer because it's what OP expected, and is well written, so it's shure better than mine answer. – Vinícius Ferrão Apr 10 '14 at 14:44
  • @ViníciusFerrão: Yes, I can guarantee it. I've been running this config on a production server for three years without problems. We recommend using the raw disk device because new ZFS users usually think of ZFS as only a filesystem and forget that it does volume management. It's a new concept for them. New users want to put it on LVM or create partitions for the filesystem. Also, ZFS is usually smart enough to figure out the block size for the disk if you use a raw device. Therefore, using raw disk devices is almost always the best course of action. Those who deviate have a reason to do so. – Starfish Apr 10 '14 at 18:49
  • 3
    @ViníciusFerrão: Also take note that FreeBSD and FreeNAS are using an identical approach for ZFS encryption. `geli` is used to create an encrypted device and the plaintext data is made available via a second device which ZFS uses. See the second bullet point at http://doc.freenas.org/index.php/Volumes#Encryption. – Starfish Apr 10 '14 at 18:55
  • I had a dillema to whom grant the bounty at first but I like Starfish's idea and the link to FreeNAS documentation ultimately convinced me that layering ZFS on top of encryptied volumes is not so bad. So @Starfish it is! :) – MasterM Apr 15 '14 at 10:30
  • How do you deal with l2arc and ZIL devices? Do these need to be encrypted with LUKS as well? – CMCDragonkai Nov 11 '15 at 09:21
  • 3
    @CMCDragonkai Since both L2ARC and SLOG devices contain bits and pieces of data from your pool, if you are encrypting the storage in order to provide confidentiality (which is often the point of using encrypted storage in the first place), you almost certainly want to run any L2ARC and SLOG devices similarly encrypted as well. – user Jan 20 '16 at 10:03
  • For the `data - ZFS RAID1 - LUKS - disks` setup, do we need separate keys for each drive? So the system does need to do encryption twice? Can we configure the system to encrypt for one time only if we use the same key across the drives? Will `data - ZFS - LUKS - mdadm R1 - disks` be possible? When ZFS detects a read error, can ZFS make use of the mdadm mirror for recovery? I need LUKS with detached headers for deniable encryption. Thanks very much for a very informative post, especially you updated in 2020 for ZFS added encryption feature. This is very appreciated. – midnite Dec 06 '21 at 11:15
4

An alternative implementation is to create a ZVOL block device (http://zfsonlinux.org/example-zvol.html), use LUKS to encrypt the newly created ZVOL and then create an ext4 (or other) filesystem on top of the encrypted block device.

Tux Racer
  • 41
  • 1