8

I have installed Ubuntu 18.04 on a Dell PowerEdge R720xd using ZFS. There are two 1TB boot drives in a ZFS mirror configuration. I followed the instructions in the ZFS on Linux Wiki.

(Note: My system is using an LSI LSI00244 (9201-16i) Host-bus Adapter (HBA) instead of the onboard RAID card, since ZFS and this RAID card don't get along.)

When booting Ubuntu, it takes about ten seconds for the drives to be enumerated by the system (there are 14 drives - two for the OS and 12 for data storage that will be set up in other zpools later). However, the boot process tries to import the boot pool before the drives have been enumerated.

A BusyBox error message flashes past on the screen, and it basically says:

The pool failed to import.

Manually import the pool in this BusyBox shell and then type exit to continue the boot process.

If I wait a few seconds after that message, I see that the 14 drives get listed.

I type zpool import rpool at the BusyBox prompt, which works (confirmed with zpool list) and then exit to continue the boot process. (This then leads to my next problem, a Kernel crash, but that's a separate question.)

I tried adding rootdelay=15 to the boot options, but that doesn't seem to work since it seems to want to run that delay after the ZFS pool import.

How can I get the boot process to wait for the devices to show up before it tries to import the pool?

Moshe Katz
  • 3,053
  • 3
  • 26
  • 41
  • Does Ubuntu support booting from ZFS mirror? What you are trying to accomplish is trivial with FreeBSD. All my FreeBSD file servers have OS installed on the ZFS mirrors and that is done from the vanilla FreeBSD installer. I know that Alpine Linux supports booting from ZFS. FreeBSD also has beadm https://www.freebsd.org/cgi/man.cgi?query=beadm – Predrag Punosevac Jun 05 '18 at 01:26
  • 1
    @PredragPunosevac Yes, Ubuntu does support it, according to multiple sources, including conversations with Ubuntu developers. What I am trying to accomplish is *supposed* to be trivial with Ubuntu too. – Moshe Katz Jun 05 '18 at 01:27

2 Answers2

12

I finally found this in /etc/default/zfs:

# Wait for this many seconds in the initrd mountroot?
# This delays startup and should be '0' on most systems. This might help on
# systems which have their ZFS root on a USB disk that takes just a little
# longer to be available
# Only applicable for Debian GNU/Linux {dkms,initramfs}.
ZFS_INITRD_POST_MODPROBE_SLEEP='0'

Here's how to set it.

  1. Boot from a Ubuntu Live CD (or any other recovery media)
  2. Import the pool to /mnt using zpool import rpool -R /mnt
  3. Bind the necessary filesyststems mount --rbind /dev /mnt/dev; mount --rbind /proc /mnt/proc; mount --rbind /sys /mnt/sys
  4. Chroot into /mnt: chroot /mnt /bin/bash --login
  5. Edit /etc/default/zfs to change the value above from 0 to 15
  6. Run update-initramfs and update-grub
  7. Exit the Chroot environment and reboot
Moshe Katz
  • 3,053
  • 3
  • 26
  • 41
  • Good find... But ZFS root is bad news still. – ewwhite Jun 05 '18 at 02:19
  • 2
    @ewwhite Why? I spoke to several Ubuntu developers who are now recommending it. (Some of them say "either ZFS or BTRFS", but all of them say "if your choice is `mdadm` or ZFS, absolutely choose ZFS"). – Moshe Katz Jun 05 '18 at 02:21
  • Old answer, but relevant. This continues to happen on Ubuntu 20.04 Focal instances that were installed via debootstrap -- not sure if standard installations handle this differently. – nod Apr 30 '20 at 22:21
  • @nod the reason it is still true in 20.04 is because this setting is entirely dependent on your hardware and the operating system has no way to automatically know the correct value. – Moshe Katz Apr 30 '20 at 22:41
  • True, but Xenial's zfs implementation inserts `ZFS_AUTOIMPORT_TIMEOUT='30'` by default into /etc/default/zfs -- Focal does not. – nod May 08 '20 at 16:53
  • 1
    Fyi... as of Ubuntu 22.04, the `ZFS_INITRD_POST_MODPROBE_SLEEP` setting is no longer present in `/etc/default/zfs`. However, when I manually add it, it still seems to work. – mpb May 20 '22 at 01:21
  • @MosheKatz asked: "Why is ZFS-on-root bad news?" One reason is that Linux + ZFS-on-root + snapshots (+ bind mounts? and/or + containers?) seems to cause problems in [2012](https://github.com/openzfs/zfs/issues/816), [2016](https://github.com/openzfs/zfs/issues/4514), [2019](https://github.com/openzfs/zfs/issues/9461), [2019](https://github.com/openzfs/zfs/issues/9479), [2020](https://github.com/openzfs/zfs/issues/9958), and [2020](https://github.com/openzfs/zfs/issues/10348). This problem still exists as of Ubuntu 22.04. – mpb May 27 '22 at 18:19
0

Update May 27, 2022:

I installed and ran Ubuntu 22.04 with ZFS-on-root (and an ext4 /boot partition).

However, based on problems I encountered, I recommend against using ZFS-on-root with Linux.

Specifically, it seems that: Linux + ZFS-on-root + snapshots (possibly also in combination with bind mounts and/or containers) can result in problems. More specifically: problems with snapshots.

It is possibly that these problems would be mitigated or avoided by creating one ZFS pool for the root filesystem, and a separate ZFS pool for everything else. But I have not tried this.

If you do want to try to run ZFS-on-root on Linux, I recommend first searching for both open and closed(!) issues at GitHub to familiarize yourself with the problems you may encounter.

The problem(s) I encountered seem to be mentioned in the following GitHub issues reports: 816, 4514, 9461, 9479, 9958, 10348, and possibly more.

Due to license conflicts, OpenZFS will (most probably) never be merged with the Linux kernel. Consequently, I suspect OpenZFS will never receive as much testing as a root-filesystem on Linux as will the GPL-licensed filesystems that are built directly into the Linux kernel (e.g. ext4, Btrfs, XFS).

Original answer from May 20, 2022:

As of Ubuntu 22.04, adding ZFS_INITRD_POST_MODPROBE_SLEEP='15' to /etc/default/zfs still solves the problem. (Aside: in my case, I only needed 1 second of delay.)

However, I believe it is worth pointing out that both ...

  • ZFS_INITRD_POST_MODPROBE_SLEEP
  • and ZFS_INITRD_PRE_MOUNTROOT_SLEEP

... are now deprecated and have been removed from /etc/default/zfs (note1, note2, pull request).

The rationale for the deprecation is described here.

In a nutshell, it seems that there is now a kernel command-line option of rootdelay=n that should be used instead.

If you want to see the actual OpenZFS initramfs script that uses these variables, it is here:
https://github.com/openzfs/zfs/blob/master/contrib/initramfs/scripts/zfs

mpb
  • 223
  • 2
  • 8