ZFS pool degraded on reboot

Question

I have a Ubuntu server set up with a 14-disk ZFS raidz2 pool.

About 80% of the time, on reboot, I will end up with a degraded pool with two of the disks marked as faulted. The drives that are faulted aren't always the same, but it is always exactly two drives. For example:

$ sudo zpool status
  pool: tank
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
  scan: resilvered 4K in 0h0m with 0 errors on Sun Sep 30 23:08:51 2018
config:

        NAME                      STATE     READ WRITE CKSUM
        tank                      DEGRADED     0     0     0
          raidz2-0                DEGRADED     0     0     0
            sde                   ONLINE       0     0     0
            sdc                   ONLINE       0     0     0
            sdd                   ONLINE       0     0     0
            sda                   ONLINE       0     0     0
            sdh                   ONLINE       0     0     0
            11521322863231878081  FAULTED      0     0     0  was /dev/sdf1
            15273938560620494453  FAULTED      0     0     0  was /dev/sdg1
            sdb                   ONLINE       0     0     0
            sdi                   ONLINE       0     0     0
            sdj                   ONLINE       0     0     0
            sdk                   ONLINE       0     0     0
            sdl                   ONLINE       0     0     0
            sdm                   ONLINE       0     0     0
            sdn                   ONLINE       0     0     0

errors: No known data errors

I can export and re-import the pool, and the disks are no longer faulted. For example:

$ sudo zpool export tank
$ sudo zpool import tank
$ sudo zpool status
  pool: tank
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: resilvered 4K in 0h0m with 0 errors on Sun Sep 30 23:08:51 2018
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            sde     ONLINE       0     0     0
            sdc     ONLINE       0     0     0
            sdd     ONLINE       0     0     0
            sda     ONLINE       0     0     0
            sdh     ONLINE       0     0     0
            sdg     ONLINE       0     0     1
            sdf     ONLINE       0     0     0
            sdb     ONLINE       0     0     0
            sdi     ONLINE       0     0     0
            sdj     ONLINE       0     0     0
            sdk     ONLINE       0     0     0
            sdl     ONLINE       0     0     0
            sdm     ONLINE       0     0     0
            sdn     ONLINE       0     0     0

errors: No known data errors

The HBA being used has worked correctly in another server.

Anything else I can try to avoid these faulted drives on reboot? I have another HBA I could swap in.

score 8 · Accepted Answer · answered Oct 11 '18 at 23:14

8

You should not use /dev/sdX names for your pool configuration.

Any change in SCSI enumeration, say inserting a CDROM or a USB drive, could cause the device names to change, resulting in the errors you're experiencing.

You have an option of using the /dev/disk/by-id names.

Do this with a zpool export tank and zpool import -d /dev/disk/by-id tank

answered Oct 11 '18 at 23:14

ewwhite

194,921
91
434
799

Thanks, this works. The zpool now seems to be using /dev/disk/by-id/wwn-* device names and imports successfully on reboot. I was expecting device names like ata-* which use the device model and serial number, rather than wwn-* which is the LU WWN Device ID reported by hdparm and smartctl. No worries though, as long as it works. For anybody finding this later, there is a zfs issue that uses this same workaround. https://unix.stackexchange.com/questions/288599/forcing-zpool-to-use-dev-disk-by-id-in-ubuntu-xenial – David Chappelle Oct 12 '18 at 02:32
1

Wow @ewwhite ... thanks for saving me literally hours of fixes. This works flawlessly. – oemb1905 Feb 25 '22 at 07:23
1

@oemb1905 I'm glad I could help – ewwhite Feb 25 '22 at 08:37

ZFS pool degraded on reboot

1 Answers1

Linked