FreeNAS confusion: not sure what happened. raidz1 "DEGRADED", disk "FAULTED"

Question

Sorry for the long narrative, but I'm thoroughly confused.

I'm using FreeNAS-8.0.4-RELEASE-p2-x64 (11367) on a box with 5x3TB SATA disks configured as a raidz volume.

A few days ago, the console gave me this alert:

CRITICAL: The volume raid-5x3 (ZFS) status is DEGRADED

zpool status gave:

  pool: raid-5x3
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
  see: http://www.sun.com/msg/ZFS-8000-8A
 scrub: resilver completed after 3h25m with 7607009 errors on Sun Aug 12 06:26:44 2012
config:

    NAME                      STATE    READ WRITE CKSUM
    raid-5x3                  DEGRADED    0    0 7.29M
      raidz1                  DEGRADED    0    0 14.7M
        ada0p2                ONLINE      0    0    0
        10739480653363274060  FAULTED      0    0    0  was /dev/ada1p2
        ada2p2                ONLINE      0    0    0
        ada3p2                ONLINE      0    0    3  254M resilvered
        ada1p2                ONLINE      0    0    0

errors: 7607009 data errors, use '-v' for a list

I did a zpool status -v and got:

Permanent errors have been detected in the following files:

and it listed 2,660 files (out of 50,000 or so)

plus things like:

raid-5x3/alpha:<0x0>
raid-5x3/alpha:<0xf5ec>
raid-5x3/alpha:<0xf5ea>

We turned the server off, put in a new drive, in addition to the five already in there.

Went to console and view disks, it just said "loading" forever *couldn't get to the "Replace" option!

Then we got:

zpool status -v
  pool: raid-5x3
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
  see: http://www.sun.com/msg/ZFS-8000-8A
 scrub: none requested
config:

        NAME                      STATE    READ WRITE CKSUM
        raid-5x3                  DEGRADED    0    0    0
          raidz1                  DEGRADED    0    0    0
            ada0p2                ONLINE      0    0    0
            10739480653363274060  UNAVAIL      0    0    0  was /dev/ada1p2
            ada2p2                ONLINE      0    0    0
            ada3p2                ONLINE      0    0    0
            ada1p2                ONLINE      0    0    0

errors: Permanent errors have been detected in the following files:

        raid-5x3/alpha:<0x0>
        /mnt/raid-5x3/alpha/staff/Sound FX jw/Sound FX - scary horror/11 DR-EerieAct3-Waterphone..aif
        /mnt/raid-5x3/alpha/staff/Wheelhouse Shoots/ROCKY_THE_MUSICAL/ SHOOTS/WESTPORT/Cannon-CARD-B/CONTENTS/CLIPS001/AA0876/AA087601.SIF

... then 2,860 files and "raid-5x3/alpha:<....>" entries ...

camcontrol devlist:

<ST3000DM001-9YN166 CC4C>          at scbus4 target 0 lun 0 (ada0,pass0)
<WDC WD30EZRX-00MMMB0 80.00A80>    at scbus4 target 1 lun 0 (aprobe1,pass6,ada4)
<WDC WD30EZRX-00MMMB0 80.00A80>    at scbus5 target 0 lun 0 (ada1,pass1)
<ST3000DM001-9YN166 CC4C>          at scbus5 target 1 lun 0 (ada2,pass2)
<ASUS DRW-24B1ST  a 1.04>        at scbus6 target 0 lun 0 (cd0,pass3)
<Hitachi HDS5C3030ALA630 MEAOA580>  at scbus7 target 0 lun 0 (ada3,pass4)
< USB Flash Memory 1.00>          at scbus8 target 0 lun 0 (da0,pass5)

gpart show

=>    63  7831467  da0  MBR  (3.7G)
      63  1930257    1  freebsd  [active]  (943M)
  1930320      63      - free -  (32K)
  1930383  1930257    2  freebsd  (943M)
  3860640    3024    3  freebsd  (1.5M)
  3863664    41328    4  freebsd  (20M)
  3904992  3926538      - free -  (1.9G)

=>      0  1930257  da0s1  BSD  (943M)
        0      16        - free -  (8.0K)
      16  1930241      1  !0  (943M)

=>        34  5860533101  ada0  GPT  (2.7T)
          34          94        - free -  (47K)
        128    4194304    1  freebsd-swap  (2.0G)
    4194432  5856338703    2  freebsd-zfs  (2.7T)

=>        34  5860533101  ada1  GPT  (2.7T)
          34          94        - free -  (47K)
        128    4194304    1  freebsd-swap  (2.0G)
    4194432  5856338703    2  freebsd-zfs  (2.7T)

=>        34  5860533101  ada2  GPT  (2.7T)
          34          94        - free -  (47K)
        128    4194304    1  freebsd-swap  (2.0G)
    4194432  5856338703    2  freebsd-zfs  (2.7T)

=>        34  5860533101  ada3  GPT  (2.7T)
          34          94        - free -  (47K)
        128    4194304    1  freebsd-swap  (2.0G)
    4194432  5856338703    2  freebsd-zfs  (2.7T)

=>        34  5860533101  ada4  GPT  (2.7T)
          34          94        - free -  (47K)
        128    4194304    1  freebsd-swap  (2.0G)
    4194432  5856338703    2  freebsd-zfs  (2.7T)

glabel status

                                      Name  Status  Components
                            ufs/FreeNASs3    N/A  da0s3
                            ufs/FreeNASs4    N/A  da0s4
                            ufs/FreeNASs1a    N/A  da0s1a
gptid/446dd91d-8f15-11e1-a14c-f46d049aaeca    N/A  ada4p1
gptid/447999cb-8f15-11e1-a14c-f46d049aaeca    N/A  ada4p2

Seemed the new drive wasn't connected properly?

Re-attached it and rebooted.

Now console showed green light alert.

But when I went to "View All Volumes", it just said "Loading..."

Then:

glabel status

          Name  Status  Components
 ufs/FreeNASs3    N/A  da0s3
 ufs/FreeNASs4    N/A  da0s4
ufs/FreeNASs1a    N/A  da0s1a

camcontrol devlist: Code: at scbus0 target 0 lun 0 (ada0,pass0) at scbus4 target 0 lun 0 (ada1,pass1) at scbus4 target 1 lun 0 (ada2,pass2) at scbus5 target 0 lun 0 (ada3,pass3) at scbus5 target 1 lun 0 (ada4,pass4) at scbus6 target 0 lun 0 (cd0,pass5) at scbus7 target 0 lun 0 (ada5,pass6) < USB Flash Memory 1.00> at scbus8 target 0 lun 0 (da0,pass7)

gpart show

=>    63  7831467  da0  MBR  (3.7G)
      63  1930257    1  freebsd  [active]  (943M)
  1930320      63      - free -  (32K)
  1930383  1930257    2  freebsd  (943M)
  3860640    3024    3  freebsd  (1.5M)
  3863664    41328    4  freebsd  (20M)
  3904992  3926538      - free -  (1.9G)

=>      0  1930257  da0s1  BSD  (943M)
        0      16        - free -  (8.0K)
      16  1930241      1  !0  (943M)

=>        34  5860533101  ada1  GPT  (2.7T)
          34          94        - free -  (47K)
        128    4194304    1  freebsd-swap  (2.0G)
    4194432  5856338703    2  freebsd-zfs  (2.7T)

=>        34  5860533101  ada2  GPT  (2.7T)
          34          94        - free -  (47K)
        128    4194304    1  freebsd-swap  (2.0G)
    4194432  5856338703    2  freebsd-zfs  (2.7T)

=>        34  5860533101  ada3  GPT  (2.7T)
          34          94        - free -  (47K)
        128    4194304    1  freebsd-swap  (2.0G)
    4194432  5856338703    2  freebsd-zfs  (2.7T)

=>        34  5860533101  ada4  GPT  (2.7T)
          34          94        - free -  (47K)
        128    4194304    1  freebsd-swap  (2.0G)
    4194432  5856338703    2  freebsd-zfs  (2.7T)

=>        34  5860533101  ada5  GPT  (2.7T)
          34          94        - free -  (47K)
        128    4194304    1  freebsd-swap  (2.0G)
    4194432  5856338703    2  freebsd-zfs  (2.7T)

zpool status:

  pool: raid-5x3
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
  see: http://www.sun.com/msg/ZFS-8000-8A
 scrub: none requested
config:

        NAME        STATE    READ WRITE CKSUM
        raid-5x3    ONLINE      0    0    0
          raidz1    ONLINE      0    0    0
            ada1p2  ONLINE      0    0    0
            ada2p2  ONLINE      0    0    2
            ada4p2  ONLINE      0    0    0
            ada5p2  ONLINE      0    0    0
            ada3p2  ONLINE      0    0    0

errors: 7607009 data errors, use '-v' for a list

At this point, someone on the FreeNAS forums said, "You're screwed, at some point you had 2 disks fail, bye bye data."

Is this true?

I clicked the 'scrub' button ... zpool status showed "resilver in progress .... 900h to go" ... which is like a month... and which kept going up to 30,000hrs...

Cut to: today, we rechecked all the connections on all the drives.

Then it started resilvering again, but much faster.

Several of the files – which were previously reported as corrupt – I randomly checked, and they now "seem" to be OK. (Meaning I was able to copy them and play them – most of our data is video files.)

What I'd like to do is COPY everything for which we do not have a backup, and which is not corrupt, to another machine, and then upgrade this one to RAIDZ2.

I'm thinking maybe what happened is that 2 drives became dislodged. I think the hotswap bay we have is poor quality.

But, then again, they DID appear connected, just faulted ... I don't know.

The resilver completed, in 3.5 hours.

Now zpool status says:

  pool: raid-5x3
 state: ONLINE
 scrub: resilver completed after 3h31m with 0 errors on Fri Aug 17 21:46:12 2012
config:

        NAME        STATE    READ WRITE CKSUM
        raid-5x3    ONLINE      0    0    0
          raidz1    ONLINE      0    0    0
            ada1p2  ONLINE      0    0    0
            ada2p2  ONLINE      0    0    0  236G resilvered
            ada4p2  ONLINE      0    0    0
            ada5p2  ONLINE      0    0    0  252G resilvered
            ada3p2  ONLINE      0    0    0

errors: No known data errors

Does this mean the data is recovered?? "No known errors" sounds promising!

I've now initiated a scrub. (8 hours to go.)

We don't have a backup for ALL the data ... so we need to figure out which of those files are corrupt, and which are usable.

Did a drive fail? If so, which one? Or did it just come loose?

Do I need to replace one? Two?

Is any of our data safe? If so, which files?

How do I check that? And should I wait until scrubbing is done? — Dan, Aug 18 '12 at 08:40

score 2 · Answer 1 · answered May 16 '14 at 04:28

This same thing happened to me when I accidentally removed the incorrect drive when replacing a failed one. When I checked the array it said the entire vdev had failed with two disks faulted. Once I reconnected the non-failed disk and replaced the actual faulted disk the pool fully resilvered itself with no noticeable impact on the files that were initially listed as damaged/lost/corrupted.

As I understand it, when the disk was jostled loose momentarily it was marked as failed, then since one disk had actually failed, the whole array was also marked as failed. Then as soon as it was reconnected, it's scanned and shows up as part of the existing pool, and resilvered with the rest. Since 3/4 disks is enough to have viable data, as long as the data hasn't changed too much, everything can recover once the failed disk is replaced.

Checksum errors aren't hard errors -- they're recoverable errors. When a checksum error happens, ZFS corrects the error immediately in RAIDZ, this is one of the major reasons that ZFS is such a great file system. That said, seeing checksum errors is an indication of a problem. Keep a close eye on the disks, try to have a spare handy in case one fails. If you start seeing checksum errors again, replace whatever drive/cable they are happening on. It may be the backplane, it may be the connector, it may be whatever, check everything. If you're worried about the whole bay, replace it.

No known errors only refers to the current status of the pool, not the complete history of it. Since the disk was replaced and the array resilvered, the stats look to have been cleared as well; the checksum errors are zeroed out. There were two errors previously, and those would remain until cleared.

ZFS reporting can be trusted, so any of the files that weren't listed in the damaged files list should be fine. As mentioned in another answer, if you've got known good copies of any of the files, doing some spot check hashes can't hurt. Getting full backups going wouldn't be a bad idea either. ;)

score 0 · Accepted Answer · answered Aug 19 '12 at 04:54

I haven't used zfs enough to say on data integrity. zpool list gave raidz1-0 ONLINE 0 0 0 c8t1d0p0 ONLINE 0 0 0 c8t2d0p0 ONLINE 0 0 0 c8t3d0p0 ONLINE 0 0 0

smartctl -d sat,12 -a /dev/rdsk/c8t3d0p0 listed specific attributes. -d sat -d ata -d scsi may get it to work depending on what type of drives you have. look at raw read error rate, relocated sector count, seek error rate, hardware ecc, and pending sector relocation. -t short or -t long instead of -a will start a test. the long test can take one or many hours depending on the drive. the read seak and hardware errors are not neccessarly a failure. how ever if they keep jumping between listings they can be a good indicator. Also some drive manufacturers may require you use their tools to get a drive rma'd.

If I were in your position and unsure about the drive I would get a replacement and after either verifying its good through working with the manufacturer or having it rma. I would put in the extra drive as a hot spare so that if it happens again you have another drive ready and waiting. z2 is not a bad idea either but I would still keep a spare in the wings. with the size of drives now they do take a long time to rebuild and it is possible to hit a 2nd error.

as to corrupted files if you still have an original some where else of one of those "corrupted" files do a md5 calculation of it and then see if the corrupted file's md5 matches. that will tell you if bits got damaged. if not then there may be invisible damage to the files. whether its enough damage to be a problem will be up to you.

FreeNAS confusion: not sure what happened. raidz1 "DEGRADED", disk "FAULTED"

2 Answers2

Linked