Can't get my RAID array out of degraded mode

Question

I've got a 4-drive RAID 10 array that has just had a drive failure. I ignorantly have never practiced how to recover from a failure (I'm a programmer, just keeping this server as a hobbyist) so I'm having to learn this all the hard way right now.

I managed, through Google and this site (thank you guys!) to figure out how to fail, remove, add and resync the new drive, but it keeps failing during the resync and simply marking the new disk as a spare.

With more Googling and some more command-line-fu, I've figured out that the remaining "good" drive actually has some bad sectors producing read errors during the sync, so mdadm is aborting and marking as spare.

I've used badblocks to confirm the presence of bad sectors (there appear to be quite a few), but I have no idea if those sectors are actually in use or not (so far, I haven't noticed any corrupt data). I've also read that fsck can potentially repair this data, but I've also read that it has the real possibility of completely fubaring the drive, too. As such, I haven't tried it, yet.

I've tried using mdadm's --force flag to ignore these errors during re-sync, but it doesn't appear to help at all.

I have all my critical data backed up already, but there is a large amount of non-critical data that I'd really rather not lose if it can be avoided (it's all replaceable, but would take a very long time). Additionally, my critical backups are all in the cloud, so even restoring those, while easy, would be very time-consuming.

Also, if needed, I have one more unused, new replacement drive on hand.

Below is all the info about the system that I know to give. Please let me know if you need more! How do I get this array fully rebuilt?

Drive Layout

sda + sdb = RAID1A (md126)

sdc + sdd = RAID1B (md127)

md126 + md127 = RAID10 (md125)

The problem array is md126, the new unsynced drive is sdb, and the problem drive is sda

`root@vault:~# cat /proc/mdstat`

Personalities : [raid1] [raid0] [linear] [multipath] [raid6] [raid5] [raid4] [raid10]
md125 : active raid0 md126p1[1] md127p1[0]
      5860528128 blocks super 1.2 512k chunks

md126 : active raid1 sda1[1] sdb1[2](S)
      2930265390 blocks super 1.2 [2/1] [U_]

md127 : active raid1 sdc1[1] sdd1[0]
      2930265390 blocks super 1.2 [2/2] [UU]

unused devices: <none>

`root@vault:~# parted -l`

Model: ATA ST3000DM001-9YN1 (scsi)
Disk /dev/sda: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name          Flags
 1      17.4kB  3001GB  3001GB               RAID: RAID1A  raid


Model: ATA ST3000DM001-9YN1 (scsi)
Disk /dev/sdb: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name          Flags
 1      17.4kB  3001GB  3001GB               RAID: RAID1A  raid


Model: ATA ST3000DM001-1CH1 (scsi)
Disk /dev/sdc: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name          Flags
 1      17.4kB  3001GB  3001GB               RAID: RAID1B  raid


Model: ATA ST3000DM001-9YN1 (scsi)
Disk /dev/sdd: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name          Flags
 1      17.4kB  3001GB  3001GB               RAID: RAID1B  raid

`root@vault:~# sudo mdadm --detail /dev/md126`

/dev/md126:
        Version : 1.2
  Creation Time : Thu Nov 29 19:09:32 2012
     Raid Level : raid1
     Array Size : 2930265390 (2794.52 GiB 3000.59 GB)
  Used Dev Size : 2930265390 (2794.52 GiB 3000.59 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Thu Jun  2 11:53:44 2016
          State : clean, degraded
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1

           Name : :RAID1A
           UUID : 49293460:3199d164:65a039d6:a212a25e
         Events : 5200173

    Number   Major   Minor   RaidDevice State
       1       8        1        0      active sync   /dev/sda1
       2       0        0        2      removed

       2       8       17        -      spare   /dev/sdb1

Edit: Here are the contents of the kernel log during the failed recovery process.

root@vault:~# mdadm --assemble --update=resync --force /dev/md126 /dev/sda1 /dev/sdb1

`root@vault:~# tail -f /var/log/kern.log`

Jun  5 12:37:57 vault kernel: [151562.172914] RAID1 conf printout:
Jun  5 12:37:57 vault kernel: [151562.172917]  --- wd:1 rd:2
Jun  5 12:37:57 vault kernel: [151562.172919]  disk 0, wo:0, o:1, dev:sda1
Jun  5 12:37:57 vault kernel: [151562.172921]  disk 1, wo:1, o:1, dev:sdb1
Jun  5 12:37:57 vault kernel: [151562.173858] md: recovery of RAID array md126
Jun  5 12:37:57 vault kernel: [151562.173861] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
Jun  5 12:37:57 vault kernel: [151562.173863] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
Jun  5 12:37:57 vault kernel: [151562.173865] md: using 128k window, over a total of 2930265390k.
Jun  5 12:37:57 vault kernel: [151562.248457]  md126: p1
Jun  5 12:37:58 vault kernel: [151562.376906] md: bind<md126p1>
Jun  5 13:21:52 vault kernel: [154196.675777] ata3.00: exception Emask 0x0 SAct 0xffe00 SErr 0x0 action 0x0
Jun  5 13:21:52 vault kernel: [154196.675782] ata3.00: irq_stat 0x40000008
Jun  5 13:21:52 vault kernel: [154196.675785] ata3.00: failed command: READ FPDMA QUEUED
Jun  5 13:21:52 vault kernel: [154196.675791] ata3.00: cmd 60/00:48:a2:a4:e0/05:00:38:00:00/40 tag 9 ncq 655360 in
Jun  5 13:21:52 vault kernel: [154196.675791]          res 41/40:00:90:a7:e0/00:05:38:00:00/00 Emask 0x409 (media error) <F>
Jun  5 13:21:52 vault kernel: [154196.675794] ata3.00: status: { DRDY ERR }
Jun  5 13:21:52 vault kernel: [154196.675797] ata3.00: error: { UNC }
Jun  5 13:21:52 vault kernel: [154196.695048] ata3.00: configured for UDMA/133
Jun  5 13:21:52 vault kernel: [154196.695077] sd 2:0:0:0: [sda] tag#9 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jun  5 13:21:52 vault kernel: [154196.695081] sd 2:0:0:0: [sda] tag#9 Sense Key : Medium Error [current] [descriptor]
Jun  5 13:21:52 vault kernel: [154196.695085] sd 2:0:0:0: [sda] tag#9 Add. Sense: Unrecovered read error - auto reallocate failed
Jun  5 13:21:52 vault kernel: [154196.695090] sd 2:0:0:0: [sda] tag#9 CDB: Read(16) 88 00 00 00 00 00 38 e0 a4 a2 00 00 05 00 00 00
Jun  5 13:21:52 vault kernel: [154196.695092] blk_update_request: I/O error, dev sda, sector 954247056
Jun  5 13:21:52 vault kernel: [154196.695111] ata3: EH complete
Jun  5 13:21:55 vault kernel: [154199.675248] ata3.00: exception Emask 0x0 SAct 0x1000000 SErr 0x0 action 0x0
Jun  5 13:21:55 vault kernel: [154199.675252] ata3.00: irq_stat 0x40000008
Jun  5 13:21:55 vault kernel: [154199.675255] ata3.00: failed command: READ FPDMA QUEUED
Jun  5 13:21:55 vault kernel: [154199.675261] ata3.00: cmd 60/08:c0:8a:a7:e0/00:00:38:00:00/40 tag 24 ncq 4096 in
Jun  5 13:21:55 vault kernel: [154199.675261]          res 41/40:08:90:a7:e0/00:00:38:00:00/00 Emask 0x409 (media error) <F>
Jun  5 13:21:55 vault kernel: [154199.675264] ata3.00: status: { DRDY ERR }
Jun  5 13:21:55 vault kernel: [154199.675266] ata3.00: error: { UNC }
Jun  5 13:21:55 vault kernel: [154199.676454] ata3.00: configured for UDMA/133
Jun  5 13:21:55 vault kernel: [154199.676463] sd 2:0:0:0: [sda] tag#24 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jun  5 13:21:55 vault kernel: [154199.676467] sd 2:0:0:0: [sda] tag#24 Sense Key : Medium Error [current] [descriptor]
Jun  5 13:21:55 vault kernel: [154199.676471] sd 2:0:0:0: [sda] tag#24 Add. Sense: Unrecovered read error - auto reallocate failed
Jun  5 13:21:55 vault kernel: [154199.676474] sd 2:0:0:0: [sda] tag#24 CDB: Read(16) 88 00 00 00 00 00 38 e0 a7 8a 00 00 00 08 00 00
Jun  5 13:21:55 vault kernel: [154199.676477] blk_update_request: I/O error, dev sda, sector 954247056
Jun  5 13:21:55 vault kernel: [154199.676485] md/raid1:md126: sda: unrecoverable I/O read error for block 954244864
Jun  5 13:21:55 vault kernel: [154199.676488] ata3: EH complete
Jun  5 13:21:55 vault kernel: [154199.676597] md: md126: recovery interrupted.
Jun  5 13:21:55 vault kernel: [154199.855992] RAID1 conf printout:
Jun  5 13:21:55 vault kernel: [154199.855995]  --- wd:1 rd:2
Jun  5 13:21:55 vault kernel: [154199.855998]  disk 0, wo:0, o:1, dev:sda1
Jun  5 13:21:55 vault kernel: [154199.856000]  disk 1, wo:1, o:1, dev:sdb1
Jun  5 13:21:55 vault kernel: [154199.872013] RAID1 conf printout:
Jun  5 13:21:55 vault kernel: [154199.872016]  --- wd:1 rd:2
Jun  5 13:21:55 vault kernel: [154199.872018]  disk 0, wo:0, o:1, dev:sda1

If bad block from a good drive prevent the rebuild, then I would not thrust your array, as even if it sync back, you might have corruption and the worst is that the controller didnt flagged the other drive fail.. bad sign. — yagmoth555, Jun 02 '16 at 22:50
@yagmoth555 agreed. However, I'd rather have some partially corrupted data than none at all. — KOGI, Jun 03 '16 at 03:44
Which distro are you using, and have you messed with the syslog/rsyslog setup? — MadHatter, Jun 03 '16 at 10:29
@MadHatter Ubuntu 16.04, fresh install + mysql + samba (haven't messed with syslog/rsyslog) — KOGI, Jun 04 '16 at 00:24
Can't help with ubuntu, sorry; any ubuntu people know where the kernel logs? — MadHatter, Jun 04 '16 at 02:11
@MadHatter I found the kernel log location and have added it to the bottom of the original question above. — KOGI, Jun 05 '16 at 20:24

score 3 · Answer 1 · answered Jun 02 '16 at 22:10

3

Maybe give this guys approach a try if you are brave (or desperate) enough: http://matrafox.info/how-to-get-md-raid-array-to-rebuild-even-if-read-errors.html

His hack forces the badblocks to be reallocated on the "good" disk by writing nonsense to them. Which shouldnt matter for your data, since the sector in question is dead anyways, and the disks controller remaps the bad sector to a spare one.

Use at your own risk!

answered Jun 02 '16 at 22:10

tim

1,197
3
10
23

How do I know if my bad sectors are in use or not? – KOGI Jun 03 '16 at 03:44
1

There are various ways but the short answer is: if you read all your data your system will produce read errors for every bad sector in use in the dmesg / messages log. – tim Jun 03 '16 at 05:53

MadHatter · Accepted Answer · 2016-06-07T10:13:06.770

3

The critical lines are

Jun  5 13:21:55 vault kernel: [154199.676477] blk_update_request: I/O error, dev sda, sector 954247056
Jun  5 13:21:55 vault kernel: [154199.676485] md/raid1:md126: sda: unrecoverable I/O read error for block 954244864
Jun  5 13:21:55 vault kernel: [154199.676488] ata3: EH complete
Jun  5 13:21:55 vault kernel: [154199.676597] md: md126: recovery interrupted.

To recover md126 the kernel needs to copy sda1 to sdb1 - but it's having a read error on sda, which is the only surviving half of that mirror. This array is toast. It's time to shotgun /dev/sda and restore from backups (or, absent backups, save what you can from the existing array before shotgunning it).

Edit: If you're trying to get data off the failing drive, tools like safecopy may come in useful (disclaimer: I have no connection with the authors or the project).

edited Jun 07 '16 at 10:13

answered Jun 06 '16 at 13:01

MadHatter

78,442
20
178
229

Is there any way to just force no data into that sector and continue? It would corrupt whatever file was using that sector, but allow the sync to continue... – KOGI Jun 06 '16 at 16:25
1

You're assuming it was a file. It might have been a directory entry, in which case you could lose a lot more. – MadHatter Jun 06 '16 at 17:42
So, is there any way to copy as much info as possible to a new RAID1 array, and when that's complete swap that new array into the RAID0? Basically replace `/dev/md126` with a new array, but keep `/dev/md127` and `/dev/md125` in place? – KOGI Jun 08 '16 at 18:13
I do not believe so, no. `md125` has the FS, and is RAID0, so must be recreated if any of its components is reinitialised. – MadHatter Jun 09 '16 at 05:07
So, my ENTIRE array is toast, then? – KOGI Jun 09 '16 at 15:15
You can try to read all the data off it as it is now, as I said in my edit above; but the whole array will need to be reinitialised, with at least one more new HDD, before it can be reused. – MadHatter Jun 09 '16 at 15:45

Can't get my RAID array out of degraded mode

Drive Layout

root@vault:~# cat /proc/mdstat

root@vault:~# parted -l

root@vault:~# sudo mdadm --detail /dev/md126

root@vault:~# tail -f /var/log/kern.log

2 Answers2

`root@vault:~# cat /proc/mdstat`

`root@vault:~# parted -l`

`root@vault:~# sudo mdadm --detail /dev/md126`

`root@vault:~# tail -f /var/log/kern.log`