RAID10 Recovery

0

I had a RAID10 array fail a few nights ago, I have to admit that I am fairly lost on all of this. I have googled this extensively and I don't understand what is going on.

If anyone here can provide some insigth to get me going in a semi-productive direction, I would be very grateful.

I have posted the output of dmesg and mdadm here: http://pastebin.com/PwtaHP9e

root@sysresccd /root % cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : inactive sdb2[2](S) sda2[4](S) sdd2[3](S) sdc2[1](S)
      3905625088 blocks super 1.1

md0 : active raid1 sdc1[1] sdd1[3] sdb1[2]
      255936 blocks super 1.0 [4/3] [_UUU]

unused devices: 

root@sysresccd /root % dmesg | grep raid
[   10.228754] md: raid0 personality registered for level 0
[   10.230465] md: raid1 personality registered for level 1
[   10.255010] raid6: int64x1   2039 MB/s
[   10.272036] raid6: int64x2   2265 MB/s
[   10.289035] raid6: int64x4   1949 MB/s
[   10.306028] raid6: int64x8   1425 MB/s
[   10.323032] raid6: sse2x1    3218 MB/s
[   10.340017] raid6: sse2x2    4085 MB/s
[   10.357021] raid6: sse2x4    6691 MB/s
[   10.357022] raid6: using algorithm sse2x4 (6691 MB/s)
[   10.358839] md: raid6 personality registered for level 6
[   10.358846] md: raid5 personality registered for level 5
[   10.358847] md: raid4 personality registered for level 4
[   10.363035] md: raid10 personality registered for level 10
[   10.750614] md/raid1:md0: active with 3 out of 4 mirrors
[   44.530653] md: invalid raid superblock magic on sda1
root@sysresccd /root %



root@sysresccd /root % dmesg | grep md
[    0.000000] Linux version 3.2.28-std300-amd64 (root@catalyst) (gcc version 4.4.7 (Gentoo 4.4.7 p1.0, pie-0.4.5) ) #2 SMP Tue Aug 21 18:35:38 UTC 2012
[    2.568955] usb usb1: Manufacturer: Linux 3.2.28-std300-amd64 ehci_hcd
[    2.584953] usb usb2: Manufacturer: Linux 3.2.28-std300-amd64 ehci_hcd
[    2.588163] usb usb3: Manufacturer: Linux 3.2.28-std300-amd64 uhci_hcd
[    2.590887] usb usb4: Manufacturer: Linux 3.2.28-std300-amd64 uhci_hcd
[    2.597811] usb usb5: Manufacturer: Linux 3.2.28-std300-amd64 uhci_hcd
[    2.600571] usb usb6: Manufacturer: Linux 3.2.28-std300-amd64 uhci_hcd
[   10.225876] md: linear personality registered for level -1
[   10.227323] md: multipath personality registered for level -4
[   10.228754] md: raid0 personality registered for level 0
[   10.230465] md: raid1 personality registered for level 1
[   10.358839] md: raid6 personality registered for level 6
[   10.358846] md: raid5 personality registered for level 5
[   10.358847] md: raid4 personality registered for level 4
[   10.363035] md: raid10 personality registered for level 10
[   10.733472] md: md0 stopped.
[   10.734535] md: bind
[   10.734667] md: bind
[   10.734803] md: bind
[   10.734942] md: bind
[   10.734958] md: kicking non-fresh sda1 from array!
[   10.734963] md: unbind
[   10.749093] md: export_rdev(sda1)
[   10.750614] md/raid1:md0: active with 3 out of 4 mirrors
[   10.750641] md0: detected capacity change from 0 to 262078464
[   10.751856]  md0: unknown partition table
[   10.992977] md: md1 stopped.
[   10.994538] md: bind
[   10.994672] md: bind
[   10.994820] md: bind
[   10.994982] md: bind
[   44.523203] md: Autodetecting RAID arrays.
[   44.530653] md: invalid raid superblock magic on sda1
[   44.530656] md: sda1 does not have a valid v0.90 superblock, not importing!
[   44.530701] md: could not open unknown-block(8,2).
[   44.530711] md: could not open unknown-block(8,17).
[   44.530714] md: could not open unknown-block(8,18).
[   44.530718] md: could not open unknown-block(8,33).
[   44.530722] md: could not open unknown-block(8,34).
[   44.530726] md: could not open unknown-block(8,49).
[   44.530729] md: could not open unknown-block(8,50).
[   44.530731] md: Scanned 8 and added 0 devices.
[   44.530732] md: autorun ...
[   44.530733] md: ... autorun DONE.




root@sysresccd /root % mdadm --assemble --scan -v
mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/md/0
mdadm: cannot open device /dev/sr0: No medium found
mdadm: cannot open device /dev/sdd2: Device or resource busy
mdadm: cannot open device /dev/sdd1: Device or resource busy
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: cannot open device /dev/sdc2: Device or resource busy
mdadm: cannot open device /dev/sdc1: Device or resource busy
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: cannot open device /dev/sdb2: Device or resource busy
mdadm: cannot open device /dev/sdb1: Device or resource busy
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: cannot open device /dev/sda2: Device or resource busy
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: cannot open device /dev/loop0: Device or resource busy
mdadm: /dev/sda1 is identified as a member of /dev/md/localhost.localdomain:0, slot 0.
mdadm: no uptodate device for slot 1 of /dev/md/localhost.localdomain:0
mdadm: no uptodate device for slot 2 of /dev/md/localhost.localdomain:0
mdadm: no uptodate device for slot 3 of /dev/md/localhost.localdomain:0
mdadm: added /dev/sda1 to /dev/md/localhost.localdomain:0 as 0
mdadm: /dev/md/localhost.localdomain:0 has been started with 1 drive (out of 4).
mdadm: looking for devices for further assembly
mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/md/0
mdadm: no recogniseable superblock on /dev/md/0
mdadm: cannot open device /dev/sr0: No medium found
mdadm: cannot open device /dev/sdd2: Device or resource busy
mdadm: cannot open device /dev/sdd1: Device or resource busy
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: cannot open device /dev/sdc2: Device or resource busy
mdadm: cannot open device /dev/sdc1: Device or resource busy
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: cannot open device /dev/sdb2: Device or resource busy
mdadm: cannot open device /dev/sdb1: Device or resource busy
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: cannot open device /dev/sda2: Device or resource busy
mdadm: cannot open device /dev/sda1: Device or resource busy
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: cannot open device /dev/loop0: Device or resource busy

root@sysresccd /root % mdadm --examine /dev/sd[abcd]2
/dev/sda2:
          Magic : a92b4efc
        Version : 1.1
    Feature Map : 0x1
     Array UUID : 160f35df:54055a13:77806c03:b12aea12
           Name : localhost.localdomain:1
  Creation Time : Thu Sep 12 02:37:10 2013
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 1953007616 (931.27 GiB 999.94 GB)
     Array Size : 3905492992 (1862.28 GiB 1999.61 GB)
  Used Dev Size : 1952746496 (931.14 GiB 999.81 GB)
    Data Offset : 2048 sectors
   Super Offset : 0 sectors
          State : clean
    Device UUID : ae4e65b3:95c4da54:d295cf4c:de32e851

Internal Bitmap : 8 sectors from superblock
    Update Time : Mon Oct 21 02:21:07 2013
       Checksum : f86dd3de - correct
         Events : 0

         Layout : near=2
     Chunk Size : 512K

   Device Role : spare
   Array State : ..AA ('A' == active, '.' == missing)
/dev/sdb2:
          Magic : a92b4efc
        Version : 1.1
    Feature Map : 0x1
     Array UUID : 160f35df:54055a13:77806c03:b12aea12
           Name : localhost.localdomain:1
  Creation Time : Thu Sep 12 02:37:10 2013
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 1952747520 (931.14 GiB 999.81 GB)
     Array Size : 3905492992 (1862.28 GiB 1999.61 GB)
  Used Dev Size : 1952746496 (931.14 GiB 999.81 GB)
    Data Offset : 262144 sectors
   Super Offset : 0 sectors
          State : clean
    Device UUID : 9fdb73ad:ab3b67ef:62cf56cb:48a7cbae

Internal Bitmap : 8 sectors from superblock
    Update Time : Mon Oct 21 02:21:07 2013
       Checksum : 6873c900 - correct
         Events : 17186

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : ..AA ('A' == active, '.' == missing)
/dev/sdc2:
          Magic : a92b4efc
        Version : 1.1
    Feature Map : 0x1
     Array UUID : 160f35df:54055a13:77806c03:b12aea12
           Name : localhost.localdomain:1
  Creation Time : Thu Sep 12 02:37:10 2013
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 1952747520 (931.14 GiB 999.81 GB)
     Array Size : 3905492992 (1862.28 GiB 1999.61 GB)
  Used Dev Size : 1952746496 (931.14 GiB 999.81 GB)
    Data Offset : 262144 sectors
   Super Offset : 0 sectors
          State : clean
    Device UUID : 5d5f5fbf:4b2140f9:3f622131:79f52f32

Internal Bitmap : 8 sectors from superblock
    Update Time : Mon Oct 21 02:08:42 2013
       Checksum : 6d691077 - correct
         Events : 17175

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAAA ('A' == active, '.' == missing)
/dev/sdd2:
          Magic : a92b4efc
        Version : 1.1
    Feature Map : 0x1
     Array UUID : 160f35df:54055a13:77806c03:b12aea12
           Name : localhost.localdomain:1
  Creation Time : Thu Sep 12 02:37:10 2013
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 1952747520 (931.14 GiB 999.81 GB)
     Array Size : 3905492992 (1862.28 GiB 1999.61 GB)
  Used Dev Size : 1952746496 (931.14 GiB 999.81 GB)
    Data Offset : 262144 sectors
   Super Offset : 0 sectors
          State : clean
    Device UUID : 57e6e332:c199a57b:94846dd1:5551635d

Internal Bitmap : 8 sectors from superblock
    Update Time : Mon Oct 21 02:21:07 2013
       Checksum : 2ed0910d - correct
         Events : 17186

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : ..AA ('A' == active, '.' == missing)

xandout

Posted 2013-10-22T22:01:48.147

Reputation: 33

Thank you @Hennes, I tried to get it formatted and failed. – xandout – 2013-10-22T22:57:48.460

1I am not all that good with the markup myself, but some nice 'tricks' are: 1) Four space in front of code or literal output. 2) If that is too much work, wrap it in pre and slash pre tags. And for one literal command use ticks (`). – Hennes – 2013-10-23T07:52:53.090

Answers

1

Things certainly look strange. Did your drive sda fail and did you replace it?

md0 is active with one failed disk. That shouldn't be a problem, you might just mount it. To get it fixed you need mdadm /dev/md0 -a /dev/sda1 to add the kicked out sda1 again and it should resync.

md1 is stopped and I don't know why exactly. Perhaps you can get it startet with mdadm --run /dev/md1 perhaps with added --force. If that works it should starting resyncing as sda2 is marked as spare.

If it doesn't work you might have to recreate md1 completely with data loss. As the devices are added to the raid at the moment you will have to remove them first with mdadm /dev/md1 --remove /dev/sda2 --remove /dev/sdb2 --remove /dev/sdc2 --remove /dev/sdd2/ and then recreate it with mdadm --create /dev/md1 --level=10 /dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2

Perhaps you have to --fail all devices before md1 will let them go. Perhaps you have to --zero-superblock before you can recreate.

Much luck to you.

Michael Suelmann

Posted 2013-10-22T22:01:48.147

Reputation: 578

Asked: 2013-10-22T22:01:48.147

Viewed: 783 times

Active: 2013-10-23T00:01:56.057