Recovering RAID5 array after zeroing wrong superblock

Question

I have a raid5 array on 4 disks, sda1, sdb1, sdd1 and sde1. sdd1 was removed but is a healthy drive. While attempting to re-add it, I accidentally zeroed out the superblock for sdb1.

sdb1's data is consistent with sda1 and sde1, just missing it's superblock.

Is it possible to recover the superblock on that device and reassemble the array (degraded) without losing any data?

Below is the output of 'mdadm -E /dev/sd{a,d,e}1'.

Am I reading the output correctly in that sde1 is Device 0, sdd1 is Device 2 and sda1 is Device 3? Then that would mean that sdb1 was Device 1. Could I recreate the array with the following command without data loss:

$ sudo mdadm --create /dev/md0 --assume-clean --level=5 --raid-devices=4 /dev/sde1 /dev/sdb1 missing /dev/sda1

/dev/sda1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 81a36846:cf4f0489:219e2546:b1f5b90e
           Name : cowbell:0
  Creation Time : Sun Sep 25 20:24:46 2011
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
     Array Size : 5860538880 (5589.05 GiB 6001.19 GB)
    Data Offset : 1024 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : a0abbc72:f8bb1e2b:e8f81f5c:ed62a979

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Jul 23 18:05:09 2013
       Checksum : 9f46a56 - correct
         Events : 717820

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AA.A ('A' == active, '.' == missing)
/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 81a36846:cf4f0489:219e2546:b1f5b90e
           Name : cowbell:0
  Creation Time : Sun Sep 25 20:24:46 2011
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3907027053 (1863.02 GiB 2000.40 GB)
     Array Size : 5860538880 (5589.05 GiB 6001.19 GB)
  Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 79f5632d:46d5c083:a1c0130c:b83b0654

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Jul 23 17:42:02 2013
       Checksum : a78d6f5b - correct
         Events : 717392

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAA ('A' == active, '.' == missing)
/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 81a36846:cf4f0489:219e2546:b1f5b90e
           Name : cowbell:0
  Creation Time : Sun Sep 25 20:24:46 2011
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3907027053 (1863.02 GiB 2000.40 GB)
     Array Size : 5860538880 (5589.05 GiB 6001.19 GB)
  Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 6abd9160:c8ae382c:e6c73d01:37ac057b

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Jul 23 18:05:09 2013
       Checksum : 9f5d8fa6 - correct
         Events : 717820

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AA.A ('A' == active, '.' == missing)

I have a backup of the important things, yes. There are a bunch of things I'd rather not lose, but wouldn't be the end of the world if I did. — Justin Miller, Jul 24 '13 at 18:23

score 2 · Accepted Answer · edited Apr 13 '17 at 12:14

2

In addition to backups, you might also want to consider just making full dd copies of the disks before attempting any recovery.

That being said, looks like you're on the right track for recovery without data loss. You're interpreting the device numbers correctly. That command looks like what you need.

See my answer here - these things are surprisingly resilient against data loss, and the destruction of the superblock is not damaging to the ability to rebuild the array in the same geometry (see test 4 in that answer).

edited Apr 13 '17 at 12:14

Community

1

answered Jul 24 '13 at 20:00

Shane Madden

112,982
12
174
248

Ok, so I executed the command above and the array started, but when I tried to mount it, I got the "wrong fs type, bad option, bad superblock on /dev/md0...". I created the array according to my output above, so I'm not sure what I did wrong. I don't recall changing any of the defaults like chunk size or anything, so what could be going wrong? – Justin Miller Jul 24 '13 at 21:44
@JustinMiller Yeah, the left-symmetric layout and 512k chunk sizes are defaults, so that should match your initial build. Maybe take a look at some of the forensics work that Anton did in [this answer](http://serverfault.com/a/521485/72586), and compare what you're seeing on your system? – Shane Madden Jul 24 '13 at 22:25
WOOOOOO! I got it mounted and all my data is there! Turns out the above command was fine, the superblocks just had the wrong data offsets. Thankfully I had the 'mdadm -E' from all but one of the drives and fortunately it only took 2 tries before I got the right one for sdb1 (it was same as one of the other drives added around the same time). I will write songs about you and Anton!!! Seriously though, the info that you both posted should, hands down, be on a blog about advanced RAID recovery. Invaluable. – Justin Miller Jul 25 '13 at 12:44
@JustinMiller Awesome, great to hear it! – Shane Madden Jul 25 '13 at 15:58

score 0 · Answer 2 · answered Jul 25 '13 at 09:49

I would try to verify that the underlying physical disks are synced first before I continue re-creating MD device with assumtion it is clean. You can do that with

mdadm -E /dev/sd[abc]1 | grep Event
Events : 0.53120
Events : 0.53108
Events : 0.53120

The output shows that sda and sdc are synced while sdc is late.

Recovering RAID5 array after zeroing wrong superblock

2 Answers2