Unable to mount a RAID 5 volume

Question

Evening all,

Total n00b when it comes to Linux, I use Xpenology (Synology) as NAS software and following a faulty SATA cable lost my RAID 5 volume - within the OS it's sat in a crashed state.

Thanks to the help of another ServerFault user I managed to insert the missing disk(s) and bring it back from degraded, but I have no idea what the next step is.

SUPERSYNO> cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md2 : active raid5 sdd5[0] sdk5[7] sdj5[6] sdi5[5] sdh5[4] sdg5[3] sdf5[2] sde5[1]
  20478048192 blocks super 1.2 level 5, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]

md3 : active raid5 sde6[0] sdj6[2] sdh6[5] sdk6[4] sdi6[6]
  3906971648 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]

md1 : active raid1 sdc2[0] sdd2[2] sde2[1] sdf2[3] sdg2[4] sdh2[5] sdi2[6] sdj2[7] sdk2[8]
  2097088 blocks [12/9] [UUUUUUUUU___]

md0 : active raid1 sdc1[0] sdd1[8] sde1[1] sdf1[7] sdg1[5] sdh1[4] sdi1[6] sdj1[3] sdk1[2]
  2490176 blocks [12/9] [UUUUUUUUU___]

unused devices: <none>

I have no idea why there are 4 md's in here, maybe this is the reason why?

SUPERSYNO> mdadm --detail /dev/md2
/dev/md2:
Version : 1.2
Creation Time : Sat Dec 22 12:19:43 2018
Raid Level : raid5
Array Size : 20478048192 (19529.39 GiB 20969.52 GB)
Used Dev Size : 2925435456 (2789.91 GiB 2995.65 GB)
Raid Devices : 8
Total Devices : 8
Persistence : Superblock is persistent

Update Time : Sat Dec 22 22:13:59 2018
      State : clean
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0

     Layout : left-symmetric
 Chunk Size : 64K

       Name : SUPERSYNO:2  (local to host SUPERSYNO)
       UUID : 7d8b04cf:aebb8d01:6034359b:c4bb62db
     Events : 110

Number   Major   Minor   RaidDevice State
   0       8       53        0      active sync   /dev/sdd5
   1       8       69        1      active sync   /dev/sde5
   2       8       85        2      active sync   /dev/sdf5
   3       8      101        3      active sync   /dev/sdg5
   4       8      117        4      active sync   /dev/sdh5
   5       8      133        5      active sync   /dev/sdi5
   6       8      149        6      active sync   /dev/sdj5
   7       8      165        7      active sync   /dev/sdk5

If I try to mount it, I get

SUPERSYNO> mount /dev/md2 /mnt
mount: mounting /dev/md2 on /mnt failed: No such device

If I run fsck.ext4 I get this:

SUPERSYNO> fsck.ext4 /dev/md2
e2fsck 1.42.6 (21-Sep-2012)
ext2fs_open2: Bad magic number in super-block
fsck.ext4: Superblock invalid, trying backup blocks...
fsck.ext4: Bad magic number in super-block while trying to open /dev/md2

The superblock could not be read or does not describe a correct ext2 filesystem.  
If the device is valid and it really contains an ext2 filesystem (and not swap or 
ufs or something else), then the superblock is corrupt, and you might try running 
e2fsck with an alternate superblock:
e2fsck -b 8193 <device>

If I do an fdisk -l I get this:

SUPERSYNO> fdisk -l

Disk /dev/sdc: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot      Start         End      Blocks  Id System
/dev/sdc1               1         311     2490240  fd Linux raid autodetect
Partition 1 does not end on cylinder boundary
/dev/sdc2             311         572     2097152  fd Linux raid autodetect
Partition 2 does not end on cylinder boundary
fdisk: device has more than 2^32 sectors, can't use all of them

Disk /dev/sdd: 2199.0 GB, 2199023255040 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot      Start         End      Blocks  Id System
/dev/sdd1               1      267350  2147483647+ ee EFI GPT
fdisk: device has more than 2^32 sectors, can't use all of them

Disk /dev/sde: 2199.0 GB, 2199023255040 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot      Start         End      Blocks  Id System
/dev/sde1               1      267350  2147483647+ ee EFI GPT
fdisk: device has more than 2^32 sectors, can't use all of them

Disk /dev/sdf: 2199.0 GB, 2199023255040 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot      Start         End      Blocks  Id System
/dev/sdf1               1      267350  2147483647+ ee EFI GPT
fdisk: device has more than 2^32 sectors, can't use all of them

Disk /dev/sdg: 2199.0 GB, 2199023255040 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot      Start         End      Blocks  Id System
/dev/sdg1               1      267350  2147483647+ ee EFI GPT
fdisk: device has more than 2^32 sectors, can't use all of them

Disk /dev/sdh: 2199.0 GB, 2199023255040 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot      Start         End      Blocks  Id System
/dev/sdh1               1      267350  2147483647+ ee EFI GPT
fdisk: device has more than 2^32 sectors, can't use all of them

Disk /dev/sdi: 2199.0 GB, 2199023255040 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot      Start         End      Blocks  Id System
/dev/sdi1               1      267350  2147483647+ ee EFI GPT
fdisk: device has more than 2^32 sectors, can't use all of them

Disk /dev/sdj: 2199.0 GB, 2199023255040 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot      Start         End      Blocks  Id System
/dev/sdj1               1      267350  2147483647+ ee EFI GPT
fdisk: device has more than 2^32 sectors, can't use all of them

Disk /dev/sdk: 2199.0 GB, 2199023255040 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot      Start         End      Blocks  Id System
/dev/sdk1               1      267350  2147483647+ ee EFI GPT

Prior to these results, in a bid to re-order the drives I ran the command below which took around 9 hours to complete. Hopefully this didn't wipe it?

mdadm -Cf /dev/md2 -e1.2 -n8 -l5 /dev/sdd5 /dev/sde5 /dev/sdf5 /dev/sdg5 /dev/sdh5 /dev/sdi5 /dev/sdj5 /dev/sdk5 -u7d8b04cf:aebb8d01:6034359b:c4bb62db

Do I still have data? Can it be remounted? Any help would be really appreciated

score 1 · Answer 1 · answered Dec 22 '18 at 23:28

1

Yes, you wiped the drives with that last command.

The different number of MD devices is because it's based on partitions, not device. One is probably boot, one root, one data; something like that.

Why you have raid1 volumes with 12 components is a mystery to me, though.

You shouldn't use RAID5 with that many disks BTW. Search the net for why RAID doesn't provide enough redundancy with today's capacity.

And about what originally went wrong; there doesn't seem to be enough info about the cause.

answered Dec 22 '18 at 23:28

Halfgaar

7,921
5
42
81

Dammit. Was hoping you wouldn't say that. Can I just double check? I ran that last command as recommended from here : https://forum.synology.com/enu/viewtopic.php?f=39&t=102148 and a comment about half way down here: https://serverfault.com/questions/347606/recover-raid-5-data-after-created-new-array-instead-of-re-using I figured the -u7d8b04cf:aebb8d01:6034359b:c4bb62db was telling the re-assemble to pick up where it left off. I'm guessing not. Thank you for the advice, mortified! – Graham Jordan Dec 23 '18 at 09:02
From what I can tell by looking quickly, that `mdadm --create` in the SF post is sort of black magic: it is to re-create the array in such a way that it happens to use the right source blocks to sync the disks. I think it went wrong in how you brought it back from the degraded state (what exactly did you do?). I wouldn't even know how to re-add 2 disks with a loose cable, because they will contain outdated info. I would need to do some careful experiments to see if mdadm is careful with integrity of the remainder of the disks to make it possible to just re-add disappeared disks. – Halfgaar Dec 24 '18 at 10:25
@Halfgaar Re-adding disks to the RAID without recreating it should be safe. But if you wait until two disks are missing from a RAID-5, it might not be totally safe anymore. If a RAID-5 has lost one disk it can continue on the remaining disks, but with risk of data loss if anything happens to the remaining disks. Re-adding the one lost disk will cause a sync that will overwrite all of the re-added disk with up to date data. – kasperd Jan 05 '19 at 14:39

Unable to mount a RAID 5 volume

1 Answers1