1

I have ubuntu 18.04.2 based server with Software based RAID5 partition for Data. After rebooting the system. It is unable to mount automatically. When I try to mount using command "sudo mount -a". I get following error.

mount: wrong fs type, bad option, bad superblock on /dev/md0,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

dmesg | tail command out is as following

[   45.596679] Bridge firewalling registered
[   45.628950] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[   45.712031] ip_tables: (C) 2000-2006 Netfilter Core Team
[   45.765630] Initializing XFRM netlink socket
[   45.772559] Netfilter messages via NETLINK v0.30.
[   45.774814] ctnetlink v0.93: registering with nfnetlink.
[   45.944103] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready
[   46.792337] aufs au_opts_verify:1597:dockerd[2191]: dirperm1 breaks the protection by the permission bits on the lower branch
[   75.420002] EXT4-fs (md0): can't read group descriptor 5242
[  605.733833] EXT4-fs (md0): can't read group descriptor 5242

Output of mdadm --detail /dev/md0 is as following

dev/md0:
        Version : 1.2
  Creation Time : Sun Jan  6 20:39:33 2002
     Raid Level : raid5
     Array Size : 46883119104 (44711.23 GiB 48008.31 GB)
  Used Dev Size : 5860389888 (5588.90 GiB 6001.04 GB)
   Raid Devices : 9
  Total Devices : 8
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Fri Jun  7 09:19:58 2019
          State : clean, degraded 
 Active Devices : 8
Working Devices : 8
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : ubuntu:0  (local to host ubuntu)
           UUID : e5109f94:e146b6d5:686e76d7:36787295
         Events : 869392

    Number   Major   Minor   RaidDevice State
       0       8       33        0      active sync   /dev/sdc1
       1       8       49        1      active sync   /dev/sdd1
       2       8       65        2      active sync   /dev/sde1
       3       8       81        3      active sync   /dev/sdf1
       8       0        0        8      removed
       5       8      113        5      active sync   /dev/sdh1
       6       8      129        6      active sync   /dev/sdi1
       7       8      145        7      active sync   /dev/sdj1
       8       8      161        8      active sync   /dev/sdk1

Output for fsck /dev/md0

fsck from util-linux 2.27.1
e2fsck 1.42.13 (17-May-2015)
fsck.ext4: Attempt to read block from filesystem resulted in short read while trying to open /dev/md0
Could this be a zero-length partition?

Output for mke2fs -n /dev/sdg1

mke2fs 1.42.13 (17-May-2015)
/dev/sdg1 contains a linux_raid_member file system labelled 'ubuntu:0'
Proceed anyway? (y,n) y
Creating filesystem with 1465130240 4k blocks and 183144448 inodes
Filesystem UUID: f2f50279-73b6-4d89-9251-e5056ace74de
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
    102400000, 214990848, 512000000, 550731776, 644972544

Please help.

Thank you

Manu
  • 11
  • 2
  • Have you tried `# fsck /dev/md0`? – RonJohn Jun 07 '19 at 18:02
  • Yes I did. I have updated the query. – Manu Jun 07 '19 at 19:35
  • That looks bad. Do you have a recent backup? I found this which might help: https://unix.stackexchange.com/questions/114429/short-read-while-trying-to-open-partition – RonJohn Jun 07 '19 at 19:51
  • NO, we didn't :'( . The raid is ~50TB. It will not be possible to create mirror for 50TB file system. – Manu Jun 07 '19 at 20:12
  • I am thinking to follow instruction on https://serverfault.com/questions/206511/how-do-i-replace-a-disk-marked-as-removed-from-a-linux-md-raid-5-array . However, I am worried about loosing data as well. – Manu Jun 07 '19 at 21:09
  • You are missing a disk. Try to replace it. – Michael Hampton Jun 08 '19 at 00:50
  • @MichaelHampton, Disk it there. I have updated the outcome of `mkd2fs` command in query. – Manu Jun 09 '19 at 06:16
  • The output for /dev/sdg1 creates the impression that you don't have a clear understanding of the technologies used in this setup. If your data needs to be recovered, and you don't have a backup, I suggest keeping the hands off the system, and trying to understand what you are dealing with (e.g. on a freshly installed virtual system, with some virtual drives atteched to it). You need to clearly understand the differences between `block device, raid array, file system` and the releations to at least the commands `mdadm, fdisk, mke2fs, fsck` and as well the related device names in `/dev`. – hargut Jun 09 '19 at 08:13
  • Hi @hargut . Yes, I do not have much knowledge of the technology and trying to recover data.That is why I am here. I am gathering the information and avoiding any kind of command which might alter the data in the disks. /dev/sdg1 is the device which appears removed from the raid (`mdadm --detail /dev/md0` command). – Manu Jun 09 '19 at 14:28
  • If your data is critical, take the time to understand with what you are dealing. Everything else is risky for the data. From my point of view it does not make sense to run any file system related command directly against the block device of sdg1, as it will not have a filesystem related data at the positions where such tools would expect it. – hargut Jun 09 '19 at 15:01

0 Answers0