mdadm assemble ignoring --force? Raid10 minimal difference in 'Actions' between Harddisks

Question

My server Is runnIng a 4 dIsk software raId 10 based on mdadm.

EarlIer today a small power outage shut down the server and now It only boots to the InItrd emergency system (ubuntu 16.04), complaInIng about two of the four dIsks beIng 'possIbly out of date'. There Is a mInor dIfference In Events between the four dIsks. Two have 6531 events, two 6527 events.

I have trIed to reassemble the raId forcefully:

mdadm --assemble /dev/md0 /dev/sdb /dev/sdc /dev/sdd /dev/sde --force -v

As well as wIthout force:

mdadm --assemble /dev/md0 /dev/sdb /dev/sdc /dev/sdd /dev/sde -v

It keeps complaInIng about the drIves beIng out of date.

How could I get mdadm to Ignore the small dIfference In events?

Here are the outputs:

root@ubuntu:/home/ubuntu# mdadm --examine /dev/sdc
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 8397999f:cd6b31f4:64d31961:759bded9
           Name : debian:0
  Creation Time : Thu May  4 19:39:57 2017
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 624880304 (297.97 GiB 319.94 GB)
     Array Size : 585850880 (558.71 GiB 599.91 GB)
  Used Dev Size : 585850880 (279.36 GiB 299.96 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=39029424 sectors
          State : active
    Device UUID : 0774b3a6:acc10734:1e7c6f76:98b36729

Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Aug 11 13:06:38 2017
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 86b594a0 - correct
         Events : 6531

         Layout : offset=2
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AA.A ('A' == active, '.' == missing, 'R' == replacing)


root@ubuntu:/home/ubuntu# mdadm --examine /dev/sdd
/dev/sdd:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 8397999f:cd6b31f4:64d31961:759bded9
           Name : debian:0
  Creation Time : Thu May  4 19:39:57 2017
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 976511024 (465.64 GiB 499.97 GB)
     Array Size : 585850880 (558.71 GiB 599.91 GB)
  Used Dev Size : 585850880 (279.36 GiB 299.96 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=390660144 sectors
          State : active
    Device UUID : 6e83f7bf:a19005dc:d714aa81:dc11bd5f

Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Aug 11 13:06:38 2017
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 9e7cdf7b - correct
         Events : 6531

         Layout : offset=2
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AA.A ('A' == active, '.' == missing, 'R' == replacing)


root@ubuntu:/home/ubuntu# mdadm --examine /dev/sde
/dev/sde:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 8397999f:cd6b31f4:64d31961:759bded9
           Name : debian:0
  Creation Time : Thu May  4 19:39:57 2017
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 624880304 (297.97 GiB 319.94 GB)
     Array Size : 585850880 (558.71 GiB 599.91 GB)
  Used Dev Size : 585850880 (279.36 GiB 299.96 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=39029424 sectors
          State : active
    Device UUID : f4231c7d:5f51fc96:648cb20e:a07f7845

Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Aug 11 13:04:25 2017
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 7465b00b - correct
         Events : 6527

         Layout : offset=2
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)



root@ubuntu:/home/ubuntu# mdadm --examine /dev/sdf
/dev/sdf:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 8397999f:cd6b31f4:64d31961:759bded9
           Name : debian:0
  Creation Time : Thu May  4 19:39:57 2017
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 585852560 (279.36 GiB 299.96 GB)
     Array Size : 585850880 (558.71 GiB 599.91 GB)
  Used Dev Size : 585850880 (279.36 GiB 299.96 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=1680 sectors
          State : active
    Device UUID : d2440649:3f578698:cf4d7abe:306e7fa3

Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Aug 11 13:04:25 2017
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 4d5502a6 - correct
         Events : 6527

         Layout : offset=2
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)


root@ubuntu:/home/ubuntu# mdadm --assemble /dev/md0 /dev/sdc /dev/sdd /dev/sde /dev/sdf -v --force
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdc is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdd is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sde is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdf is identified as a member of /dev/md0, slot 3.
mdadm: added /dev/sdd to /dev/md0 as 1
mdadm: added /dev/sde to /dev/md0 as 2 (possibly out of date)
mdadm: added /dev/sdf to /dev/md0 as 3 (possibly out of date)
mdadm: added /dev/sdc to /dev/md0 as 0
mdadm: /dev/md0 assembled from 2 drives - not enough to start the array.

root@ubuntu:/home/ubuntu# mdadm --assemble /dev/md0 /dev/sdc /dev/sdd /dev/sde /dev/sdf -v --force --run
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdc is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdd is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sde is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdf is identified as a member of /dev/md0, slot 3.
mdadm: added /dev/sdd to /dev/md0 as 1
mdadm: added /dev/sde to /dev/md0 as 2 (possibly out of date)
mdadm: added /dev/sdf to /dev/md0 as 3 (possibly out of date)
mdadm: added /dev/sdc to /dev/md0 as 0
mdadm: failed to RUN_ARRAY /dev/md0: Input/output error
mdadm: Not enough devices to start the array.

Copy here the error message. And "I" is always uppercase on English. — peterh, Aug 12 '17 at 00:26
It is not an error message. If you can't explain your problem clearly, your question will be closed as unclear, and later it will be deleted. — peterh, Aug 13 '17 at 16:29
Re read the log, first I am providing drive info and at the bottom there is the concrete error that is happening when I am trying to assemble is. Also my question is clear. There is a visible difference in actions between the drives. --force should make mdadm ignore those differences, however it is not ignoring them, as you can see in the log. — Kotwarrior, Aug 13 '17 at 17:30

score 2 · Answer 1 · edited Jun 01 '18 at 06:04

I had the same problem with ubuntu-16.04 and a RAID5. Despite all the metadata looking good and a small difference in the number of events mdadm --force wouldn't start the RAID.

I downloaded mdadm-4.0 source, ran make, and ran the same command from that dir (./mdadm) and it started right up.

Seems like the ubuntu-16.04 does ignore --force and makes it impossible to start arrays with even the most minor of problems.

So with 16.04:

mdadm: /dev/sda3 is identified as a member of /dev/md1, slot 0.
mdadm: /dev/sdb3 is identified as a member of /dev/md1, slot 1.
mdadm: /dev/sdd3 is identified as a member of /dev/md1, slot 3.
mdadm: added /dev/sdb3 to /dev/md1 as 1
mdadm: no uptodate device for slot 4 of /dev/md1
mdadm: added /dev/sdd3 to /dev/md1 as 3 (possibly out of date)
mdadm: added /dev/sda3 to /dev/md1 as 0
mdadm: /dev/md1 assembled from 2 drives - not enough to start the array.

With mdadm-4.0 compiled from source:

mdadm: looking for devices for /dev/md1
mdadm: /dev/sda3 is identified as a member of /dev/md1, slot 0.
mdadm: /dev/sdb3 is identified as a member of /dev/md1, slot 1.
mdadm: /dev/sdd3 is identified as a member of /dev/md1, slot 3.
mdadm: forcing event count in /dev/sdd3(3) from 15676 upto 15681
mdadm: clearing FAULTY flag for device 2 in /dev/md1 for /dev/sdd3
mdadm: Marking array /dev/md1 as 'clean'
mdadm: added /dev/sdb3 to /dev/md1 as 1
mdadm: no uptodate device for slot 2 of /dev/md1
mdadm: added /dev/sdd3 to /dev/md1 as 3
mdadm: added /dev/sda3 to /dev/md1 as 0
mdadm: /dev/md1 has been started with 3 drives (out of 4).

mdadm assemble ignoring --force? Raid10 minimal difference in 'Actions' between Harddisks

1 Answers1