Missing mdam RAID1 array during scan

Question

I have built a new server (Ubuntu 18.04.1) with 4 HDD.

My plan is to use RAID1 overall and came up with this layout after investigation:

sda, sdb four primary partitions (all as fd) with sda1/sdb1 as /boot, sda2/sdb2 as /, sda3/sdb3 as /var and sda4/sdb4 as part of /home

sdc,sdd two primary partitions (all as fd) with sdc1/sdd1 as rest of /home and sdc2/sdd2 to be used as swap.

Currently the swapfile resides in /

~$ ls -lart /swapfile 
-rw------- 1 root root 2147483648 aug 17 08:50 /swapfile

and my expectation is that this swap file should be on md2 (which is missing).

Question is: Can I just create a new array with mdadm and somehow make the OS use that area for swap instead of the swapfile ?

In my first attempts I tried to prepare all configurations during install but failed miserably and in my last attempt I was only setting up /boot and / in the raid configuration and no LVM config.

The planned part worked as expected but somehow a remnant from previous attempts has left me with a missing swap partition.

What I mean is that the mdadm.conf has entry for the swap(besides md0:/boot and md1:/):

ARRAY /dev/md/2  metadata=1.2 UUID=4b13781e:ea3a8209:74173443:77c65a9d name=LOPEZSRV:2

And checking blkid:

~$ sudo blkid
/dev/sdc2: UUID="4b13781e-ea3a-8209-7417-344377c65a9d" UUID_SUB="6ee8ecac-1e75-f5dc-3def-508cba28a512" LABEL="LOPEZSRV:2" TYPE="linux_raid_member" PARTUUID="f260af1c-02"
/dev/sdd2: UUID="4b13781e-ea3a-8209-7417-344377c65a9d" UUID_SUB="3efa76ad-3662-87e6-ed4d-b072cb0e7274" LABEL="LOPEZSRV:2" TYPE="linux_raid_member" PARTUUID="3bce6013-02"

shows them being a member of an array

Also checking the details

~$ sudo mdadm -E /dev/sdc2
/dev/sdc2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 4b13781e:ea3a8209:74173443:77c65a9d
           Name : LOPEZSRV:2  (local to host LOPEZSRV)
  Creation Time : Thu Aug 16 14:08:06 2018
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 3906762752 (1862.89 GiB 2000.26 GB)
     Array Size : 1953381376 (1862.89 GiB 2000.26 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=18446744073701738496 sectors
          State : active
    Device UUID : 6ee8ecac:1e75f5dc:3def508c:ba28a512

Internal Bitmap : 8 sectors from superblock
    Update Time : Thu Aug 16 14:40:25 2018
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 19220d93 - correct
         Events : 393


   Device Role : Active device 0
   Array State : AA ('A' == active, '.' == missing, 'R' == replacing)

~$ sudo mdadm -E /dev/sdd2
/dev/sdd2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 4b13781e:ea3a8209:74173443:77c65a9d
           Name : LOPEZSRV:2  (local to host LOPEZSRV)
  Creation Time : Thu Aug 16 14:08:06 2018
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 3906762752 (1862.89 GiB 2000.26 GB)
     Array Size : 1953381376 (1862.89 GiB 2000.26 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=18446744073701738496 sectors
          State : active
    Device UUID : 3efa76ad:366287e6:ed4db072:cb0e7274

Internal Bitmap : 8 sectors from superblock
    Update Time : Thu Aug 16 14:40:25 2018
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 6b6a513d - correct
         Events : 393


   Device Role : Active device 1
   Array State : AA ('A' == active, '.' == missing, 'R' == replacing)

But doing a scan:

~$ sudo mdadm --detail --scan
ARRAY /dev/md/1 metadata=1.2 name=LOPEZSRV:1 UUID=5c7e415a:fe060391:ae611987:474558ec
ARRAY /dev/md/0 metadata=1.2 name=LOPEZSRV:0 UUID=e6a5ed7b:c613d763:e8e6c8a3:8a13ecb1

I only get the arrays for /boot and /

Looking in /proc/mdstat:

~$ cat /proc/mdstat 
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sda1[0] sdb1[1]
      1949696 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sdb2[1] sda2[0]
      97589248 blocks super 1.2 [2/2] [UU]

also only shows the two arrays for /boot and /.

Adding outputs:

~$ sudo mdadm --assemble --scan  --verbose 
mdadm: looking for devices for /dev/md/0
mdadm: No super block found on /dev/md/0 (Expected magic a92b4efc, got 000000ef)
mdadm: no RAID superblock on /dev/md/0
mdadm: No super block found on /dev/md/1 (Expected magic a92b4efc, got 0000040d)
mdadm: no RAID superblock on /dev/md/1
mdadm: /dev/sdd2 has wrong uuid.
mdadm: No super block found on /dev/sdd1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdd1
mdadm: No super block found on /dev/sdd (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdd
mdadm: /dev/sdc2 has wrong uuid.
mdadm: No super block found on /dev/sdc1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdc1
mdadm: No super block found on /dev/sdc (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdc
mdadm: No super block found on /dev/sdb4 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb4
mdadm: No super block found on /dev/sdb3 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb3
mdadm: /dev/sdb2 has wrong uuid.
mdadm: /dev/sdb1 is busy - skipping
mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got a0ddb71c)
mdadm: no RAID superblock on /dev/sdb
mdadm: No super block found on /dev/sda4 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sda4
mdadm: No super block found on /dev/sda3 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sda3
mdadm: /dev/sda2 has wrong uuid.
mdadm: /dev/sda1 is busy - skipping
mdadm: No super block found on /dev/sda (Expected magic a92b4efc, got a0ddb71c)
mdadm: no RAID superblock on /dev/sda
mdadm: cannot open device /dev/sr0: No medium found
mdadm: looking for devices for /dev/md/1
mdadm: No super block found on /dev/md/0 (Expected magic a92b4efc, got 000000ef)
mdadm: no RAID superblock on /dev/md/0
mdadm: No super block found on /dev/md/1 (Expected magic a92b4efc, got 0000040d)
mdadm: no RAID superblock on /dev/md/1
mdadm: /dev/sdd2 has wrong uuid.
mdadm: No super block found on /dev/sdd1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdd1
mdadm: No super block found on /dev/sdd (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdd
mdadm: /dev/sdc2 has wrong uuid.
mdadm: No super block found on /dev/sdc1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdc1
mdadm: No super block found on /dev/sdc (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdc
mdadm: No super block found on /dev/sdb4 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb4
mdadm: No super block found on /dev/sdb3 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb3
mdadm: /dev/sdb2 is busy - skipping
mdadm: /dev/sdb1 has wrong uuid.
mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got a0ddb71c)
mdadm: no RAID superblock on /dev/sdb
mdadm: No super block found on /dev/sda4 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sda4
mdadm: No super block found on /dev/sda3 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sda3
mdadm: /dev/sda2 is busy - skipping
mdadm: /dev/sda1 has wrong uuid.
mdadm: No super block found on /dev/sda (Expected magic a92b4efc, got a0ddb71c)
mdadm: no RAID superblock on /dev/sda
mdadm: cannot open device /dev/sr0: No medium found
mdadm: looking for devices for /dev/md/2
mdadm: No super block found on /dev/md/0 (Expected magic a92b4efc, got 000000ef)
mdadm: no RAID superblock on /dev/md/0
mdadm: No super block found on /dev/md/1 (Expected magic a92b4efc, got 0000040d)
mdadm: no RAID superblock on /dev/md/1
mdadm: No super block found on /dev/sdd1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdd1
mdadm: No super block found on /dev/sdd (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdd
mdadm: No super block found on /dev/sdc1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdc1
mdadm: No super block found on /dev/sdc (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdc
mdadm: No super block found on /dev/sdb4 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb4
mdadm: No super block found on /dev/sdb3 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb3
mdadm: /dev/sdb2 has wrong uuid.
mdadm: /dev/sdb1 has wrong uuid.
mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got a0ddb71c)
mdadm: no RAID superblock on /dev/sdb
mdadm: No super block found on /dev/sda4 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sda4
mdadm: No super block found on /dev/sda3 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sda3
mdadm: /dev/sda2 has wrong uuid.
mdadm: /dev/sda1 has wrong uuid.
mdadm: No super block found on /dev/sda (Expected magic a92b4efc, got a0ddb71c)
mdadm: no RAID superblock on /dev/sda
mdadm: cannot open device /dev/sr0: No medium found
mdadm: /dev/sdd2 is identified as a member of /dev/md/2, slot 1.
mdadm: /dev/sdc2 is identified as a member of /dev/md/2, slot 0.
mdadm: failed to add /dev/sdd2 to /dev/md/2: Invalid argument
mdadm: failed to add /dev/sdc2 to /dev/md/2: Invalid argument
mdadm: failed to RUN_ARRAY /dev/md/2: Invalid argument

~$ sudo mdadm --assemble /dev/md2 --uuid=4b13781e:ea3a8209:74173443:77c65a9d --verbose
mdadm: looking for devices for /dev/md2
mdadm: no recogniseable superblock on /dev/md/0
mdadm: no recogniseable superblock on /dev/md/1
mdadm: No super block found on /dev/sdd1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdd1
mdadm: No super block found on /dev/sdd (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdd
mdadm: No super block found on /dev/sdc1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdc1
mdadm: No super block found on /dev/sdc (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdc
mdadm: No super block found on /dev/sdb4 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb4
mdadm: No super block found on /dev/sdb3 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb3
mdadm: /dev/sdb2 has wrong uuid.
mdadm: /dev/sdb1 has wrong uuid.
mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got a0ddb71c)
mdadm: no RAID superblock on /dev/sdb
mdadm: No super block found on /dev/sda4 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sda4
mdadm: No super block found on /dev/sda3 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sda3
mdadm: /dev/sda2 has wrong uuid.
mdadm: /dev/sda1 has wrong uuid.
mdadm: No super block found on /dev/sda (Expected magic a92b4efc, got a0ddb71c)
mdadm: no RAID superblock on /dev/sda
mdadm: cannot open device /dev/sr0: No medium found
mdadm: /dev/sdd2 is identified as a member of /dev/md2, slot 1.
mdadm: /dev/sdc2 is identified as a member of /dev/md2, slot 0.
mdadm: failed to add /dev/sdd2 to /dev/md2: Invalid argument
mdadm: failed to add /dev/sdc2 to /dev/md2: Invalid argument
mdadm: failed to RUN_ARRAY /dev/md2: Invalid argument

~$ cat /proc/mdstat 
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sda1[0] sdb1[1]
      1949696 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sdb2[1] sda2[0]
      97589248 blocks super 1.2 [2/2] [UU]

unused devices: <none>

~$ sudo modprobe raid1
~$ sudo mdadm --assemble /dev/md2 --uuid=4b13781e:ea3a8209:74173443:77c65a9d --verbose --force --update=summaries
mdadm: looking for devices for /dev/md2
mdadm: no recogniseable superblock on /dev/md/0
mdadm: no recogniseable superblock on /dev/md/1
mdadm: No super block found on /dev/sdd1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdd1
mdadm: No super block found on /dev/sdd (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdd
mdadm: No super block found on /dev/sdc1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdc1
mdadm: No super block found on /dev/sdc (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdc
mdadm: No super block found on /dev/sdb4 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb4
mdadm: No super block found on /dev/sdb3 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb3
mdadm: /dev/sdb2 has wrong uuid.
mdadm: /dev/sdb1 has wrong uuid.
mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got a0ddb71c)
mdadm: no RAID superblock on /dev/sdb
mdadm: No super block found on /dev/sda4 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sda4
mdadm: No super block found on /dev/sda3 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sda3
mdadm: /dev/sda2 has wrong uuid.
mdadm: /dev/sda1 has wrong uuid.
mdadm: No super block found on /dev/sda (Expected magic a92b4efc, got a0ddb71c)
mdadm: no RAID superblock on /dev/sda
mdadm: cannot open device /dev/sr0: No medium found
mdadm: --update=summaries not understood for 1.x metadata
double free or corruption (!prev)
Avbruten (SIGABRT)

~$ dmesg    
[lör aug 18 08:35:54 2018] md: md2 stopped.
[lör aug 18 08:35:54 2018] md: sdd2 does not have a valid v1.2 superblock, not importing!
[lör aug 18 08:35:54 2018] md: md_import_device returned -22
[lör aug 18 08:35:54 2018] md: md2 stopped.

hargut · Answer 1 · 2018-08-17T15:49:19.737

1

To assemble drives which are not in the /etc/mdadm.conf one can use mdadm --assemble --scan. This command will let mdam look through all drives that are not active in an mdadm array and search for a mdadm superblock. If mdadm superblocks are discovered, and have the same UUID mdadm will try to automatically assemble the arrays. As your output shows from blkid that uuids are the same, this proceedure should fix your problem.

Did you try mdadm --assemble --scan?

edited Aug 17 '18 at 15:49

answered Aug 17 '18 at 14:42

hargut

3,848
6
10

Results in: mdadm: failed to add /dev/sdd2 to /dev/md/2: Invalid argument mdadm: failed to add /dev/sdc2 to /dev/md/2: Invalid argument mdadm: failed to RUN_ARRAY /dev/md/2: Invalid argument – IgorLopez Aug 17 '18 at 15:02
What does `mdadm -E /dev/sdd2` give? – hargut Aug 17 '18 at 15:32
This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. - [From Review](/review/low-quality-posts/379650) – Patrick Mevzek Aug 17 '18 at 15:34
@Patrick Mevzek: I've expanded the answer with technical details and explaination why this command should work and be the right answer. – hargut Aug 17 '18 at 15:50
I did not put mdadm -E /dev/sdd2 at first since it was the same output as for sdc2 but it is now in the post. And yes to the question, My first reponse to hargut is the result from: ~$ sudo mdadm --assemble --scan – IgorLopez Aug 17 '18 at 16:36
`mdadm -E ...` outputs look clean, no errors, same amount of events, same uuid. Can you try `--assemble --scan --verbose`, or --assemble /dev/md2 --uuid --verbose`. – hargut Aug 17 '18 at 16:45
I have added the verbose output from the --assemble operation. I do not understand which of the UUID for the second suggestion. Do you mean the Array UUID used for sdc2/sdd2 since there is no /dev/md2 device yet? – IgorLopez Aug 17 '18 at 17:05
Looks good, verbose output shows that devices are seen, and would be joined properly. – hargut Aug 17 '18 at 19:25
Yes, second command with array uuid. In case you're unsure I suggest to check `man mdadm`. `/dev/md2` should be created when the array is assembled. – hargut Aug 17 '18 at 19:28
Done that and added the result in the post (last section). Still no md2 in /procstat/mdstat. Note that I am a total noob when it comes to RAID but my plan was to remove the entry for /dev/md/2 in mdadm.conf and then create the array in a normal way and update the mdadm.conf afterwards. I assume that it could work, e.g. I will have an md2 device but is it possible for me to mount md2 as swap in fstab so the OS uses that and what should I do with the swapfile located at root? – IgorLopez Aug 17 '18 at 20:11
Check `dmesg` output after attempting to assemble the Raid. It could help to assemble with `--force --update=summaries`. – hargut Aug 18 '18 at 05:16
It could also be necessary to call `modprobe raid1` before trying to assemle the raid. – hargut Aug 18 '18 at 05:28

score 0 · Answer 2 · answered Aug 18 '18 at 05:25

To disable the swapfile you can do the following without rebooting: swapoff /your/path/to/swapfile and removing the swap entrie from /etc/fstab. In case your machine is under load and using swap you have to be careful before running swapoff. Depending on the utilization of swap and the amount of free memory swapoff could cause trouble.

with rebooting: remove the according line in /etc/fstab and reboot.

To add the new swapdrive you need mkswap /dev/yourswapdrive, swapon /dev/yourswapdrive and an acvording line in /etc/fstab to enable swap at boot.

(In your case it should be sufficient to replace /your/path/to/swapfile with /dev/yourswapdrive in /etc/fstab.)

Since there is no progress with assembling the md2 array(post is updated) I think I will just repartition sdc and sdd (they contain no data yet) and create the raids manually and then do the swapoff, swapon operations. — IgorLopez, Aug 18 '18 at 06:47

score 0 · Answer 3 · answered Aug 18 '18 at 10:06

The solution to the problem not being able to reassemble the array is now solved by help from hargut.

It was neccessary to remove the old Superblocks in /dev/sdc2, /dev/sdd2 and then continue creating new Arrays.

~$ sudo mdadm --zero-superblock /dev/sdc2
~$ sudo mdadm --zero-superblock /dev/sdd2
~$ sudo mdadm --create /dev/md2 --level=mirror --raid-devices=2 /dev/sd[c-d]1
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md2 started.
~$ sudo mdadm --create /dev/md3 --level=mirror --raid-devices=2 /dev/sd[c-d]2
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md3 started.
~$ sudo mdadm --detail --scan
ARRAY /dev/md/1 metadata=1.2 name=LOPEZSRV:1 UUID=5c7e415a:fe060391:ae611987:474558ec
ARRAY /dev/md/0 metadata=1.2 name=LOPEZSRV:0 UUID=e6a5ed7b:c613d763:e8e6c8a3:8a13ecb1
ARRAY /dev/md2 metadata=1.2 name=LOPEZSRV:2 UUID=73164a72:102df9c1:7a67a2f1:024f733d
ARRAY /dev/md3 metadata=1.2 name=LOPEZSRV:3 UUID=1aeb2b00:f47e62bb:add9da71:60799799
~$ cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md3 : active raid1 sdd2[1] sdc2[0]
      1949474816 blocks super 1.2 [2/2] [UU]
      [===>.................]  resync = 16.8% (328936512/1949474816) finish=216.5min speed=124710K/sec
      bitmap: 15/15 pages [60KB], 65536KB chunk

md2 : active raid1 sdd1[1] sdc1[0]
      3903488 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sda1[0] sdb1[1]
      1949696 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sdb2[1] sda2[0]
      97589248 blocks super 1.2 [2/2] [UU]

unused devices: <none>

Since I am a noob I have no clue to the different ARRAY paths.

From install I got the

ARRAY /dev/md/0 for dev/md0
ARRAY /dev/md/1 for /dev/md1

but during manual creating the arrays I got

ARRAY /dev/md2 for /dev/md2
ARRAY /dev/md3 for /dev/md3

But I assume it does not matter.

I will update mdadm.conf with the new arrays.

Now I have only to wait for another 216 minutes before doing the swapoff/swapon operations.

Missing mdam RAID1 array during scan

3 Answers3