I'm trying to stop mdadm from automatically assembling RAID arrays when I attach disks. I had a solution that worked on Ubuntu 14.04, but it doesn't work on 16.04. I've found a thing that works for 16.04, but it's a bit of a hack, so I'm hoping someone will be able to tell me how to do it properly.
The background: I want to be able to start a server, then connect disks (specifically, the server is an instance on AWS, and the disks are EBS volumes). The disks form a RAID array that was previously detached from another instance.
I do not want mdadm to automatically assemble the raid array; for various reasons specific to my setup, it's better for me to do a mdadm --assemble manually.
In Ubuntu 14.04, this was reasonably simple. In /etc/mdadm/mdadm.conf, I added a line saying:
AUTO -all
This is what the man page for mdadm.conf says is the right thing to do. I also ran update-initramfs -u to make sure that the system has that setting enabled on boot. And in 14.04, it works fine: the raid array is not auto-assembled when I attach the disks.
But in 16.04 the system reassembles the array despite of that setting. I've tried restarting the mdadm before connecting the disks to make sure it picks up the config change, running update-initramfs -c -k all just in case the there was a different kernel being used on boot or the initfs needed to be completely re-created, and rebooting just in case there's an extra service that needs restarting. None of those help: as soon as the disks are attached, they're auto-assembled.
The one thing that I've found that does work is the idea from this ServerFault answer -- putting a line in mdadm.conf to tell it to scan /dev/null for partitions to scan for MD superblocks:
DEVICE /dev/null
However, this feels like a pretty horrible (if clever!) hack. It also leads to disconcerting errors in the system journal when disks are attached:
Process '/sbin/mdadm --incremental /dev/xvdg1 --offroot' failed with exit code 1.
What's the right way to prevent the auto-assemble?
[Update] So I've managed to get a minimal repro, and it's definitely a problem, even in the base Ubuntu AMI on AWS (and thus I'd assume in Ubuntu 16.04 in general).
Here's what I did:
- Created a 16.04 Ubuntu instance on AWS
- Attached two 10GB EBS volumes
Created a RAID array using them and put a "canary" file on there to make sure that it really was the same array when I mounted it later:
/sbin/mdadm --create -l0 -n2 /dev/md0 /dev/xvdf /dev/xvdg mkfs.ext4 /dev/md0 mount /dev/md0 /mnt touch /mnt/tweetStopped the instance, detached the volumes.
- Started another instance using the Ubuntu AMI ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20170202 (ami-f0768de6)
Logged in to the new instance, and tailed the system journal to see what happened when I attached the volumes. As at this stage, I'd not changed
/etc/mdadm/mdadm.conf, I was expecting to see it auto-assemble the array.root@ip-10-0-0-67:~# journalctl -f -- Logs begin at Thu 2017-06-08 18:46:12 UTC. -- Jun 08 18:46:27 ip-10-0-0-67 systemd-logind[1099]: New session 1 of user ubuntu. Jun 08 18:46:27 ip-10-0-0-67 systemd[1375]: Reached target Paths. ... Jun 08 18:46:57 ip-10-0-0-67 kernel: blkfront: xvdf: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: enabled; Jun 08 18:46:57 ip-10-0-0-67 kernel: md: bind<xvdf> Jun 08 18:47:10 ip-10-0-0-67 kernel: blkfront: xvdg: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: enabled; Jun 08 18:47:10 ip-10-0-0-67 kernel: md: bind<xvdg> Jun 08 18:47:10 ip-10-0-0-67 kernel: md/raid0:md127: md_size is 41910272 sectors. Jun 08 18:47:10 ip-10-0-0-67 kernel: md: RAID0 configuration for md127 - 1 zone Jun 08 18:47:10 ip-10-0-0-67 kernel: md: zone0=[xvdf/xvdg] Jun 08 18:47:10 ip-10-0-0-67 kernel: zone-offset= 0KB, device-offset= 0KB, size= 20955136KB Jun 08 18:47:10 ip-10-0-0-67 kernel: Jun 08 18:47:10 ip-10-0-0-67 kernel: md127: detected capacity change from 0 to 21458059264 ^C root@ip-10-0-0-67:~# mount /dev/md127 /mnt root@ip-10-0-0-67:~# ls /mnt lost+found tweetSo at this point I'd confirmed that the raid array was auto-assembled with the default config, which is no surprise. The next step was to add the
AUTO -allto/etc/mdadm/mdadm.conf, which left it like this:# mdadm.conf # # Please refer to mdadm.conf(5) for information about this file. # # by default (built-in), scan all partitions (/proc/partitions) and all # containers for MD superblocks. alternatively, specify devices to scan, using # wildcards if desired. #DEVICE partitions containers # auto-create devices with Debian standard permissions CREATE owner=root group=disk mode=0660 auto=yes # automatically tag new arrays as belonging to the local system HOMEHOST <system> # instruct the monitoring daemon where to send mail alerts MAILADDR root # definitions of existing MD arrays AUTO -all # This file was auto-generated on Thu, 02 Feb 2017 18:23:27 +0000 # by mkconf $Id$Next, do a full update-initramfs
root@ip-10-0-0-67:~# update-initramfs -c -k all update-initramfs: Generating /boot/initrd.img-4.4.0-62-generic W: mdadm: /etc/mdadm/mdadm.conf defines no arrays. root@ip-10-0-0-67:~#Next, stop the instance (not terminate -- just switch it off temporarily) and detach the EBS volumes.
Start the instance again, and confirm that the volumes aren't there.
ubuntu@ip-10-0-0-67:~$ cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] unused devices: <none>Tail the system journal while attaching the volumes again, and it auto-assembles:
ubuntu@ip-10-0-0-67:~$ sudo journalctl -f Jun 08 18:55:25 ip-10-0-0-67 systemd[1]: apt-daily.timer: Adding 2h 21min 27.220312s random time. ... Jun 08 18:56:02 ip-10-0-0-67 kernel: blkfront: xvdf: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: enabled; Jun 08 18:56:02 ip-10-0-0-67 kernel: md: bind<xvdf> Jun 08 18:56:15 ip-10-0-0-67 kernel: blkfront: xvdg: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: enabled; Jun 08 18:56:15 ip-10-0-0-67 kernel: md: bind<xvdg> Jun 08 18:56:15 ip-10-0-0-67 kernel: md/raid0:md127: md_size is 41910272 sectors. Jun 08 18:56:15 ip-10-0-0-67 kernel: md: RAID0 configuration for md127 - 1 zone Jun 08 18:56:15 ip-10-0-0-67 kernel: md: zone0=[xvdf/xvdg] Jun 08 18:56:15 ip-10-0-0-67 kernel: zone-offset= 0KB, device-offset= 0KB, size= 20955136KB Jun 08 18:56:15 ip-10-0-0-67 kernel: Jun 08 18:56:15 ip-10-0-0-67 kernel: md127: detected capacity change from 0 to 21458059264 ^CConfirm that the array is indeed the one originally created:
ubuntu@ip-10-0-0-67:~$ sudo mount /dev/md127 /mnt ubuntu@ip-10-0-0-67:~$ ls /mnt lost+found tweet
[Another update] So it looks like the mdadm.conf that is generated into the iniramfs isn't the one from /etc/mdadm/mdadm.conf! On the same machine as the last test:
ubuntu@ip-10-0-0-67:~$ mkdir initramfs
ubuntu@ip-10-0-0-67:~$ cd initramfs/
ubuntu@ip-10-0-0-67:~/initramfs$ uname -a
Linux ip-10-0-0-67 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@ip-10-0-0-67:~/initramfs$ ls /boot
abi-4.4.0-62-generic config-4.4.0-62-generic grub initrd.img-4.4.0-62-generic System.map-4.4.0-62-generic vmlinuz-4.4.0-62-generic
ubuntu@ip-10-0-0-67:~/initramfs$ zcat /boot/initrd.img-4.4.0-62-generic | cpio -i
79595 blocks
ubuntu@ip-10-0-0-67:~/initramfs$ cat etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#
# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers
# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes
# automatically tag new arrays as belonging to the local system
HOMEHOST <system>
# instruct the monitoring daemon where to send mail alerts
MAILADDR root
# definitions of existing MD arrays
ARRAY /dev/md/0 metadata=1.2 UUID=599586fc:a53f9227:04bc7b65:8ad7ab99 name=ip-10-0-0-70:0
ubuntu@ip-10-0-0-67:~/initramfs$
I've also confirmed that if the machine is stopped, has the disks removed, is started again, and then the initramfs is re-generated, the mdadm.conf that's put into there is one like the above, but without the ARRAY line.
It's looking very much like /etc/mdadm/mdadm.conf is completely ignored! But man mdadm.conf on the same machine definitely says that that is the right location for it.
[Yet another update]
Behaviour is definitely different on 14.04. I spun up a 14.04 instance with the default mdadm.conf, and tailed syslog while attaching the disks:
root@ip-10-81-154-136:~# tail -f /var/log/syslog
...
Jun 8 19:29:47 ip-10-81-154-136 kernel: [83702921.994799] blkfront: xvdf: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: disabled;
Jun 8 19:29:47 ip-10-81-154-136 kernel: [83702921.999875] xvdf: unknown partition table
Jun 8 19:29:47 ip-10-81-154-136 kernel: [83702922.053469] md: bind<xvdf>
Jun 8 19:30:04 ip-10-81-154-136 kernel: [83702938.748110] blkfront: xvdg: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: disabled;
Jun 8 19:30:04 ip-10-81-154-136 kernel: [83702938.754217] xvdg: unknown partition table
Jun 8 19:30:04 ip-10-81-154-136 kernel: [83702938.812789] md: bind<xvdg>
Jun 8 19:30:04 ip-10-81-154-136 kernel: [83702938.817276] md/raid0:md127: md_size is 41910272 sectors.
Jun 8 19:30:04 ip-10-81-154-136 kernel: [83702938.817280] md: RAID0 configuration for md127 - 1 zone
Jun 8 19:30:04 ip-10-81-154-136 kernel: [83702938.817281] md: zone0=[xvdf/xvdg]
Jun 8 19:30:04 ip-10-81-154-136 kernel: [83702938.817286] zone-offset= 0KB, device-offset= 0KB, size= 20955136KB
Jun 8 19:30:04 ip-10-81-154-136 kernel: [83702938.817287]
Jun 8 19:30:04 ip-10-81-154-136 kernel: [83702938.817309] md127: detected capacity change from 0 to 21458059264
Jun 8 19:30:04 ip-10-81-154-136 kernel: [83702938.824751] md127: unknown partition table
^C
root@ip-10-81-154-136:~# mount /dev/md127 /mnt
root@ip-10-81-154-136:~# ls /mnt
lost+found tweet
...which meant that they were auto-assembled, as you'd expect.
Then I added the AUTO -all to the mdadm.conf and ran update-initramfs -c -k all, stopped the machine, detached the disks, started it again, and once again tailed syslog while attaching the disks:
root@ip-10-81-154-136:~# tail -f /var/log/syslog
...
Jun 8 19:34:29 ip-10-81-154-136 kernel: [43448402.304449] blkfront: xvdf: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: disabled;
Jun 8 19:34:29 ip-10-81-154-136 kernel: [43448402.309578] xvdf: unknown partition table
Jun 8 19:34:51 ip-10-81-154-136 kernel: [43448424.217476] blkfront: xvdg: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: disabled;
Jun 8 19:34:51 ip-10-81-154-136 kernel: [43448424.222645] xvdg: unknown partition table
^C
So they weren't assembled. Double-check:
root@ip-10-81-154-136:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
unused devices: <none>
**strong text**
Definitely not assembled. Can they be assembled manually?
root@ip-10-81-154-136:~# mdadm --assemble /dev/md0 /dev/xvdf /dev/xvdg
mdadm: /dev/md0 has been started with 2 drives.
root@ip-10-81-154-136:~# mount /dev/md0 /mnt
root@ip-10-81-154-136:~# ls /mnt
lost+found tweet
...yes, they can. And what do we have in the initramfs?
root@ip-10-81-154-136:~# mkdir initramfs
root@ip-10-81-154-136:~# cd initramfs/
root@ip-10-81-154-136:~/initramfs# uname -a
Linux ip-10-81-154-136 3.13.0-100-generic #147-Ubuntu SMP Tue Oct 18 16:48:51 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
root@ip-10-81-154-136:~/initramfs# ls /boot
abi-3.13.0-100-generic config-3.13.0-53-generic initrd.img-3.13.0-53-generic vmlinuz-3.13.0-100-generic
abi-3.13.0-53-generic grub System.map-3.13.0-100-generic vmlinuz-3.13.0-53-generic
config-3.13.0-100-generic initrd.img-3.13.0-100-generic System.map-3.13.0-53-generic
root@ip-10-81-154-136:~/initramfs# zcat /boot/initrd.img-3.13.0-100-generic | cpio -i
131798 blocks
root@ip-10-81-154-136:~/initramfs# cat etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#
# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers
# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes
# automatically tag new arrays as belonging to the local system
HOMEHOST <system>
# instruct the monitoring daemon where to send mail alerts
MAILADDR root
# definitions of existing MD arrays
ARRAY /dev/md/0 metadata=1.2 UUID=599586fc:a53f9227:04bc7b65:8ad7ab99 name=ip-10-0-0-70:0
root@ip-10-81-154-136:~/initramfs#
Once again, nothing from my changes to /etc/mdadm/mdadm.conf made it to the initramfs. On the other hand, at least it appears to be paying attention to the one in the normal filesystem!
Any thoughts would be much appreciated.
