Boot from other root partition to shrink original root filesystem on LVM

Question

How can I change the root partition for Debian 10 to boot from?

I'd like to decrease the size of my root filesystem that sits on an LVM logical volume from remote, that is over SSH only and without booting to a Live CD.

Since we can't shrink the rootfs while it's mounted, I figured I'll just clone the existing rootfs, boot to that and do my resizing from there, then boot to the original rootfs and delete the temp one.

I tried in a VM using Ubuntu Server 18.04 since AFAIK it uses the same "boot-chain" as Debain 10. However, I wasn't able to reliably set the cloned partition as root. After a reboot, I checked with mount and the original root is still used even though

• /etc/fstab has been updated
• /boot/grub/grub.cfg has been updated
• initramfs has been updated

Current configuration

• 1TB RAID1 md0
• 4TB RAID1 md1
• PV on md0
• PV on md1p1
• VG vg0 with both PVs
• LV root as original root fs (UUID xxx)
• LV newroot as temporary root fs (UUID yyy)
• ext4 on vg0-root (UUID aaa)
• ext4 on vg0-newroot (UUID bbb)

/etc/fstab was changed accordingly (replaced device mapper path).

So far, I've rsync'ed the entire old root aaa to newroot bbb, I replaced every occurrence of UUIDs aaa and xxx with bbb and yyy in /boot/grub/grub.cfg (Note: I'm not using grub2 or UEFI boot).

initramfs was updated using update-initramfs -u (after fixing /etc/initramfs-tools/conf.d/resume).

In that order. But my tests in a VM either booted straight to the old root or threw me into a GRUB rescue shell.

update-grub recognized the newroot volume and added matching entries in grub.cfg, selecting them manually at boot in my test VM (not possible via SSH...) also booted the old root.

I've also seen there is a ROOT option for /etc/initramfs-tools/initramfs.conf to hardcode the root partition:

Allows optional root bootarg hardcoding, when no root bootarg can be passed. A root bootarg overrides that special setting.

But since there is a bootarg for root in my grub.cfg this setting shouldn't take effect.

What else is there to configure in order to use another UUID as root on next boot?

Answer 1

Turns out I forgot to replace the root= bootarg in /boot/grub/grub.cfg...

For completeness' sake:

How to move the root partition to another PV in LVM via SSH without a Live system

Note: Always backup imporant data! This method is a hack and should not be used on production systems. In any case, data written in the time between LV clone and boot from the new root partition will always be lost.

Create new LVM on other disk

fdisk /dev/sdb #Create new partition for PV (sdb1)
pvcreate /dev/sdb1
vgcreate vg1 /dev/sdb1
lvcreate -n newroot -L10G vg1 #Size has to be larger than effective usage on current root
mkfs.ext4 -L newroot /dev/mapper/vg1-newroot

Collect info about new LVM

lvdisplay
vgdisplay
pvdisplay
blkid

The following commands will refer to the information presented here as:

• $pv0uuid = UUID of old PV
• $pv1uuid = UUID of new PV
• $vg{0,1}uuid = UUID of {old,new} VG
• $lv{0,1}UUID = UUID of {old,new} LV
• $root{0,1}UUID = UUID of {old,new} root filesystem
• $vg{0,1} = Name of {old,new} VG
• $lv{0,1} = Name of {old,new} LV

And /etc/grub/grub.cfg will be presented as $grubcfg. For UEFI systems, this is most likely /boot/grub2/grub.cfg.

I recommend setting these as variables in bash using pv0uuid=xxxxxx-xxxx... so the commands below can be copy-pasted.

Set new root in GRUB configuration

These commands will replace text in the GRUB configuration:

sed -i "s/$pv0uuid/$pv1uuid/g" $grubcfg
sed -i "s/$vg0uuid/$vg1uuid/g" $grubcfg
sed -i "s/$lv0uuid/$lv1uuid/g" $grubcfg
sed -i "s/$root0uuid/$root1uuid/g" $grubcfg
sed -i "s/$vg0/$vg1/g" $grubcfg #BEWARE COMMON NAMES
sed -i "s/$lv0/$lv1/g" $grubcfg #BEWARE COMMON NAMES
sed -i "s/\/dev\/mapper\/$vg0-$lv0/\/dev\/mapper\/$vg1-$lv1/g" /etc/fstab #BEWARE COMMON NAMES

BEWARE COMMON NAMES: sed will replace every occurrence of matched text, it is therefore recommended to make these changes by hand since replacing words like root used elsewhere in the GRUB or fstab configuration will render it invalid!

I recommend to use nano and search (CTRL+W) for the VG and LV names and replace them manually. Searching for the PV device name is also recommended, just in case...

Clone the root volume

mount /dev/mapper/$vg1-$lv1 /mnt
rsync -rAa --one-file-system / /mnt/
umount /mnt

Reboot

Reboot (and pray). Validate the new LV is being mounted to /.

Delete old LVM

lvremove /dev/$vg0/*
vgremove $vg0
pvremove /dev/sda2 #Path to $pv0uuid

Cleanup

Optional, but recommended

• Finally delete old VG from /etc/lvm/archive or it will be searched for during every boot and cause delays
• update-grub
• update-initramfs -u

Notes

Tested on Ubuntu 18.04, other distributions might handle GRUB configuration differently (regarding UUIDs vs. device paths and update commands).

Obviously, stop all unnecessary services beforehand to minimize data loss.

There might be a way to avoid (most) data loss using systemd targets, custom scripts and LVM snapshots. One would have to sync the data from $lv0 to $lv1 again at the last possible moment before the reboot. Out of scope for this post...

If /boot is not a separate partition, double-check $grubcfg on the new root before rebooting again.

Answer 2

Not sure if this is going to help you but here is what I successfully tried to shrink root partition which is on LV:

1. Create root partition snapshot. Make sure it has some space allocated as you will be changing it.
2. fsck the snapshot.
3. Resize the snapshot (shrink it) and make sure to have some buffer between top of the filesystem and boundary of target size of LV - just to play it safe.
4. Merge the snapshot back to your rootfs LV.
5. Reboot. The merge is actually done at this point (probably during boot).
6. After a reboot - shrink rootfs volume and possibly resize rootfs again - so it fills any space to the top of LV.

There is one drawback: you WILL loose some data between steps 1 & 5 and you MAY have some inconsistencies (depending on the FS in use) in step 1 - as you make a snapshot on live filesystem.

It worked for me surprisingly well.

In ideal world the step 1 would be done during boot, before re/mounting rootfs rw, so the snapshot is clean. I haven't figured out how to do it from dracut - the tools I've tried to use didn't work there for me.