5

The other day I had a proper brain fart moment while expanding a disk on a Linux guest under Vmware. I stretched the Vmware disk file to the desired size and then I did what I usually do on Linux guests without LVM: I deleted the LVM partition and recreated it, starting in the same spot as the old one, but extended to the new size of the disk. (Which will be followed by fsck and resize2fs.)

And then I realized that LVM doesn't behave the same way as ext2/3/4 on raw partitions... After restoring the Linux guest from the most recent backup (taken only five hours earlier, luckily) I'm now curious on how I could have recovered from the following scenario. It's after all virtually guaranteed that I'll be a dumb ass in the future as well.

Virtual Linux guest with one disk, partitioned into one /boot (primary) partition (/dev/sda1) of 256MB, and the rest in a logical, extended partition (/dev/sda5).

/dev/sda5 is then setup as a physical volume with pvcreate, and one volume group (vgroup00) created on top of it with the usual vgcreate command. vgroup00 is then split into two logical volumes root and swap, which are used for / and swap, logically. / is an ext4 file system.

Since I had backups of the broken guest I was able to recreate the volume group with vgcfgrestore from the backup LVM setup found under /etc/lvm/backup, with the same UUID for the physical volume and all that. After running this I had two logical volumes with the same size as earlier, with 4GB free space where I had stretched the disk.

However, when I tried to run "fsck /dev/mapper/vgroup00-root" it complained about a broken superblock. I tried to locate backup superblocks by running "mke2fs -n /dev/mapper/vgroup00-root" but none of those worked either. Then I tried to run TestDisk but when I asked it to find superblocks it only gave an error about not being able to open the file system due to a broken file system.

So, with the default allocation policy for LVM2 in Ubuntu Server 10.04 64-bit, is it possible that the logical volumes are allocated from the end of the volume group? That would definitely explain why the restored logical volumes didn't contain the expected data. Could I have recovered by recreating /dev/sda5 with exactly the same size and disk position as earlier? Are there any other tools I could have used to find and recover the file system? (And clearly, the question is not whether or not I should have done this in a different way from the start, I know that. This is a question about what to do when shit has already hit the fan.)

Vegar Nilsen
  • 163
  • 2
  • 5
  • to recover a deleted LVM volume I'd guess you'd need to go low level and re-create the volume using device mapper, not lvm. Also, you can use `lvdisplay --maps` to see LE to PE mapping (the exact and most important thing you need to recreate). – Hubert Kario Oct 09 '11 at 23:39
  • I would need to have run "lvdisplay --maps" before deleting the volumes, right? (When I run "lvdisplay --maps" on the new server, it shows the LE to PE are mapped the way I would expect, with the first logical volume starting at PE 0.) – Vegar Nilsen Oct 10 '11 at 09:05
  • yes, you need to have the mapping from before deletion. – Hubert Kario Oct 12 '11 at 12:50
  • 1
    try inspecting the block device with `sudo less -f /dev/sda5` which should show you all recent changes to lvm metadata. This may be more accurate than what vgcfgrestore finds in /etc/lvm especially when the disk is corrupted. Try extracting the right one by timestamp to a file and run vgcfgrestore from the file. – Martian Nov 19 '11 at 19:50

3 Answers3

3

Every time you perform an operation with LVM, by default, the previous metadata is archived in /etc/lvm/archive. You can use vgcfgrestore to restore it, or grab the extends by hand (harder, but lvcreate(8) should cover it).

Edit:

And to make it as easy as possible, I should add that you can find the last backup before your destructive operation by looking at descriptions:

# grep description /etc/lvm/archive/vg01_*
/etc/lvm/archive/vg01_00001.vg:description = "Created before executing 'lvremove -f /dev/vg01/foo'"
/etc/lvm/archive/vg01_00002.vg:description = "Created before executing 'lvremove -f /dev/vg01/bar'"
/etc/lvm/archive/vg01_00003.vg:description = "Created before executing 'lvremove -f /dev/vg01/baz'"

Edit:

The normal allocation policy (default one) will allocate a stripe from the first free PE when there is enough room to do so. If you want to confirm where the LV was allocated, you can look in the archive files, those are perfectly readable by humans.

Pierre Carrier
  • 2,607
  • 17
  • 28
0

Thanks to your existing backup of the disk image before the deletion but after creation, you can simply copy the LVM metadata back into place:

  1. Boot into a VM with (a cp --reflink copy of) the backup as its disk and a small second disk to keep a few MiB of data in step 3.

  2. Check the payload offset in the PV. Typically this is 1 MiB.

  3. Copy the LVM metadata from before the payload to a file on the extra storage, e.g. dd bs=1024k count=1 if=/dev/sda5 of=/mnt/pv-meta.img.

  4. Boot rescue system on a VM with the current disk and the extra storage attached. (Don't boot the VM with current and backup disk attached at the same time as LVM doesn't like seeing the same PV/VG/LV UUIDs twice.)

  5. Restore the PV metadata to /dev/sda5

  6. Reboot

If you messed with the size of /dev/sda5 after resizing the disk bring /dev/sda5 back to its previous size before step 5 in case this confuses LVM. The proper way to add the new space is to create a new PV as /dev/sda6 and add it to the VG.

Joachim Wagner
  • 153
  • 8
0

There aren't any really good recovery options, and there are no tools that support this to my knowledge. However, see the data recovery section of LVM dangers and caveats for some articles on manual recovery.

Generally it's best to do a raw image backup of the broken volume(s) or underlying disks, and do the data recovery on the backup versions, so that you have a way of retrying the recovery.

The above answer also has a section on resizing LVM volumes - expanding them is reasonably safe, and it's usually better to use lvresize instead of deleting and recreating.

On a related point: since you are using VMware, you should also take care that hard disk write cache flushes (write barriers) propagate correctly from the guest Linux kernel through the hypervisor and any host OS. It's also important that write barriers are set up in the guest FS and guest kernel, which should be 2.6.33 or higher.

Community
  • 1
RichVel
  • 3,524
  • 1
  • 17
  • 23