8

I am currently dealing with a multiple disk failure on a Linux LVM Volume Group that is backed up by a RAID-5 md device. One disk has been taken out completely and another one is showing a limited number of corrupt sectors, due to what seems to have been a misbehaving power supply.

The problem is that once an I/O error hits, md takes the array down, since it does not have enough devices for it to be operational. Where md the only one involved, I could mdadm --stop the array and then recreate it to get all devices active again.

Unfortunately, the array is a PV in an LVM volume group and I cannot seem to get the kernel to release it. vgchange -an does not seem to do anything, bar spew out a couple of I/O errors.

I am obviously missing something, but how in the name of -insert-favorite-deity- do I get LVM to release the underlying PV without rebooting the server?

thkala
  • 1,210
  • 1
  • 13
  • 19

2 Answers2

6

I would say that you are in a pretty bad shape. If you are simply desperate to somehow recover some of your data and don't care much about the LVM (whose metadata is probably already corrupt, judging by the IO errors from vgchange -an), I would recommend going low level. Remember, LVM is just a wrapper around the kernel device mapper, so you can use dmsetup to manipulate your logical volumes. dmsetup table will give you a list of currently active logical disks so it is a good idea to back up its output in case you cannot even access your LVM later on. Then you can try stopping the devices you want to stop with dmsetup remove or even dmsetup remove_all. But make sure they are unmounted first.

And of course, copy as much data as possible to a safe location.

chutz
  • 7,569
  • 1
  • 28
  • 57
  • Actually the LVM is fine - as long as `md` does not decide to throw out one of the disks. When that happens every single access to the system results in an I/O error. BTW `dmsetup remove_all --force` does not seem to be able overcome the obstacle of LVM being set in its ways... – thkala Sep 17 '12 at 18:41
4

When vgchange -an does not seem to do anything (on an Ubuntu system) then consider blaming udev rules after reading this bug report (I found it Googling for "dmsetup remove lvm vgchange"): https://bugs.launchpad.net/ubuntu/+source/lvm2/+bug/1088081

The workaround is as follows (thanks to Martin): 

sudo umount /mnt # asume the drive in question is mounted under /mnt  
sudo service udev stop
sudo lvchange -a n <LV-name>
sudo cryptsetup luksClose <LUKS-devicename>  
sudo service udev start
slm
  • 7,355
  • 16
  • 54
  • 72
smalcolm
  • 41
  • 1