35

We had a disk fail in a server and replaced it before removing the drive from LVM.

The server has 4 physical drives (PV's), each with it's own volume group (VG). Each VG has 2 or more logical volumes (LV's.) Now LVM is complaining about the missing drive. So we have a VG (vg04) with two LV's that have become orphans than we need to clear out of the system.

The problem is every time we run any LVM command we get these 'read failed' errors:

# lvscan
  /dev/vg04/swap: read failed after 0 of 4096 at 4294901760: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4294959104: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4096: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903864832: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903922176: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 4096: Input/output error

# vgreduce vg04 --removemissing --force
  /dev/vg04/swap: read failed after 0 of 4096 at 4294901760: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4294959104: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4096: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903864832: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903922176: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 4096: Input/output error
  Volume group "vg04" not found

# vgchange -a n /dev/vg04
  /dev/vg04/swap: read failed after 0 of 4096 at 4294901760: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4294959104: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4096: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903864832: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903922176: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 4096: Input/output error
  Volume group "vg04" not found

# lvchange -a n /dev/vg04/vz
  /dev/vg04/swap: read failed after 0 of 4096 at 4294901760: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4294959104: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4096: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903864832: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903922176: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 4096: Input/output error
  Volume group "vg04" not found
  Skipping volume group vg04

The missing VG and LV's are not important, we just want to remove them.

As you can see we've tried all the suggestions made, so far without luck.

Output from 'lvm dumpconfig' can be checked at http://pastebin.com/MHiBzrLJ

Dokbua
  • 1,052
  • 1
  • 9
  • 18

3 Answers3

53

The solution was to run dmsetup, in this case the two commands

dmsetup remove vg04-vz
dmsetup remove vg04-swap

Before doing this, I checked with the command 'dmsetup info' that the 'open count' for both LV's were zero.

WARNING: dmsetup can wreck serious havoc with your disks so anyone using this information in the future please make sure you read the man page.

slm
  • 7,355
  • 16
  • 54
  • 72
Dokbua
  • 1,052
  • 1
  • 9
  • 18
  • 3
    That did the trick. It's too bad LVM is such a leaky abstraction, but I guess this is as good as it gets. – Navin Mar 05 '16 at 14:45
  • 1
    Good call with that warning. If I understand correctly, the similarly named `remove_all --force --deferred --retry` would blow away every LVM managed drive within reach. It would keep trying and not give up, like the Terminator. I am super curious abut why a footgun like this was warranted. – Dan Ross Mar 08 '17 at 00:12
  • There is also a single device "terminate with extreme prejudice" (and/ or vengeance, if the device kept you busy whole night): `# dmsetup --force --retry --deferred remove -` This is especially true, if for instance a iscsi device was hanging and you removed it with `# iscsiadm -m node -U all -T ` – AdamKalisz Jul 20 '17 at 14:32
10

I experienced similar after removing a VMware disk /dev/sdb.

Use lsscsi to determine which device is being used (you may need to install the program) then run lsscsi:

# lsscsi
[1:0:0:0]    cd/dvd  NECVMWar VMware IDE CDR10 1.00  /dev/sr0
[2:0:0:0]    disk    VMware   Virtual disk     1.0   /dev/sda
[2:0:1:0]    disk    VMware   Virtual disk     1.0   /dev/sdb <== the removed physical disk
[2:0:2:0]    disk    VMware   Virtual disk     1.0   /dev/sdc

Now write the delete file with

# echo 1 > /sys/class/scsi_device/2\:0\:1\:0/device/delete

The device is immediately gone.

Flup
  • 7,688
  • 1
  • 31
  • 43
Effendi
  • 101
  • 1
  • 2
  • 1
    Thank you!! This worked perfectly for me for getting rid of errors like "/dev/sdb: read failed after 0 of 4096 at 0: Input/output error" after removing a stale vg1-xyz via dmsetup. – cat pants Jun 26 '13 at 22:16
  • 1
    Only this one works for me. You made my day. – odiszapc Jul 13 '14 at 09:23
2

vgchange -a n /dev/vg04

If you have logical volumes still active you may have to

lvchange -a n /dev/vg04/swap
lvchange -a n dev/vg04/vz

etc.

HopelessN00b
  • 53,385
  • 32
  • 133
  • 208
Rob Shinn
  • 79
  • 1