1

I added a 4TB disk formatted EXT4 (/dev/sdk1) to my lvm2 array using webmin. I then asked webmin to extend the filesystem to include that disk. It reported an error, which i didnt take a snapshot of. However it has added the disk to the logical volume, and the volume reports the new total size, however still has the old amount of free space. When i look at the pvdisplay, it shows the new disk as full! How can i correct this error. I've tried e2fsck, which reports the lv as clean. Below are my vgdisplay, pvdisplay and lvdisplay results:

name@Server:~$ sudo pvdisplay
--- Physical volume ---
PV Name               /dev/sdg1
VG Name               samba
PV Size               3.64 TiB / not usable 2.00 MiB
Allocatable           yes (but full)
PE Size               4.00 MiB
Total PE              953861
Free PE               0
Allocated PE          953861
PV UUID               VCtRWQ-c2u3-jIb3-AwGH-Wdpm-BNAz-U1QvzZ

--- Physical volume ---
PV Name               /dev/sdh1
VG Name               samba
PV Size               931.51 GiB / not usable 4.00 MiB
Allocatable           yes (but full)
PE Size               4.00 MiB
Total PE              238466
Free PE               0
Allocated PE          238466
PV UUID               0zHiKC-TTBn-L1TK-qmSo-y34W-G4mn-VCJXhA

--- Physical volume ---
PV Name               /dev/sdf1
VG Name               samba
PV Size               1.82 TiB / not usable 4.00 MiB
Allocatable           yes (but full)
PE Size               4.00 MiB
Total PE              476931
Free PE               0
Allocated PE          476931
PV UUID               5eOiZr-BdYk-Jk6c-X5fA-jMvf-7pdM-qvYI6G

--- Physical volume ---
PV Name               /dev/sde1
VG Name               samba
PV Size               3.64 TiB / not usable 2.00 MiB
Allocatable           yes (but full)
PE Size               4.00 MiB
Total PE              953861
Free PE               0
Allocated PE          953861
PV UUID               mrxqc3-qurh-eMER-dwRB-YTOB-fNf0-KPlfmg

--- Physical volume ---
PV Name               /dev/sdj1
VG Name               samba
PV Size               931.51 GiB / not usable 4.00 MiB
Allocatable           yes (but full)
PE Size               4.00 MiB
Total PE              238466
Free PE               0
Allocated PE          238466
PV UUID               SG2gP9-NLII-OsWd-tWGJ-Di8r-pDtT-rkcnTj

--- Physical volume ---
PV Name               /dev/sdm1
VG Name               samba
PV Size               1.82 TiB / not usable 4.00 MiB
Allocatable           yes (but full)
PE Size               4.00 MiB
Total PE              476931
Free PE               0
Allocated PE          476931
PV UUID               Ycqd4P-noXD-Nzgx-auAc-zzrS-F4KE-JSSMCd

--- Physical volume ---
PV Name               /dev/sdl1
VG Name               samba
PV Size               1.82 TiB / not usable 4.00 MiB
Allocatable           yes (but full)
PE Size               4.00 MiB
Total PE              476931
Free PE               0
Allocated PE          476931
PV UUID               XexUjT-PlM4-PQMu-9frH-dAVi-S8tC-hunsby

--- Physical volume ---
PV Name               /dev/sda1
VG Name               samba
PV Size               7.28 TiB / not usable 4.00 MiB
Allocatable           yes (but full)
PE Size               4.00 MiB
Total PE              1907720
Free PE               0
Allocated PE          1907720
PV UUID               ruDYpO-W9n8-XpMp-yLtg-3hfr-AOcX-DyAKlO

--- Physical volume ---
PV Name               /dev/sdk1
VG Name               samba
PV Size               3.64 TiB / not usable 2.00 MiB
Allocatable           yes (but full)
PE Size               4.00 MiB
Total PE              953861
Free PE               0
Allocated PE          953861
PV UUID               SEn3i2-8TcV-D2sm-6Ahe-pgQ4-hPzm-8wTDx1

name@Server:~$ sudo lvdisplay
--- Logical volume ---
LV Path                /dev/samba/media
LV Name                media
VG Name                samba
LV UUID                t3ItFs-3HWB-ZAZj-A0Ia-MDAe-sITn-tWaml1
LV Write Access        read/write
LV Creation host, time Server, 2017-08-03 17:36:54 +0100
LV Status              available
# open                 1
LV Size                25.47 TiB
Current LE             6677028
Segments               9
Allocation             inherit
Read ahead sectors     auto
- currently set to     256
Block device           252:0

name@Server:~$ sudo vgdisplay
--- Volume group ---
VG Name               samba
System ID             
Format                lvm2
Metadata Areas        9
Metadata Sequence No  43
VG Access             read/write
VG Status             resizable
MAX LV                0
Cur LV                1
Open LV               1
Max PV                0
Cur PV                9
Act PV                9
VG Size               25.47 TiB
PE Size               4.00 MiB
Total PE              6677028
Alloc PE / Size       6677028 / 25.47 TiB
Free  PE / Size       0 / 0   
VG UUID               S2A8Yl-CKR3-hhEf-DOoc-c564-e5DQ-GZ2vMb

Any and all help appreciated. I wondered if i could just remove the new disk - as it should have no data on it - and then use the force remove option and see if i get the old array back, but as it has already allocated the new pv to the lv, will i lose the lv?

Andrew
  • 13
  • 4

1 Answers1

1

"I added a 4TB disk formatted EXT4 (/dev/sdk1) to my lvm2" - That is your first problem. Add blank disks to LVM. You don't want to find old superblocks on disks when you make an LV that fills it. Filesystems go on top of LVs, not over PVs.

Also, if you're adding entire disks to LVM you don't have to put a partition table on them. It's much more simple and reliable to add raw disks, especially when replacing them.

In order for me to address your problem more precisely, I'll have to see what that error was during add. The information provided here will guide you as to where and how you can obtain that information. The most useful ones are going to be from lvmdump and the metadata archive in /etc/lvm/archive/.

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/Cluster_Logical_Volume_Manager/troubleshooting.html

Regarding your question about removing and re-adding, yes you should be able to do that if your VG has enough space in other PVs to unallocate the one you wish to remove. Removing a PV from a VG is as simple as instructing LVM to move all extents on that disk to other disks in the VG, then shrinking the VG by removing that disk from the array:

pvmove /dev/sdk1
vgreduce samba /dev/sdk1

UPDATE

Since reviewing the data provided, it looked like all LVM changes succeeded, with webmin having added the new disk to the existing "samba" VG - extending the VG by roughly 4TiB, and then extending the "media" LV into the remaining free physical extents provided by that new disk. What failed (effectively silently) was the resize2fs /dev/samba/media command, which was supposed to resize the existing EXT4 filesystem on top of the "media" LV after the LV itself was extended.

Spooler
  • 7,016
  • 16
  • 29
  • Thanks for your quick reply. Here are the contents of lvmdump and /etc/lvm/archive – Andrew Feb 23 '18 at 23:05
  • https://www.dropbox.com/sh/9k4v0d3dpzqqmfa/AABGBqqirf0bJ5kG8ABJ8vY_a?dl=0 – Andrew Feb 23 '18 at 23:15
  • edit - already posted above. – Andrew Feb 23 '18 at 23:19
  • its reporting sdk1 as full, so it probably wont let me do pvmove. I'll have a go. – Andrew Feb 23 '18 at 23:26
  • sudo pvmove /dev/sdk1 No extents available for allocation – Andrew Feb 23 '18 at 23:28
  • andrew@PenguinServer:~$ df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/samba-media 22T 18T 2.8T 87% /media/samba – Andrew Feb 23 '18 at 23:29
  • df -h reports it as 22TB, which is correct before i added the 4TB, but lvs -a reports it as 25.47TB – Andrew Feb 23 '18 at 23:31
  • sudo lvs -a LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert media samba -wi-ao---- 25.47t – Andrew Feb 23 '18 at 23:32
  • You extended the LV when you added the PV, and expanded your filesystem into the LV as well. Your disk is reporting as full because LVM has allocated all of the physical extents to your "samba" VG. After seeing that, I'm confident that this whole issue has been generated by confusion and the conflation of filesytsems with LVM block abstractions. Filesystems on top of LVs and LVM itself have very little to do with each other. – Spooler Feb 23 '18 at 23:35
  • Ok, but shouldn't the reported amount of freespace in the LV have increased by 4TB, rather than staying the same at 2.8TB - I was expecting to see 6+TB free space? Thats how it has happened before. – Andrew Feb 23 '18 at 23:37
  • Most importantly, how do i fix it so that i can use that 4TB of extra space? – Andrew Feb 23 '18 at 23:41
  • Ah, looks like your filesystem resize may have failed, leaving a larger LV than the EXT4 superblock knows about. Try expanding it to its max size and see if you still get an error. – Spooler Feb 23 '18 at 23:47
  • I retried the resize2fs command on /dev/mapper/samba-media, and it seems to be doing something now. Before it would not let me do it. Thankyou for all your help. Much appreciated. – Andrew Feb 23 '18 at 23:48