(LVM) Moving data to a different physical volume within the same logical volume

Question

I have a logical volume called storage. It contains two ~1TB physical volumes: /dev/sdc and /dev/sdd. smartctl -a /dev/sdd informs me that /dev/sdd is failing. I can still read data off it and I have just backed up all contents off the LV.

I have about 200GB of actual data — little enough that one of the PVs can hold all of it. I want to remove /dev/sdd from the logical volume and transfer all its data to other PVs (/dev/sdc) on this LV.

pvmove fails because all extents are allocated.

# pvmove /dev/sdd /dev/sdc
  WARNING: No free extents on physical volume "/dev/sdc".
  No specified PVs have space available.

Problem: How do I ensure there is no "real" data on /dev/sdd and safely remove it?

I've elided the other LVM organization not relevant to this question from the output below.

# lvdisplay --maps
  --- Logical volume ---
  LV Path                /dev/spinny/storage
  LV Name                storage
  VG Name                spinny
  LV UUID                LeJEIR-0Zvf-YBLD-GrCT-Kvv0-mcvz-4a66Ib
  LV Write Access        read/write
  LV Creation host, time archiso, 2020-01-01 12:32:05 -0500
  LV Status              available
  # open                 0
  LV Size                <1.82 TiB
  Current LE             476934
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:1

  --- Segments ---
  Logical extents 0 to 238466:
    Type        linear
    Physical volume /dev/sdc
    Physical extents    0 to 238466

  Logical extents 238467 to 476933:
    Type        linear
    Physical volume /dev/sdd
    Physical extents    0 to 238466

# pvdisplay --maps
  --- Physical volume ---
  PV Name               /dev/sdc
  VG Name               spinny
  PV Size               931.51 GiB / not usable 1.71 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              238467
  Free PE               0
  Allocated PE          238467
  PV UUID               fMOJ38-TiZZ-DspR-hOy3-4aGn-lKXo-4mYSrU

  --- Physical Segments ---
  Physical extent 0 to 238466:
    Logical volume  /dev/spinny/storage
    Logical extents 0 to 238466

  --- Physical volume ---
  PV Name               /dev/sdd
  VG Name               spinny
  PV Size               931.51 GiB / not usable 1.71 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              238467
  Free PE               0
  Allocated PE          238467
  PV UUID               XJ2D5p-l3iP-VBlR-peXB-up1y-ZpNh-UDkcfB

  --- Physical Segments ---
  Physical extent 0 to 238466:
    Logical volume  /dev/spinny/storage
    Logical extents 238467 to 476933

# blkid
/dev/sdc: UUID="fMOJ38-TiZZ-DspR-hOy3-4aGn-lKXo-4mYSrU" TYPE="LVM2_member"
/dev/sdd: UUID="XJ2D5p-l3iP-VBlR-peXB-up1y-ZpNh-UDkcfB" TYPE="LVM2_member"
/dev/mapper/spinny-storage: UUID="108a53d4-dc5a-48b7-971f-b62c617fdb15" BLOCK_SIZE="4096" TYPE="ext4"

# smartctl -a /dev/sdd
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.6.11-arch1-1] (local build)
# ... clipped ...
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   001   001   051    Pre-fail  Always   FAILING_NOW 14946
  3 Spin_Up_Time            0x0027   170   169   021    Pre-fail  Always       -       2466
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       80
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   095   095   000    Old_age   Always       -       3728
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       80
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       19
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       66
194 Temperature_Celsius     0x0022   103   096   000    Old_age   Always       -       40
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   183   183   000    Old_age   Always       -       2919
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: unknown failure    90%      3680         -
# 2  Extended offline    Completed: read failure       90%      3663         2064

score 1 · Answer 1 · answered May 16 '20 at 14:50

Your "storage" LV is 2TB in size. You cannot remove half of the physical space without damaging it.

What you can do, is shrink the size of the filesystem, then resize the LV, and then you can safely remove the PV from the volume group.

Unmount "storage": umount /dev/spinny/storage
Shrink the filesystem so it can fit on sdc alone. As you only have 200G you can go down to 900G for safety. Depending on your filesystem type, the tool you need is different, and if your filesystem does not support shrinking you're out of luck. If you are using ext4, you might have luck with: fsck -C0 -f /dev/spinny/storage && resize2fs /dev/spinny/storage 800G.
Shrink the logical volume to be no smaller than the filesystem size (above), and no bigger than the size of sdc.: lvresize -L900G spinny/storage
Since sdd is the second drive, chances are by now you'll have no extends on it. pvs at this point should show that there is no data on sdd and you can safely take it out with vgreduce spinny /dev/sdd.

You may now grow the LV and filesystem to fill sdc if you prefer:

lvresize -l 100%PV spinny/storage
resize2fs /dev/spinny/storage

To be certain the PV is empty, `pvmove /dev/sdd` before `vgreduce`. If the LV extent type were striped and not linear, or if there were other LVs on it, there would be extents all over that PV. — John Mahowald, May 16 '20 at 16:37
Unfortunately I couldn't get `resize2fs` to work due to I/O errors even after going through with `e2fsck`. The ultimate solution for me was to rebuild the VG and restore data from backup. — Emily Horsman, May 16 '20 at 22:55

(LVM) Moving data to a different physical volume within the same logical volume

1 Answers1