7

I cannot shrink btrfs filesystem although there is still data and metadata space left :

$ sudo btrfs filesystem resize -11G /home;echo $?
Resize '/home' of '-11G'
ERROR: unable to resize '/home': No space left on device
1

Here are some btrfs filesystem info about /home :

$ sudo btrfs filesystem df /home | column -t
Data,           single:  total=92.01GiB,   used=80.68GiB
System,         DUP:     total=8.00MiB,    used=16.00KiB
System,         single:  total=4.00MiB,    used=0.00B
Metadata,       DUP:     total=1.00GiB,    used=631.41MiB
Metadata,       single:  total=8.00MiB,    used=0.00B
GlobalReserve,  single:  total=224.00MiB,  used=0.00B
$ sudo btrfs filesystem show /home
Label: none  uuid: c7ee56a8-ef45-46c8-86d1-13879201a1e7
    Total devices 1 FS bytes used 81.30GiB
    devid    1 size 100.00GiB used 94.04GiB path /dev/mapper/home_VG-home

$ sudo btrfs filesystem usage -T /home
Overall:
    Device size:         100.00GiB
    Device allocated:         94.04GiB
    Device unallocated:        5.96GiB
    Device missing:          0.00B
    Used:             81.91GiB
    Free (estimated):         17.29GiB  (min: 14.31GiB)
    Data ratio:               1.00
    Metadata ratio:           1.99
    Global reserve:      224.00MiB  (used: 0.00B)

             Data     Metadata Metadata  System  System              
Id Path      single   single   DUP       single  DUP      Unallocated
-- --------- -------- -------- --------- ------- -------- -----------
 1 /dev/dm-0 92.01GiB  8.00MiB   2.00GiB 4.00MiB 16.00MiB     5.96GiB
-- --------- -------- -------- --------- ------- -------- -----------
   Total     92.01GiB  8.00MiB   1.00GiB 4.00MiB  8.00MiB     5.96GiB
   Used      80.68GiB    0.00B 631.41MiB   0.00B 16.00KiB            

and here the output of dmesg :

$ dmesg | tail -11
[44202.411949] BTRFS info (device dm-0): new size for /dev/dm-0 is 97706311680
[44202.412156] BTRFS info (device dm-0): relocating block group 120288444416 flags 1
[44208.119721] BTRFS info (device dm-0): relocating block group 119214702592 flags 1
[44211.611669] BTRFS info (device dm-0): relocating block group 118140960768 flags 1
[44212.495603] BTRFS info (device dm-0): relocating block group 117067218944 flags 1
[44213.006830] BTRFS info (device dm-0): relocating block group 95592382464 flags 1
[44216.613870] BTRFS info (device dm-0): relocating block group 120288444416 flags 1
[44222.780073] BTRFS info (device dm-0): relocating block group 119214702592 flags 1
[44225.843279] BTRFS info (device dm-0): relocating block group 118140960768 flags 1
[44226.575236] BTRFS info (device dm-0): relocating block group 117067218944 flags 1
[44226.930918] BTRFS info (device dm-0): relocating block group 95592382464 flags 1

EDIT1 : The btrfs balance failed :

$ sudo btrfs balance start /home
ERROR: error during balancing '/home': No space left on device
There may be more info in syslog - try dmesg | tail

There nothing in dmesg | tail about it.

EDIT2 : I had to do the following to be able to start the btrfs balance :

$ sudo btrfs balance start -musage=0 -dusage=0 -v /home
Dumping filters: flags 0x7, state 0x0, force is off
  METADATA (flags 0x2): balancing, usage=0
  SYSTEM (flags 0x2): balancing, usage=0
  DATA (flags 0x2): balancing, usage=0
Done, had to relocate 0 out of 95 chunks

EDIT3 : The btrfs balance has ran for 68 minutes and then failed :

$ time sudo btrfs balance start -v /home 
Dumping filters: flags 0x7, state 0x0, force is off
  DATA (flags 0x0): balancing
  METADATA (flags 0x0): balancing
  SYSTEM (flags 0x0): balancing
ERROR: error during balancing '/home': Input/output error
There may be more info in syslog - try dmesg | tail

real    68m10.221s
user    0m0.008s
sys     4m20.236s

Here is what dmesg shows :

[74421.794756] ata2.00: exception Emask 0x0 SAct 0xc00 SErr 0x0 action 0x0
[74421.794766] ata2.00: irq_stat 0x40000001
[74421.794773] ata2.00: failed command: READ FPDMA QUEUED
[74421.794783] ata2.00: cmd 60/08:50:48:96:f8/00:00:25:00:00/40 tag 10 ncq 4096 in
[74421.794783]          res 41/40:08:48:96:f8/00:00:25:00:00/40 Emask 0x409 (media error) <F>
[74421.794788] ata2.00: status: { DRDY ERR }
[74421.794791] ata2.00: error: { UNC }
[74421.794794] ata2.00: failed command: READ FPDMA QUEUED
[74421.794802] ata2.00: cmd 60/10:58:40:af:ed/00:00:20:00:00/40 tag 11 ncq 8192 in
[74421.794802]          res 41/40:58:48:96:f8/00:00:25:00:00/40 Emask 0x9 (media error)
[74421.794806] ata2.00: status: { DRDY ERR }
[74421.794809] ata2.00: error: { UNC }
[74421.798253] ata2.00: configured for UDMA/100
[74421.798303] sd 1:0:0:0: [sdb] tag#10 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[74421.798315] sd 1:0:0:0: [sdb] tag#10 Sense Key : Medium Error [current] [descriptor] 
[74421.798326] sd 1:0:0:0: [sdb] tag#10 Add. Sense: Unrecovered read error - auto reallocate failed
[74421.798337] sd 1:0:0:0: [sdb] tag#10 CDB: Read(10) 28 00 25 f8 96 48 00 00 08 00
[74421.798344] blk_update_request: I/O error, dev sdb, sector 637048392
[74421.798366] BTRFS error (device dm-0): bdev /dev/dm-0 errs: wr 38, rd 451, flush 0, corrupt 0, gen 0
[74421.798425] sd 1:0:0:0: [sdb] tag#11 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[74421.798435] sd 1:0:0:0: [sdb] tag#11 Sense Key : Medium Error [current] [descriptor] 
[74421.798444] sd 1:0:0:0: [sdb] tag#11 Add. Sense: Unrecovered read error - auto reallocate failed
[74421.798453] sd 1:0:0:0: [sdb] tag#11 CDB: Read(10) 28 00 20 ed af 40 00 00 10 00
[74421.798459] blk_update_request: I/O error, dev sdb, sector 552447808
[74421.798523] ata2: EH complete

EDIT 4 : I'm actually using /dev/sdb :

$ sudo smartctl -a /dev/sdb
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.4.0-143-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Toshiba 2.5" HDD MQ01ABD...
Device Model:     TOSHIBA MQ01ABD100
Serial Number:    84EWT2U5T
LU WWN Device Id: 5 000039 5b1f852cb
Firmware Version: AX1P4M
User Capacity:    1 000 204 886 016 bytes [1,00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 2.6, 3.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Mon Apr  1 23:34:41 2019 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:        (  120) seconds.
Offline data collection
capabilities:            (0x5b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:    (   2) minutes.
Extended self-test routine
recommended polling time:    ( 243) minutes.
SCT capabilities:          (0x003d) SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   050    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   100   100   050    Pre-fail  Offline      -       0
  3 Spin_Up_Time            0x0027   100   100   001    Pre-fail  Always       -       1735
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       5639
  5 Reallocated_Sector_Ct   0x0033   100   100   050    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   050    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   100   100   050    Pre-fail  Offline      -       0
  9 Power_On_Hours          0x0032   080   080   000    Old_age   Always       -       8259
 10 Spin_Retry_Count        0x0033   212   100   030    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       5623
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       563
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       203
193 Load_Cycle_Count        0x0032   099   099   000    Old_age   Always       -       17892
194 Temperature_Celsius     0x0022   100   100   000    Old_age   Always       -       23 (Min/Max 10/46)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       9200
198 Offline_Uncorrectable   0x0030   001   001   000    Old_age   Offline      -       255
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
220 Disk_Shift              0x0002   100   100   000    Old_age   Always       -       0
222 Loaded_Hours            0x0032   080   080   000    Old_age   Always       -       8117
223 Load_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
224 Load_Friction           0x0022   100   100   000    Old_age   Always       -       0
226 Load-in_Time            0x0026   100   100   000    Old_age   Always       -       177
240 Head_Flying_Hours       0x0001   100   100   001    Pre-fail  Offline      -       0

SMART Error Log Version: 1
ATA Error Count: 1029 (device log contains only the most recent five errors)
    CR = Command Register [HEX]
    FR = Features Register [HEX]
    SC = Sector Count Register [HEX]
    SN = Sector Number Register [HEX]
    CL = Cylinder Low Register [HEX]
    CH = Cylinder High Register [HEX]
    DH = Device/Head Register [HEX]
    DC = Device Command Register [HEX]
    ER = Error register [HEX]
    ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 1029 occurred at disk power-on lifetime: 8257 hours (344 days + 1 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 41 50 48 96 f8 40  Error: UNC at LBA = 0x00f89648 = 16291400

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 10 58 40 af ed 40 00      03:13:20.172  READ FPDMA QUEUED
  60 08 50 48 96 f8 40 00      03:13:16.469  READ FPDMA QUEUED
  60 08 48 40 96 f8 40 00      03:13:16.469  READ FPDMA QUEUED
  60 08 40 38 96 f8 40 00      03:13:16.469  READ FPDMA QUEUED
  60 08 38 30 96 f8 40 00      03:13:16.469  READ FPDMA QUEUED

Error 1028 occurred at disk power-on lifetime: 8257 hours (344 days + 1 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 41 70 48 96 f8 40  Error: UNC at LBA = 0x00f89648 = 16291400

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 10 70 78 90 f8 40 00      03:13:11.731  READ FPDMA QUEUED
  60 d0 68 a8 89 f8 40 00      03:13:11.731  READ FPDMA QUEUED
  61 e0 60 60 aa 0b 40 00      03:13:11.727  WRITE FPDMA QUEUED
  61 00 58 60 a2 0b 40 00      03:13:11.723  WRITE FPDMA QUEUED
  61 00 50 60 9a 0b 40 00      03:13:11.625  WRITE FPDMA QUEUED

Error 1027 occurred at disk power-on lifetime: 8133 hours (338 days + 21 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 41 c0 f8 bd 51 40  Error: UNC at LBA = 0x0051bdf8 = 5357048

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 58 e0 70 fa 40 40 00      00:18:59.971  READ FPDMA QUEUED
  61 08 d8 d8 45 2b 40 00      00:18:59.971  WRITE FPDMA QUEUED
  61 08 d0 d0 78 6b 40 00      00:18:59.971  WRITE FPDMA QUEUED
  61 08 c8 18 42 2b 40 00      00:18:59.971  WRITE FPDMA QUEUED
  60 08 c0 f8 bd 51 40 00      00:18:59.971  READ FPDMA QUEUED

Error 1026 occurred at disk power-on lifetime: 8133 hours (338 days + 21 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 41 00 f8 bd 51 40  Error: WP at LBA = 0x0051bdf8 = 5357048

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  61 38 10 28 5f 6b 40 00      00:18:55.963  WRITE FPDMA QUEUED
  61 08 08 68 85 6f 40 00      00:18:55.963  WRITE FPDMA QUEUED
  60 00 00 f0 bd 51 40 00      00:18:55.946  READ FPDMA QUEUED
  60 00 f0 80 75 56 40 00      00:18:55.944  READ FPDMA QUEUED
  60 00 e8 80 73 56 40 00      00:18:55.930  READ FPDMA QUEUED

Error 1025 occurred at disk power-on lifetime: 8119 hours (338 days + 7 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 41 b8 f8 7f 48 40  Error: UNC at LBA = 0x00487ff8 = 4751352

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 08 b8 f8 7f 48 40 00      01:10:35.049  READ FPDMA QUEUED
  ea 00 00 00 00 00 a0 00      01:10:35.017  FLUSH CACHE EXT
  61 08 98 88 4b cb 40 00      01:10:35.017  WRITE FPDMA QUEUED
  61 08 70 98 c1 0c 40 00      01:10:35.017  WRITE FPDMA QUEUED
  61 08 60 a0 45 cb 40 00      01:10:35.017  WRITE FPDMA QUEUED

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      6780         -
# 2  Short offline       Completed without error       00%         1         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

$ echo $?
64

and dmesg reports 2 bad sectors during the last btrfs balance operation :

$ dmesg | grep I/O.error.*sector
[74421.798344] blk_update_request: I/O error, dev sdb, sector 637048392
[74421.798459] blk_update_request: I/O error, dev sdb, sector 552447808

Remapped those the bad sectors :

$ dmesg | grep I/O.error.*sector | awk '/sector/{print "sudo hdparm --yes-i-know-what-i-am-doing --repair-sector "$NF" /dev/sdb"}' | sh -x
+ sudo hdparm --yes-i-know-what-i-am-doing --repair-sector 637048392 /dev/sdb

/dev/sdb:
re-writing sector 637048392: succeeded
+ sudo hdparm --yes-i-know-what-i-am-doing --repair-sector 552447808 /dev/sdb

/dev/sdb:
re-writing sector 552447808: succeeded

EDIT 5 : It seems this command was enough to have more than 11G unallocated :

$ sudo btrfs balance start -musage=0 -dusage=0 -v /home
Dumping filters: flags 0x7, state 0x0, force is off
  METADATA (flags 0x2): balancing, usage=0
  SYSTEM (flags 0x2): balancing, usage=0
  DATA (flags 0x2): balancing, usage=0
Done, had to relocate 0 out of 95 chunks

The btrfs filesystem resize succeeded. (I'm sorry, I've lost the output of the btrfs filesystem resize)

SebMa
  • 275
  • 1
  • 9
  • You should run an extended test. It has been around 2000 hours since the last one and you’ve got errors logged in SMART. – David Apr 01 '19 at 21:41
  • @David Will this extended test mark the bad clusters as bad so as to prevent Linux from using them ? – SebMa Apr 01 '19 at 21:43
  • depends what type of drive you have. Many will remap sectors internally, until they run out of spare sectors. – David Apr 01 '19 at 21:45
  • @David OK, I'll run it tomorrow because I'm feeling completely exhausted. Thanks for your help, man. See ya ! – SebMa Apr 01 '19 at 21:50
  • Refer to the line "Offline_Uncorrectable" and its value of "255". This is an indicator that your drive has experienced errors that it did not have reserved space to remap internally or could could not process in order to remap in the first place, 255 times. This is pretty damning, and in most modern drives it's typically a symptom of something more serious, such as a read element failure. I would advise not storing data on this disk, even if that SMART test passes. If you don't already have a backup, now is the time to start doing that. – Spooler Apr 01 '19 at 21:58
  • And if it's not failing too badly, you might be able to use BTRFS to move your data to another drive, either with the BTRFS snapshot and send feature, or by adding a different disk to the BTRFS volume and deleting this bad one. Backups first, though. Those BTRFS operations may fail in the presence of a failing disk. Hopefully it's not "that bad yet". – Spooler Apr 01 '19 at 22:00
  • @Spooler OK, I'll look into it tomorrow, thanks. – SebMa Apr 01 '19 at 22:07
  • This: `blk_update_request: I/O error, dev sdb, sector 637048392` is very clearly a hardware problem. Not the btrfs is the problem, the problem is that your hard disk has bad sectors. I casted a reopen vote. – peterh Apr 12 '19 at 15:49

1 Answers1

6

You're requesting the volume to shrink by 11GB, yet you only have about 6GB unallocated.

You can more efficiently use allocated extents by rebalancing the volume. Executing a command similar to btrfs balance start /home will start that process, and it may take some time to complete.

But I don't know if that will free up enough for a large amount of shrinkage.

Spooler
  • 7,016
  • 16
  • 29
  • I though I add `14.31GiB` free according to this : `Free (estimated): 17.29GiB (min: 14.31GiB)` – SebMa Apr 01 '19 at 02:16
  • 1
    But you're requesting a volume change, which deals with allocated extents rather than how they're used. You could have almost the entire volume "free", but if the whole thing is "allocated" you can't shrink it. Running a rebalance will reallocate used data in a more efficient way. – Spooler Apr 01 '19 at 02:34
  • Ok, thanks bro. I'll do this tomorrow 'cause it's 20 to 4 (A.M) in France. – SebMa Apr 01 '19 at 02:41
  • Can you please have a look at my EDIT3, I don't understand those errors in `dmesg` ? – SebMa Apr 01 '19 at 13:41
  • Dang. Looks like you probably have a bad disk. Is this a physical node you're working on with a real disk? If so, run `smartctl -a /dev/sda` for a full health report on that disk. The data can be hard to read, post it here if it's not clear that the disk is going bad. You can get that smartctl command from the "smartmontools" package. – Spooler Apr 01 '19 at 14:39
  • The output of `smartctl` is in EDIT 4. Even if the disk is bad, can't one just mark the bad clusters as bad for Linux not to use them ? – SebMa Apr 01 '19 at 21:38