0

my Centos tells me that I have no disk space left:

[root@zff isos]# touch test
touch: cannot touch ‘test’: No space left on device

But df tells me that I have plenty of space left:

[root@zff isos]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root  3.6T  1.4T  2.2T  39% /
devtmpfs                  32G     0   32G   0% /dev
tmpfs                     32G     0   32G   0% /dev/shm
tmpfs                     32G   67M   32G   1% /run
tmpfs                     32G     0   32G   0% /sys/fs/cgroup
/dev/sda4                497M  165M  333M  34% /boot
/dev/sda1                496M  9.8M  487M   2% /boot/efi
/dev/mapper/centos-home   75G   33M   75G   1% /home
tmpfs                    6.3G  4.0K  6.3G   1% /run/user/0

I also did not run out of inodes:

[root@zff isos]# df -i
Filesystem                  Inodes IUsed      IFree IUse% Mounted on
/dev/mapper/centos-root 3791101952 82816 3791019136    1% /
devtmpfs                   8209805   563    8209242    1% /dev
tmpfs                      8211404     1    8211403    1% /dev/shm
tmpfs                      8211404  1246    8210158    1% /run
tmpfs                      8211404    13    8211391    1% /sys/fs/cgroup
/dev/sda4                   512000    30     511970    1% /boot
/dev/sda1                        0     0          0     - /boot/efi
/dev/mapper/centos-home   39321600    11   39321589    1% /home
tmpfs                      8211404     3    8211401    1% /run/user/0

I do not know what the problem is. Any help is appreciated!

Here are some aditional information:

[root@zff isos]# pvs
  PV         VG     Fmt  Attr PSize PFree
  /dev/sda5  centos lvm2 a--  3.63t    0

[root@zff isos]# vgs
  VG     #PV #LV #SN Attr   VSize VFree
  centos   1   3   0 wz--n- 3.63t    0

[root@zff isos]# lvs
  LV   VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home centos -wi-ao----  75.00g
  root centos -wi-ao----   3.53t
  swap centos -wi-ao---- <31.45g

[root@zff isos]# sudo find /proc/*/fd -ls 2>/dev/null | grep '(deleted)'
121429799    0 lrwx------   1 root     root           64 Nov  7 15:34 /proc/8762/fd/7 -> /tmp/ffi2lKXH0\ (deleted)
121429814    0 lrwx------   1 root     root           64 Nov  7 15:34 /proc/8939/fd/8 -> /tmp/ffimBw8GN\ (deleted)


[root@zff isos]# fdisk -l
WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.

Disk /dev/sda: 3999.7 GB, 3999688294400 bytes, 7811891200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: gpt



#         Start          End    Size  Type            Name
 1         2048      1026047    500M  EFI System      EFI System Partition
 2      1026048      1230847    100M  unknown         Basic data partition
 3      1230848      5425151      2G  Microsoft basic Basic data partition
 4      5425152      6449151    500M  Microsoft basic
 5      6449152   7811889151    3.6T  Linux LVM

Disk /dev/mapper/centos-swap: 33.8 GB, 33764147200 bytes, 65945600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/centos-root: 3882.1 GB, 3882088398848 bytes, 7582203904 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/centos-home: 80.5 GB, 80530636800 bytes, 157286400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes

[root@zff /]# du -sh
du: cannot access ‘./proc/45722/task/45722/fd/4’: No such file or directory
du: cannot access ‘./proc/45722/task/45722/fdinfo/4’: No such file or directory
du: cannot access ‘./proc/45722/fd/4’: No such file or directory
du: cannot access ‘./proc/45722/fdinfo/4’: No such file or directory
1.4T    .

EDIT

[root@zff /]# df -T
Filesystem              Type      1K-blocks       Used  Available Use% Mounted on
/dev/mapper/centos-root xfs      3791076352 1440862400 2350213952  39% /
devtmpfs                devtmpfs   32839220          0   32839220   0% /dev
tmpfs                   tmpfs      32845616          0   32845616   0% /dev/shm
tmpfs                   tmpfs      32845616      68484   32777132   1% /run
tmpfs                   tmpfs      32845616          0   32845616   0% /sys/fs/cgroup
/dev/sda4               xfs          508588     168456     340132  34% /boot
/dev/sda1               vfat         507904       9980     497924   2% /boot/efi
/dev/mapper/centos-home xfs        78604800      33000   78571800   1% /home
tmpfs                   tmpfs       6569124          4    6569120   1% /run/user/0

Edit 2

[root@zff /]# xfs_ncheck /dev/mapper/centos-root
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed.  Mount the filesystem to replay the log, and unmount it before
re-running xfs_ncheck.  If you are unable to mount the filesystem, then use
the xfs_repair -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.
must run blockget -n first
alive-and-well
  • 1
  • 4
  • Remote NFS? Do you have automount in place? Where is the isos directory located? – Jaroslav Kucera Nov 07 '17 at 15:17
  • I'm also curious about the filesystem at play here. If it's BTRFS, `df` doesn't do so well at reporting correctly (it doesn't account for multiple data replicas). If it's NFS, there are a large amount of space reporting related bugs in older verions and we don't know your kernel version. Either way, we probably need that data (filesystem and kernel verion). – Spooler Nov 07 '17 at 15:27
  • /dev/mapper/centos-root xfs 3791076352 1442412784 2348663568 39% / The file System is xfs – alive-and-well Nov 07 '17 at 15:30
  • 1
    Apparently my eyes don't work. Try running an online read-only XFS check to begin. Might have log issues. However, before you do that you might run `du -schx /`. That will give a more reliable result than `df`, and we can make a better decision on what to do with that output. – Spooler Nov 07 '17 at 15:38
  • [root@zff /]# du -schx / 1.4T / 1.4T total – alive-and-well Nov 07 '17 at 15:40
  • @SmallLoanOf1M I added the output of xfs_ncheck to my original question – alive-and-well Nov 07 '17 at 15:52
  • Don't suppose you could unmount > mount the filesystem, could you? That's very likely to solve your issue. – Spooler Nov 07 '17 at 16:06
  • I guess I can't. /dev/mapper/centos-root is mounted to /. I cant unmount that can I? – alive-and-well Nov 07 '17 at 16:09
  • lol no, you definitely can't. But is a reboot and filesystem check totally out of the question? – Spooler Nov 07 '17 at 16:10
  • Let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/68332/discussion-between-smallloanof1m-and-alive-and-well). – Spooler Nov 07 '17 at 16:14

2 Answers2

0

df is only aware of filesystem changes that have been committed within the kernel. For example, if you have deleted a large amount of in-use data, df won't reflect that until the process holding open those files is stopped and resources are released. This isn't the only condition that causes this, but it's pretty common.

df will generally reflect reality after a reboot, but that's a "big hammer" solution to this. You might be able to see what's wrong by searching for files which are opened by programs, yet which are also deleted. lsof is good for this in the following form:

lsof -nP +L1

You may also be having to account for the opposite condition, in that you might have data allocated that df hasn't picked up. The solution to that condition is largely based on what filesystem and kernel you're using. If you let me know what you're running, that would give me some direction on this portion of the answer.

Spooler
  • 7,016
  • 16
  • 29
-1

Restarting the server (init 6) solved my problem. Thanks for the help.

alive-and-well
  • 1
  • 4