0

OS: Oracle Linux 7.5

df -h keeps showing that / is 100% full but searching on / with du -sh * is not showing where that utilization could possibly be, any help and suggestions will be appreciated. I also can't install lsof coz of the space issue.

[root@ /]# df -h
Filesystem                                    Size  Used Avail Use% Mounted on
devtmpfs                                       32G     0   32G   0% /dev
tmpfs                                          32G   14M   32G   1% /dev/shm
tmpfs                                          32G  1.3G   31G   4% /run
tmpfs                                          32G     0   32G   0% /sys/fs/cgroup
/dev/sda3                                      50G   48G     0 100% /
/dev/sda1                                     976M  208M  702M  23% /boot
/dev/sda2                                     469G   75M  445G   1% /home
//hidden                                       11T  8.5T  2.4T  79% /mnt/buffalo
/dev/sdb                                      2.7T  632G  2.0T  25% /rsv


[root@santo /]# du -sh *
0       bin
206M    boot
14M     dev
38M     etc
2.1M    home
0       lib
0       lib64
16K     lost+found
4.0K    media
4.0K    mnt
28K     opt
0       proc
11M     root
632G    rsv
1.3G    run
0       sbin
4.0K    srv
0       sys
32K     tmp
2.3G    usr
755M    var
chutz
  • 7,569
  • 1
  • 28
  • 57
Edwin Mwangi
  • 1
  • 2
    Dupe https://serverfault.com/questions/57098/du-vs-df-difference and https://serverfault.com/questions/275206/disk-full-du-tells-different-how-to-further-investigate and cross https://unix.stackexchange.com/questions/565377/unix-df-output-is-var-98-but-just-374mb-used-in-du-output https://unix.stackexchange.com/questions/120311/why-are-there-so-many-different-ways-to-measure-disk-usage https://unix.stackexchange.com/questions/285665/different-output-of-du-and-df – dave_thompson_085 Apr 08 '22 at 03:35

2 Answers2

2

It could be a directory starting with ..

Try du --max-depth 1 --one-file-system / which will show the top level directories, and will not go across mountpoints.

E.g.

# du --max-depth 1 --one-file-system --human-readable / | sort -h
4.0K    /media
4.0K    /srv
16K     /lost+found
20K     /mnt
488K    /tmp
14M     /SFTP
29M     /etc
72M     /boot
245M    /root
1.7G    /opt
3.7G    /var
5.7G    /usr
22G     /home
34G     /
chutz
  • 7,569
  • 1
  • 28
  • 57
1

You can accomplish some of what lsof does with a command like the following.

find /proc/*/map_files -ls | grep '(deleted)'

The names of the symlinks reported in that output include ranges in hexadecimal, which might correspond in some cases to file sizes. Only symlink targets within your full filesystem (/) are likely to be relevant in this context. It might be particularly useful to check for unlinked open files in /var/log.

If you find a suspicious unlinked open file that way, its name should include the process ID of the process which opened it, and killing that process (or rebooting) should release the space occupied by that file.

Another place to look for hidden space, besides unlinked open files (which, if lsof is installed, it can report), is mount points. If another volume is mounted over a nonempty mount point, the disk usage beneath that mount point won't be reported by du.

If it's not convenient to unmount a volume, it might be possible to explore what's underneath a mount point by exporting / via NFS and mounting it, and running du in the NFS mount. That should work unless the NFS server includes data from other volumes in what it exports. That approach might not help if it requires installing additional packages.

Eirik Fuller
  • 141
  • 1
  • Upvoted! This is the most common explanation. Hidden files in `/` are not so common, while having a process hold open an already deleted log file is a very common issue. – chutz Apr 08 '22 at 04:54