1

On a newly created machine, its root ext4 partition has allegedly run out of space.

When summing up all files on the partition, the total used space is not close to the used space as noted by df. inodes are sufficiently available and reserved space is set to 3%, no bind mounts are in the namespace or so.

> df /
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/xvda2       6061632 5900308         0 100% /
> df -h /
Filesystem       Size  Used Avail Use% Mounted on
/dev/xvda2       5,8G  5,7G     0 100% /


> find / -mount -type f -exec du {} + | sort -r -h | awk '{SUM += $1} END {print SUM}'
1312200

> df -i /
Filesystem     Inodes  IUsed  IFree IUse% Mounted on
/dev/xvda2     393216 134814 258402   35% /

update:

there are no open handles and the occupancy 'survived' a reboot as well

 > lsof | sort -n -k7,7 | tail -n 2
sort       1162      root  mem       REG              202,2 99164480      19610 /usr/lib/locale/locale-archive
tail       1163      root  mem       REG              202,2 99164480      19610 /usr/lib/locale/locale-archive

I am looking for a way what is allegedly occupying the space (and how)?

solution:

The previous question referred by @michael-hampton Disk full, du tells different. How to further investigate? solved my issue: I had a nfs mount on a path in /opt - while mounted 4.4GB of files on disk were 'hidden' from the namespace and were not visible. After unmounting all remote filesystems, I was able to spot these files.

Community
  • 1
THX
  • 213
  • 1
  • 9
  • I'd say you need to run 'fsck`. See http://man7.org/linux/man-pages/man8/fsck.8.html along with http://serverfault.com/questions/28343/will-my-system-fsck-when-i-reboot – Andrew Henle Feb 25 '17 at 19:20
  • The previous question referred by @michael-hampton Disk full, du tells different. How to further investigate? solved my issue: I had a nfs mount on a path in /opt - while mounted 4.4GB of files on disk were 'hidden' from the namespace and were not visible. After unmounting all remote filesystems, I was able to spot these files. – THX Feb 26 '17 at 20:50

2 Answers2

2

You probably have a file which has been deleted but the file descriptor is still open by a process.

Take a look at lsof | grep deleted

NinjaGaiden
  • 121
  • 1
  • unfortunately, there are no open handles - the 100% occupancy also 'survived' a reboot and lsof does not show anything suspicious(?) – THX Feb 25 '17 at 18:51
1

If it is full, writes to new files will fail with an error.

Try a tree based disk usage tool like ncdu You can see the largest directories at all levels of the hierarchy.

John Mahowald
  • 30,009
  • 1
  • 17
  • 32