root full Linux. No free blocks

Question

df -h reports the '/' partition to be 100% full. While running du -hs * on each directory of this partition shows there is still lot of space.

tune2efs which reports just one block to be free. Ran fsck as well, which also shows all blocks being used.

ionode use is 14% on the '/' partition.

Please note that /var, /usr, /dev, /tmp, /home are mounted on different partitions and have space available in them.

Can you please let me know the possible causes for all blocks to be occupied and disk reporting to be full even tough there is a lot of space.

Try `du -shx /` and see if that still says there's plenty of room. That will evaluate all of `/` without relying on you catching all of the possible directories. — Insyte, Mar 07 '13 at 06:53

score 14 · Accepted Answer · answered Mar 07 '13 at 08:34

As well as the open files problem that commonly causes otherwise free space to be held unavailable, a not uncommon problem is files shielded by mount points. For instance if you have /tmp as a separate logical volume but still have files in the directory /tmp in the root filesystem, those files will be consuming space but will be hidden by the mount.

Try dropping into single-user mode at boot so nothing is running that might hold mounts open, unmount everything, and make sure there are no files hiding in the directories that are normally used as mount points.

Thank You David for the Idea. I went working on this direction. Found out that a USB HDD was attached to the machine. The machine was rebooted in the morning. It seems that the HDD was not detected at boot up and some its data went to the '/' partition. I figured this when i unmounted /tmp and the external drive mount point. — Ankit, Mar 07 '13 at 12:21

score 11 · Answer 2 · edited Mar 07 '13 at 13:32

11

This is often caused by having a file open for writing that has been deleted but the process writing to the file hasn't been restarted thus relinquishing the file. You can use lsof to find files that are open but unlinked (deleted)

lsof +L1

should do the trick. As the man page states:

A specification of the form +L1 will select open files that have been unlinked. A specification of the form +L1 <file_system> will select unlinked open files on the specified file system.

edited Mar 07 '13 at 13:32

user

4,267
4
32
70

answered Mar 07 '13 at 06:45

user9517

114,104
20
206
289

lsof +L1 gives nothing. i.e. no open unlinked file. Anything else I can check. – Ankit Mar 07 '13 at 06:50
1

This allowed me to find the culprit on my system - nothing to do with mount-shielded volumes. I had a backup process left hanging that had consumed 90% of root volume in temp files. `lsof +L1` listed them all very simply. – Synchro Oct 23 '14 at 06:15

score 0 · Answer 3 · answered Mar 07 '13 at 06:22

0

If a file is deleted while another process holds it open, that process can continue to write and, eventually, invisibly fill the disk. As soon as the process holding open the file exits, the blocks are made available.

Try evaluating each of the running daemons. If possible, restart them. If you can't figure it out, rebooting the box should clear it up.

answered Mar 07 '13 at 06:22

Insyte

9,314
2
27
45

Rebooted the system, ran fsck, nothing helped. – Ankit Mar 07 '13 at 06:50

score 0 · Answer 4 · edited Apr 13 '17 at 12:37

0

It's probably because some percentage of the available space is reserved for the root user.

See Reserved space for root on a filesystem - why? or Disk full on linux server, blocks used is much less then blocks avalable

edited Apr 13 '17 at 12:37

Community

1

answered Mar 07 '13 at 12:16

Christoffer Hammarström

111
7

root full Linux. No free blocks

4 Answers4

Linked

Related