No free disk space

Question

I have weird situation because Linux df command says that there is no free disk space

[root@backup cache]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3              72G   70G     0 100% /
/dev/sda1             190M   11M  170M   7% /boot
tmpfs                 248M     0  248M   0% /dev/shm

but du -sh /* says

[root@backup cache]# du -sh /*
4.0K    /bacula-restores
7.4M    /bin
5.4M    /boot
3.6T    /data
116K    /dev
55M     /etc
204K    /home
76M     /lib
16K     /lost+found
12K     /media
0       /misc
16K     /mnt
8.0K    /mount
0       /net
8.0K    /opt
0       /proc
2.3G    /root
32M     /sbin
8.0K    /selinux
168K    /share
8.0K    /srv
0       /sys
361M    /test
20K     /tmp
3.2G    /usr
1.5G    /var

Could you tell me where is a problem? Where is my space? I can't figure it out :(

Answer 1

Try:

$ lsof +L1 

This will find files that have a link count less than 1 (files removed but still beeing written to).

For those times when du and df don't match up.

Answer 2

Short version: Use lsof to find the file that is unlinked but still open. Restart the application holding the file (or the whole system, if you are lazy) and you will get your free space back.

Long version: Most likely you have a file that is open by an application and being written to, but has been unlinked (deleted) from the filesystem.

Unlike NTFS you have a sort of reference counting in ext. This means that when you delete a file you just remove the reference to it. Opening the file in a program adds a reference. So you will have to figure out which program holds the file. This is why you often see references to link/unlink when it comes to file operations in UNIX. Typically finding the file is done with the tool lsof.

The good side of this behaviour is that you never get the "File is open so can't delete it" error that you often get in Windows. You can also replace system files, such as shared libraries and your software will use the old libraries until it restarts (and loads the new ones from disk), instead of having to reboot for software upgrades (Windows again). The bad side you see right now. The size of the visible files in the filesystem will typically not match the amount of data on disk.

Answer 3

Or the poor-man lsof in this case would be (for Linux)

ls -l /proc/*/fd/ | grep deleted

Answer 4

I always do a "chattr +i /mount-point" on the empty directory while it is unmounted.

If it happens that something tries to write there while it is not mounted, a "Permission denied" error message is returned. And you notice the error right away.

If there is a properly mounted resource on "/mount-point", then the "chattr +i" has no effect and everything functions as expected.

Answer 5

you may also want to check if your inodes has been exhausted. df -i.

Answer 6

I know what went wrong. I have NFS mounted as /data, NFS has got network error connection and /data was on disk mounted as / and not as NFS share.

I thought that it may be related to mechanism like this

example dir listing: / /a [dir] file1 file2 /b [dir]

another device: / fileX fileY

If we mount 'another device' to /a we got access to this device via /a directory but original /a still exist there but is overhandled. We can get access to the original directory after unmount last mounted device. It is interesting property but in fact can bring some troubles (Not mine at this moment).

Answer 7

I had this exact same problem where the files on my drive were relatively small, but disk space was 100% used up. I had an issue where my primary boot disk was filling up after I performed a sync of backup drives.

>rsync -avxHAXW /media/backup1 /media/backup2

After this sync my primary disk was full.... which baffled me. It turns out I was accidentally copying files to my boot disk, not my backup2. I corrected the mistake, deleted the massive file on my primary disk and reran rsync. Even after unmounting and remounting my backup disks my boot disk was out of space. I even tried cleaning up large files on my boot disk....

>find . -type f -print0 | xargs -0 du -s | sort -n | tail -25 | cut -f2 | xargs -I{} du -sh {}

Deleted any large files that were no longer necessary. I also made sure to empty the trash.

>empty-trash

After all of that work... disk still full. I decided to reboot the machine. After the reboot, my boot disk partition went from 100% full to 10% full! It appears the rsync process must have continued to run in the background holding onto disk resources.

I hate giving this advice, but after doing all of the disk cleanup you might just need to reboot your machine to kill off any background processes that are hanging onto data, preventing disk from truly being freed.

Answer 8

• You can try to reduce the root's reserved space thought tune2fs
• Monitor your network from another machine to check if some hidden evil process is using your machine as a server for warez traffic
• if you have a monolithic kernel try rebooting with it, this should disable LKM rootkit if present
• run rkhunter, chkrootkit
• reboot with a live distro and analize the disk from RAM

Answer 9

In addition to already suggested causes, it could be also following:

• a different disk is mounted "over" the existing folder which is full of data
• du will calculate the size spent of mounted disk and df will show really spent
• solution: (when possible) unmount all non-root disks and check the size with du -md 1 again. Fix situation by moving hidden folder to some other place or mount on different place.