df says disk is full, but it is not

Question

On a virtualized server running Ubuntu 10.04, df reports the following:

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             7.4G  7.0G     0 100% /
none                  498M  160K  498M   1% /dev
none                  500M     0  500M   0% /dev/shm
none                  500M   92K  500M   1% /var/run
none                  500M     0  500M   0% /var/lock
none                  500M     0  500M   0% /lib/init/rw
/dev/sda3             917G  305G  566G  36% /home

This is puzzling me for two reasons: 1.) df says that /dev/sda1, mounted at /, has a 7.4 gigabyte capacity, of which only 7.0 gigabytes are in use, yet it reports / being 100 percent full; and 2.) I can create files on / so it clearly does have space left.

Possibly relevant is that the directory /www is a symbolic link to /home/www, which is on a different partition (/dev/sda3, mounted at /home).

Can anyone offer suggestions on what might be going on here? The server appears to be working without issue, but I want to make sure there's not a problem with the partition table, file systems or something else which might result in implosion (or explosion) later.

Thanks to all for the helpful answers. I can't create files as a normal user so it does appear that it's the 5 percent buffer that's preventing catastrophe. Now I just need to figure out why the disk is full (I'm a bit worried something malicious could be going on because none of the log files is taking up that much space and there's not much software installed, just a simple LAMP server)... — Chris, Sep 25 '11 at 09:16
First place I'd look is /tmp. Another possibility is that you have a deleted file that a running program is holding on to. I think you can run 'lsof | grep deleted' as root to find those. — Scott, Sep 29 '11 at 17:53

score 149 · Answer 1 · edited Jan 02 '14 at 20:09

149

It's possible that a process has opened a large file which has since been deleted. You'll have to kill that process to free up the space. You may be able to identify the process by using lsof. On Linux deleted yet open files are known to lsof and marked as (deleted) in lsof's output.

You can check this with sudo lsof +L1

edited Jan 02 '14 at 20:09

Kyle Schmidt

45
8

answered Sep 27 '11 at 13:20

mkomitee

1,631
1
9
3

13

It's solved the the mystery for me. I removed a large log file from uwsgi without restart the service. When queried `df -ah`, I got disk full, but `du -sh /` tells that I should have free space. After retart uwsgi I got a lot of free space! – Fabio Montefuscolo Oct 15 '14 at 16:03
1

I had 40G worth of logs stuck in limbo and lsof +L1 gave me the x-ray vision to see what happened ;-) All I had to do was restart the service. – PJ Brunet May 07 '18 at 00:18
I've noticed this happens frequently because of custom logging added to a given services config in combination with logrotate. For instance I use Unbound (under Ubuntu). Logging defaults to syslog. But, I have mine use it's own log file. This is rotated via logrotate. In Unbound's case, logrotate needs `unbound-control log_reopen` in the logrotate file so it releases the old (deleted) open log. You could also opt to simply restart the service. ref: https://lists.nlnetlabs.nl/pipermail/unbound-users/2019-July/006333.html – B. Shea Jun 10 '20 at 15:43

score 59 · Answer 2 · answered Sep 24 '11 at 21:46

59

5% (by default) of the filesystem is reserved for cases where the filesystem fills up to prevent serious problems. Your filesystem is full. Nothing catastrophic is happening because of the 5% buffer -- root is permitted to use that safety buffer and, in your setup, non-root users have no reason to write into that filesystem.

If you have daemons that run as a non-root user but that need to manage files in that filesystem, things will break. One common such daemon is named. Another is ntpd.

answered Sep 24 '11 at 21:46

David Schwartz

31,215
2
53
82

1

To the question of *WHY* your disk is full, 7G really isn't that much space. You also appear to have *everything* dumped under one partition/filesystem (`/`). This is generally considered to be a Bad Thing (because if something goes haywire `/` fills up, and the world ends) but Linux distributions still persist in doing it because it's "simpler". I'd start by looking in `/var` (esp. `/var/log`) for huge logfiles. `du -hs /` (as root) will help you find the biggest directories and possibly point you at what needs cleaning up. – voretaq7 Sep 27 '11 at 20:14

score 43 · Answer 3 · edited Nov 21 '14 at 17:25

43

You may be out of inodes. Check inode usage with this command:

df -i

edited Nov 21 '14 at 17:25

Giacomo1968

3,522
25
38

answered Sep 24 '11 at 21:48

Clifford Ilkay

531
3
2

David Goodwin · Answer 4 · 2021-07-17T11:50:44.310

24

Most Linux filesystems (ext3, ext4) reserve 5% space for use only the root user.

You can see this with e.g

dumpe2fs /dev/sda1 | grep -i reserved

You can change the reserved amount using :

tune2fs -m 0 /dev/sda1

0 in this command stands for percent of disk size so maybe you would want to leave at least 1%.

In most cases the server will appear to continue working fine - assuming all processes are being run as 'root'.

edited Jul 17 '21 at 11:50

answered Sep 25 '11 at 08:39

David Goodwin

500
3
10

Thanks a lot, this explained why I couldn't use the remaining 50 GB of my 1 TB external drive. – Martin Pecka Sep 27 '21 at 18:46

score 14 · Answer 5 · answered Oct 23 '14 at 18:41

14

In addition to already suggested causes, in some cases it could be also following:

a different disk is mounted "over" the existing folder which is full of data
du will calculate the size spent of mounted disk and df will show really spent
solution: (when possible) unmount all non-root disks and check the size with du -md 1 again. Fix situation by moving hidden folder to some other place or mount on different place.

answered Oct 23 '14 at 18:41

Robert Lujo

241
2
4

how do you find mount points other than df? – Hogan Sep 04 '15 at 13:48
1

@Hogan: maybe calling "mount" or "cat /etc/fstab" would help? – Robert Lujo Sep 07 '15 at 10:32

zzapper · Answer 6 · 2012-12-03T16:07:05.103

13

I had this problem and was baffled by the fact deleting various large files did not improve the situation (didn't know about the 5% buffer) anyway following some clues here

From root walked down the largest directories revealed by repetitively doing:-

du -sh */

until I came a directory for webserver log files which had some absolutely massive logs

which I truncated with

:>lighttpd.error.log

suddenly df -h was down to 48% used!

edited Dec 03 '12 at 16:07

answered Nov 26 '12 at 13:25

zzapper

341
3
7

18

That should really end with "... then I set up log rotation." – hayalci Dec 03 '12 at 17:02
hayalci : found that the logrotation was pointing to the wrong directory. – zzapper Dec 18 '12 at 10:08
1

very helpful, thank you! – Randy L Jun 28 '22 at 14:56

mailq · Answer 7 · 2011-09-27T14:06:15.550

9

df -h is rounding the values. Even the percentages are rounded. Omit the -h and you see finer grained differences.

Oh. And ext3 and derivates reserve a percentage (default 5%) for the file-system for exactly this problematic constellation. If your root filesystem would be really full (0 byte remaining) you can't boot the system. So the reserved portion prevents this.

edited Sep 27 '11 at 14:06

answered Sep 24 '11 at 20:40

mailq

16,882
2
36
66

Could also be that he's ran out of free inodes. Run 'df -i' to get inodes usage. – Andrew Case Sep 24 '11 at 21:52
He didn't provide information that the disk **is** full. He only **thinks** that the disk is full. 100% used space without error is only "virtually full". – mailq Sep 24 '11 at 22:32

Roel Van de Paar · Answer 8 · 2020-09-18T06:14:09.213

If you are running out of space on /dev/shm and wondering why (given that actual used space (df -shc /dev/shm) is much smaller then /dev/shm allotted size)? lsof can help:

$ sudo lsof -s +L1 | awk '{print $7" "$2" "$3" "$10}' | grep 'dev/shm' | grep "^[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]" 
7931428864 1133806 roel /dev/shm/1600335920/subreducer/2/data/ibtmp1
12710576128 1133806 roel /dev/shm/1600335920/subreducer/2/tmp/#sql-temptable-114cee-8-e18.MAD
4173332480 1352445 roel /dev/shm/1600335920/subreducer/1/data/ibtmp1
13040484352 1352445 roel /dev/shm/1600335920/subreducer/1/tmp/#sql-temptable-14a2fd-8-eb3.MAD
9670602752 2298724 roel /dev/shm/1600338626/subreducer/2/tmp/#sql-temptable-231364-8-d2e.MAD

First file is consuming ~7.9GB, second about 12.7GB etc. The regex picks up on anything 1GB and over. You can tune the regex as needed. The cause could be that an otherwise dead process is holding on to a file. df -h will not show the issue;

Filesystem      Size  Used Avail Use% Mounted on
tmpfs            90G   90G  508K 100% /dev/shm

508K, yet...

$ du -shc | grep total
46G total

You can see the 90G <> 46G offset. It's in the files above.

Then, just kill the PID (kill -9 PID) listed in the second column of the output above.

$ kill -9 1133806

Result:

Filesystem      Size  Used Avail Use% Mounted on
tmpfs            90G   72G   19G  80% /dev/shm

Great, space cleared.

The reason for doing things this way and not just something like sudo lsof +L1 | grep '(deleted)' | grep 'dev/shm' | awk '{print $2}' | sudo xargs kill -9 is that the underlaying process(es) may still be working. If you're confident it/they is/are not, that command is a potential alternative depending on your scenario. It will kill all processes which have 'deleted' files open.

An optimized command: `sudo lsof -s +L1 | awk '{print $7" "$2" "$3" "$10}' | grep 'dev/shm' | grep -E "^[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]|^[3-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]" | awk '{print $2}' | xargs kill -9 2>/dev/null`: kills all processes which have dead files > 300MB open — Roel Van de Paar, Sep 18 '20 at 06:10

score 2 · Answer 9 · answered Feb 06 '18 at 09:14

2

I did a big update of several libraries and there was a lot of unnecessary libraries and temporal files so I free space in the "/" folder using:

apt-get install -f
sudo apt-get clean

And empty your trash

answered Feb 06 '18 at 09:14

aburbanol

121
2

1

This is reasonable general advice on reducing disk usage, but it doesn't address the question about why df says the disk is full when it's not. – Andrew Schulman Feb 06 '18 at 14:51

score 1 · Answer 10 · edited Feb 23 '21 at 17:50

As this was the first page in my Google search, I want to hopefully help someone out there. I know this is a very old post and talking about the / partition not /boot.

First my system is using xfs. Over the weekend df -h showed /boot at 100% however du -h /boot was just using 40M

To resolve my issue I did

umount /boot
xfs_repair /dev/sda1
mount -a
df -h /boot

System now showing proper usage

score 1 · Answer 11 · answered Nov 23 '16 at 22:31

1

check the /lost+found, I had a system (centos 7) and some of file in the /lost+found ate up all the space

answered Nov 23 '16 at 22:31

Jude Zhu

21
1

score 1 · Answer 12 · answered May 16 '17 at 19:25

If your partition is btrfs, there may be a subvolume taking space. A btrfs filesystem can have many subvolumes, only one of which is mounted. You can use btrfs subvolume list <dir> to list all subvolumes and btrfs subvolume delete <dir>/<subvolume> to delete one. Make sure you do not delete the one that is mounted by default.

df says disk is full, but it is not

12 Answers12

Linked