ext3 disk space problems

I ran out of disk space the other day (quite common on this particular computer). As usual I started to delete files and directories to restore a bit of space.

However this time the file system still reports that I don't have any disk space available.

Here's what df reports:

replicant:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md1              687G  672G     0 100% /
tmpfs                 250M     0  250M   0% /lib/init/rw
udev                   10M  716K  9.4M   7% /dev
tmpfs                 250M     0  250M   0% /dev/shm
overflow              1.0M  1.0M     0 100% /tmp

If I'm calculating this right I should have approximately 15GB available on /. Can anyone explain this to me? It's not just df calculating this incorrectly because I can't create files or anything on the file system.

I'm running ext3 on Debian 5.0.

Fred Asp

Posted 2011-07-20T12:03:10.740

Reputation: 31

Answers

Your filesystem has space reserved which is available only for use by the root user. This is fairly standard on Unix / Linux systems, where log files and other files required to allow the system to run are owned by the root user.

You can use the tune2fs tool to query or adjust the amount of space reserved for root.

Run:

tune2fs -l /dev/md1

To see the current parameters, look for a line such as:

Reserved block count: XXXX

This will tell you how many blocks are reserved. Take this value and multiply by the value on the Block size: YYYY line to see the amount of reserved space in bytes.

You can adjust the amount of reserved space with:

tune2fs -m 2 /dev/md1

To set the reserved space to 2%. The default is 5%. These commands must be executed as root, either via a root login of via sudo.

NOTE: Take care when using this command, as improper use can damage your filesystem!

Mike Insch

Posted 2011-07-20T12:03:10.740

Reputation: 2 363

Wow, now I got some space back but not all. What does this mean? Is there som process that is taking up space on my hdd? – Fred Asp – 2011-07-20T13:43:24.743

If you used the tune2fs -m 2 /dev/md1 command, it just means that there is still 2% of the space on your disk reserved for root. Since you don't have an independent /var partition, I wouldn't recommend going below this value as your system needs to be able to write to /var for logging and other purposes - hence the reserved space. – Mike Insch – 2011-07-20T13:45:52.960

How should I interpret my situation? That the system have 14GB reserved? Sound very much to me. Is it some program somewhere that is having very large files? Is there anything else I could do to see what the actual problem is? – Fred Asp – 2011-07-20T13:54:40.027

1Yes, your system has 13.74GB space reserved (2% of 687GB). This is not a fault with the system, it is by design and completely intentional. You may get away with tune2fs -m 1 /dev/md1, which will reduce this overhead to 6.87GB, but that is a bare minimum. It's very important that you do not set the reserved space to zero, as this will lead to guaranteed filesystem corruption the next time you run out of disk space. – Mike Insch – 2011-07-20T14:15:18.610

Thanks you a whole bunch for these answers. I can tell, I need more studying on this subject. – Fred Asp – 2011-07-21T05:36:46.507

Another benefit to the 14 GB reserve is that it reduces fragmentation. Ext3 isn't the best FS when it comes to fragmentation, and with 98% of disk in use you already have quite some. The fragmentation you'd get at 99% full would be quite bad. – MSalters – 2011-07-21T13:10:19.437

Two other possible reasons that no space has been freed:

The files you deleted are still opened by some program. As long as a program has an open file (descriptor), the file isn't actually deleted (although it's not available anymore to new programs).
The space immediately gets filled up by other files, for example logfiles in /var/log due to a process that is generating lots of messages, e.g. debugging output or errors.

Jaap Eldering

Posted 2011-07-20T12:03:10.740

Reputation: 7 596

Well, on first ans. I've restarted the computer a couple of times so it should rule that out. On the second ans. df is still reporting: size=687GB, Used=672GB and avail=0, I see what you are fishing for (some trivial stuff) and I think so myself. But I just can't see any problem. – Fred Asp – 2011-07-20T13:20:15.253

You have to look what takes up the space on the root partition. First step would be:

cd /
du -sk * |sort -n

This helps you to identify the biggest directories (probably /home or /var). Then repeat the above for the biggest directories. If you remove something, you should compare disk usage before and after, because some program may have files opened. If in doubt, you can use lsof to check it out. If your space is taken by log files, then you should revise /etc/logrotate.conf and run '/usr/sbin/logrotate /etc/logrotate.conf'. This will take care of programs that might have log files opened.

If the above does not help. Boot to single user mode and run fsck, there might be a filesystem corruption as well.

JooMing

Posted 2011-07-20T12:03:10.740

Reputation: 774

It sounds quite stupid, but have you made sure that you've emptied out your trash-can?

Sometimes when you delete files, the OS will store them in your trash-can in case you accidentally deleted the file, and only truly delete them once you've told it to Empty the Trash.

Try running fsck to make sure that the drive isn't corrupted. You may be occupying all the space with corrupted lost+found files.

akseli

Posted 2011-07-20T12:03:10.740

Reputation: 3 796

I use the rm command to delete the files. As far as I know it removes the files/dirs permanently direct. – Fred Asp – 2011-07-20T13:15:14.307

See the edit above! ^ – akseli – 2011-07-20T13:21:25.830