Linux: no space left on device error with 50% free space

Question

After spending a day crushing my head on the desk I would appreciate some help.

I'm running a Squid-Proxy on Fedora 14. Even if I try to touch /tmp/foo I'll receive a "no space left on device" error. I've already ran fsck. If I delete a file of say, 1MB, I'm able to store 1MB of data again but no more than that. Of note is that I have plenty of free space:

df output:

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda3             16311208   7614872   7867768  50% /
tmpfs                  1028548         0   1028548   0% /dev/shm
/dev/sda1               198337     55345    132752  30% /boot
/dev/sdb1             25803036  19407324   5084996  80% /mnt/squid-cache

Anyone have an idea what I can try to "reclaim" the free space?

Answer 1

At a guess you've run out of inodes rather than disk blocks - that is to say you have the limit on the number of files the filesystem can hold rather than the limit on the total amount of data.

You can use df -i to check this.

Answer 2

Linux reserves a percentage of blocks for the superuser. It's normally set at 5%. 50% is unusually high. With tune2fs -m you can change that.

Answer 3

This can happen if you are using btrfs, and in that case, these remedies will not work.

After following a lot of false trails I finally got on the right track after I started the search with

btrfs no

and got

btrfs no space left on device

as the first completion. That led to the BTRFS problem wiki. It's apparently a common problem.

But even from there it was not that easy to solve my problem. I was able to run the right utilities by booting into recovery mode, taking the selection that mounts my file systems, and then dropping to root.

The wiki article said run btrfs balance, but that would not run for all file systems, and eventually I found my way here (which is also the second search result.)

If your file system is really full, then even balance will not run. Or it will run only for small values of -dusage.

The recipe in this case it at the bottom of the post. I found it confusing and rejected it as a solution at first because the poster removes snapshots, which I assumed was essential, and since I was not taking snapshots, I thought it did not apply. But it did. The snapshot step may be important for some, but it was not for me.

The remedy, in overview is:

1. Create a big file
2. Make it a loop device
3. Attach the device to the fs that can't balance
4. Run balance
5. Detach the device
6. Nuke the file to get that space back

The referenced post gives the command line steps.