I have a linux server hosted on one of Rackspace's CloudServers and it is showing that I have both no room available, yet also shows I have 12GB not used on the root partition. This server's only job is to rsync over files nightly from another server via rsync and keep an emergency backup of those files.
I found a directory that had some backup files that weren't needed so deleted those and that is what gave me the 12GB unused. Deleting files shrinks the "Used" space, but doesn't give anything to the "Avail" space.
lsof
doesn't show that it is holding on to anything and df -i
says I have plenty of free inodes.
I've also rebooted this server multiple times and the problem persists. I've even rebooted in to rescue mode and ran an fsck
on the device and didn't see any glaring errors:
root@RESCUE-nightly-snapshot:~# fsck /dev/xvdb1
fsck from util-linux 2.20.1
e2fsck 1.42 (29-Nov-2011)
/dev/xvdb1: recovering journal
Setting free inodes count to 20630768 (was 20630766)
Setting free blocks count to 2845985 (was 2755815)
/dev/xvdb1: clean, 340752/20971520 files, 81040087/83886072 blocks
root@RESCUE-nightly-snapshot:~#
So I'm kind of stuck because normal users can't create any new files and the syslog user can't append to any log files.
Lastly, the filesystem is ext3 and something in my brain is saying maybe there is something weird with the journal. I don't know if this space could be taken up by the journal or how to even tell.
I'd appreciate any tips on how to regain this space that should've been freed.
root@nightly-snapshot:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 315G 303G 0 100% /
udev 3.9G 4.0K 3.9G 1% /dev
tmpfs 1.6G 244K 1.6G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 3.9G 0 3.9G 0% /run/shm
overflow 1.0M 0 1.0M 0% /tmp
/dev/md0 5.0T 4.2T 593G 88% /mnt/raid
root@nightly-snapshot:~# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/xvda1 20971520 340750 20630770 2% /
udev 1016188 422 1015766 1% /dev
tmpfs 1018170 319 1017851 1% /run
none 1018170 3 1018167 1% /run/lock
none 1018170 1 1018169 1% /run/shm
overflow 1018170 1 1018169 1% /tmp
/dev/md0 167772160 12607068 155165092 8% /mnt/raid
root@nightly-snapshot:~#
root@nightly-snapshot:~# lsof | grep -i deleted
root@nightly-snapshot:~#