4

Possible Duplicate:
Linux different size for df and du for root partition

On an Ubuntu 12.04 server, normal users can no longer create or add to files in /home, encountering a "No space left on device" error.

The /home directory has a capacity of 1.7 terabytes and as far as I can tell is nowhere near full in terms of actual data stored or inodes used.

df -h shows:

Filesystem      Size  Used Avail Use% Mounted on
/dev/md2        1.0T   18G  955G   2% /
udev            7.7G  4.0K  7.7G   1% /dev
tmpfs           3.1G  320K  3.1G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            7.7G     0  7.7G   0% /run/shm
cgroup          7.7G     0  7.7G   0% /sys/fs/cgroup
/dev/md3        1.7T  1.7T     0 100% /home
/dev/md1        496M   45M  426M  10% /boot

/home indeed looks rather full.

du -hs /home suggests otherwise:

1.4G    /home

There appears no inode issue - df -i:

Filesystem        Inodes  IUsed     IFree IUse% Mounted on
/dev/md2        67108864  75334  67033530    1% /
udev             2013497    527   2012970    1% /dev
tmpfs            2015816    440   2015376    1% /run
none             2015816      2   2015814    1% /run/lock
none             2015816      1   2015815    1% /run/shm
cgroup           2015816      9   2015807    1% /sys/fs/cgroup
/dev/md3       113909760 105981 113803779    1% /home
/dev/md1          131072    239    130833    1% /boot

I recently deleted a many gigabytes of application cache and log data from /home, however this was in the tens of gigabytes at best and nowhere near the capcity of /home.

Update 1:

du -hs --apparent-size /home
1.2G    /home
du -hs /home
1.4G    /home

What might be going on here?

Community
  • 1
Jon Cram
  • 309
  • 1
  • 4
  • 10
  • 1
    Can you try `du` with `--apparent-size`, please? The man page says: "print apparent sizes, rather than disk usage; although the apparent size is usually smaller, it may be larger due to holes in ('sparse') files, internal fragmentation, indirect blocks, and the like". Maybe that helps? – Axel Knauf Sep 20 '12 at 15:10
  • @AxelKnauf: thanks for the suggestion, I've updated the question with these details – Jon Cram Sep 20 '12 at 15:15
  • Look at the answers to this question: http://serverfault.com/q/132998/37499 – mjcopple Sep 20 '12 at 15:18
  • Are you running `du` as root? – Shane Madden Sep 20 '12 at 15:28
  • Have you tried fsck-ing the volume? – James Yale Sep 20 '12 at 15:46
  • Check out the linked and related questions on the right. – user9517 Sep 20 '12 at 17:39
  • possible duplicate of [Linux different size for df and du for root partition](http://serverfault.com/questions/416003/linux-different-size-for-df-and-du-for-root-partition) and [du vs. df difference](http://serverfault.com/questions/57098/du-vs-df-difference) – Jeff Ferland Sep 20 '12 at 17:40

1 Answers1

10

Check the files are still opened in write mode:

lsof | awk '/deleted/ && $4 ~ /[0-9]+w/ { print $0 }'

you will see something like this:

ossec-mai  1111    ossecm    4w      REG                3,3     ... (deleted)

then find the process that still uses the file:

cd /proc/1111/fd
ls -l 4

You can free up the space without restarting the service by getting the size down to zero:

> /proc/1111/fd/4

Give it a try.

quanta
  • 50,327
  • 19
  • 152
  • 213
  • Good suggestion, I was about to ask the same. Also mentioned in [this SO question](http://serverfault.com/questions/62119/how-do-i-find-out-what-is-using-up-all-the-space-on-my-partition?rq=1). – Axel Knauf Sep 20 '12 at 15:19
  • 1
    This may well have been it. There were some very large log files related to this. I've just rebooted the machine and df -h now shows more what I'd expect. Since things are now fine I can't be sure of the cause. I expect this issue may arise again in a couple of weeks. I'll mark this as the accepted answer then if that's the cause. – Jon Cram Sep 20 '12 at 15:22
  • In addition to already suggested causes, it could be also following: * **a different disk is mounted "over" the existing folder which is full of data** * du will calculate the size spent of mounted disk and df will show really spent * solution: (when possible) unmount all non-root disks and check the size with `du -md 1` again. Fix situation by moving *hidden* folder to some other place or mount on different place. – Robert Lujo Oct 23 '14 at 18:38