0

I'm running a Ubuntu server and this evening since rebooting it I am not getting the error message Cannot write to '' (No space left on device).

Earlier I did unzip a large directory in to /root which created around 100 files, I then removed these using rm /root/*. I did check ls prior to doing this and the only files in /root were the actual ones that I wanted to remove. Did I do something wrong here?

Here are some outputs:

df -h

root@localhost:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda        47G   47G     0 100% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
devtmpfs        494M  8.0K  494M   1% /dev
none             99M  192K   99M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            495M     0  495M   0% /run/shm
none            100M     0  100M   0% /run/user

df -i

root@localhost:~# df -i
Filesystem      Inodes IUsed   IFree IUse% Mounted on
/dev/xvda      3129344 52147 3077197    2% /
none            126527     2  126525    1% /sys/fs/cgroup
devtmpfs        126232  1410  124822    2% /dev
none            126527   792  125735    1% /run
none            126527     4  126523    1% /run/lock
none            126527     1  126526    1% /run/shm
none            126527     2  126525    1% /run/user

du -Pshx /* 2>/dev/null

root@localhost:~# du -Pshx /* 2>/dev/null
9.1M    /bin
4.0K    /boot
4.0K    /dev
6.8M    /etc
31M     /home
29M     /lib
4.0K    /lib64
16K     /lost+found
8.0K    /media
4.0K    /mnt
4.0K    /opt
0       /proc
76K     /root
192K    /run
7.6M    /sbin
4.0K    /srv
0       /sys
12K     /tmp
611M    /usr
541M    /var

Can someone please suggest something here, as I'm out of ideas?

chicks
  • 3,639
  • 10
  • 26
  • 36
user3636943
  • 1
  • 1
  • 1
  • Would something still have those deleted files in use? – Shane Madden Jun 21 '14 at 19:59
  • @ShaneMadden Hi, I did a `wget` and then `unzip`, accidently unzipping in to the current directory `/root` rather than it's own, so I deleted them. I didn't run any etc. Sequence was `wget`, `unzip` and then `rm`. – user3636943 Jun 21 '14 at 20:01
  • Anything from `find /proc/*/fd -ls | grep '(deleted)'`? – Shane Madden Jun 21 '14 at 20:06
  • What does `lsof` (http://manpages.ubuntu.com/manpages/hardy/man8/lsof.8.html#contenttoc21) say? – dartonw Jun 22 '14 at 03:48
  • It doesn't add up: on one hand you've got only one fs, your root partition, which is full at 47Gb, but `du` reaches 2Gb top. You should try to boot in maintenance mode and do a fs check (`fsck`). – didierc Jun 22 '14 at 14:09

1 Answers1

2

The magic of deleted files. As long as an application has a file open, it's not actually deleted. When this happens, usually the culprit is a daemon that holds a logfile or temporary open that you (or logrotate) have long since removed.

The command Shane Madden gave will let you find such files:

find /proc/*/fd -ls | grep '(deleted)'

Here's an excerpt of the output on my machine:

348233    0 lrwx------   1 dennis   dennis         64 jun 22 16:15 /proc/5623/fd/25 -> /tmp/vte1D81HX\ (deleted)
348238    0 lrwx------   1 dennis   dennis         64 jun 22 16:15 /proc/5623/fd/30 -> /tmp/vteYAG3HX\ (deleted)
348239    0 lrwx------   1 dennis   dennis         64 jun 22 16:15 /proc/5623/fd/31 -> /tmp/vteP7E3HX\ (deleted)
348240    0 lrwx------   1 dennis   dennis         64 jun 22 16:15 /proc/5623/fd/32 -> /tmp/vte4D91HX\ (deleted)
348243    0 lrwx------   1 dennis   dennis         64 jun 22 16:15 /proc/5623/fd/35 -> /tmp/vteU0VWHX\ (deleted)

So process 5623 has some temporary files open. Let's find out what it is:

$ cat /proc/5623/cmdline | tr '\0' ' ' ; echo
/usr/bin/python /usr/bin/terminator

Once you find out which process holds lots of deleted files and or big deleted files open, restart it to clear that situation.

Dennis Kaarsemaker
  • 18,793
  • 2
  • 43
  • 69