10

after some wrong php code was exceuted on one of our servers, which tried to open up a file which didn't exist, we ended up with huge apache log files which weren't configured yet to rotate. We just deleted those logfiles. After which we saw that our disk didn't really clean up. The output of a df -h shows the following

Filesystem              Size  Used Avail Use% Mounted on
/dev/sda1                18G   16G  1.1G  94% /
udev                    999M   12K  999M   1% /dev
tmpfs                   403M  848K  402M   1% /run
none                    5.0M     0  5.0M   0% /run/lock
none                   1007M   72K 1007M   1% /run/shm
/dev/mapper/vg_ftp-ftp  9.9G  5.3G  4.2G  56% /mnt/local/ftp

Inodes seem to be ok as well

Filesystem              Inodes  IUsed   IFree IUse% Mounted on
/dev/sda1              1168128 227009  941119   20% /
udev                    215094    442  214652    1% /dev
tmpfs                   219463    384  219079    1% /run
none                    219463      6  219457    1% /run/lock
none                    219463      3  219460    1% /run/shm
/dev/mapper/vg_ftp-ftp  655360   1553  653807    1% /mnt/local/ftp

After investigating with du and ncdu we still couldn't find where the space has vanished. Later we thought it were open file handlers which were still there so we did the lsof | grep deleted which gave us a few files, including the /sbin/mountall which is a bit scary.

mountall    287          root  txt       REG        8,1     120404       9334 /sbin/mountall (deleted)
mysqld      615         mysql    4u      REG        8,1          0        416 /tmp/iba1fEnV (deleted)
mysqld      615         mysql    5u      REG        8,1          0        899 /tmp/ibcxXlJG (deleted)
mysqld      615         mysql    6u      REG        8,1          0        900 /tmp/ibdby44r (deleted)
mysqld      615         mysql    7u      REG        8,1          0        901 /tmp/ibYaPXvd (deleted)
mysqld      615         mysql   11u      REG        8,1          0        986 /tmp/ibS9XlvZ (deleted)
ntpd       1231           ntp    3r      REG        8,1        481       1427 /etc/network/interfaces~ (deleted)

We are trying to avoid a reboot. Does anyone have an idea where this could come from or what else we could check to find the missing space?

Martin Schröder
  • 315
  • 1
  • 5
  • 24
latz
  • 201
  • 2
  • 6
  • the deleted `mysqld` files are all size `0`, so they're not the ones using the disk space. – R. S. Feb 21 '13 at 01:07

2 Answers2

12

The file you deleted will still be open and Apache will be writing to it. You will need to restart Apache to allow it to create a new file. A graceful restart should do the trick

apachctl -k graceful

or 

apache2ctl -k graceful

or whatever your distro uses.

user9517
  • 114,104
  • 20
  • 206
  • 289
  • Hi there, we've already tried that, we've shutdown apache2 completely later on verifying if there were no more apache processes running, which was the case and it didn't help – latz Feb 14 '13 at 10:17
  • So just for info, we finally did a reboot after all and the space reappeared magically. – latz Feb 14 '13 at 10:50
  • Then you should answer your own question :) – Luka Mar 14 '13 at 11:21
1

Have you tried running sync, too? It should normally execute automatically at least once every few minutes, but who knows.

Also, how exactly did you investigate with du? Perhaps some of those files were rotated somewhere into some other directory, and you haven't noticed that they haven't been deleted yet?

You could also try running find / -size +10000000c to find files larger than 10MB.

cnst
  • 12,948
  • 7
  • 51
  • 75