Solaris says my partition is using 100+ GB, but the partition looks empty. Where is the space being used?

Question

I'm telnetting into a Solaris 5.9 box that's supposed to store our oracle database. I deleted the old version of the database, tried to move in the new version (the entire database is 90GB) but I kept getting "disk is full" errors. I ran

df -hk

and found that the partition (called "/d02") I was trying to move files to had 135GB total, 123GB used, and 10GB available. However, when I run

ls -lah /d02

I get 4 directories: 3 of them are 512B and 1 is 8KB.

When I deleted the old oracle database, did the space somehow not get freed? How can I free up the space, or at least see how my space is being taken up?

Thanks for your time.

Answer 1

I figured it out. I had to kill any processes that were holding on to that data--in this case Oracle.

Also I think I was supposed to post this to serverfault. Sorry!

Answer 2

+1 for @UnixGeek answer: deleting a file does not free up disk space until the process(es) with the file open are gone.

The "standard" tool to list open files is lsof -- I still don't think it's included with Solaris (it's been several years since I was a Solaris admin, or even used it), but is available from sunfreeware.

A quick google finds the Solaris utility pfiles can do something similar (but I haven't used it).

Also, fuser on the file name, but you need to check it before you delete the file(s). fuser is also useful for finding what has a port open, when a process tell's you it's already in use.

I will plug A Sysadmin's Unixersal Translator (ROSETTA STONE), a very useful guide for anyone using multiple Unices and translating between them.

Bonus trick for recovering deleted files in /proc-based systems which are still open by a process:

cat /proc/<pid>/fd/<fd-of-delete-file>  > recovered.data

I have used this in Linux, where ls -l /proc/<pid>/fd shows the initial file name (and [deleted]) in the output.

Answer 3

Yes, you discovered the root cause. On unix, removing a file removes the link from the access list in the directory. However, it may not remove the contents until the file handle is closed, which is associated with the file. This is a rather common issue, which is hard to track down, after the fact.

Once the file handle is closed, the data links can then be released to the free list, as you found out. Most likely, a log file was still open. You might have found some odd behaviour once the database restarted, as it's log location was missing, and it starts logging to the console or other strange places like /tmp.