How to recover deleted file if it is still opened by some process?

Question

$ cat important_file > /dev/null &
[1] 9711
$ rm important_file 
$ killall -STOP cat

[1]+  Stopped                 cat important_file > /tmp/p
$ ls -l /proc/`pidof cat`/fd/
total 0
lrwx------ 1 vi vi 64 May 13 20:32 0 -> /dev/pts/29
l-wx------ 1 vi vi 64 May 13 20:32 1 -> /tmp/p
lrwx------ 1 vi vi 64 May 13 20:32 2 -> /dev/pts/29
lr-x------ 1 vi vi 64 May 13 20:32 3 -> /home/vi/important_file (deleted)

How to recover this important_file?

I tried something like

injcode -m dup2 -ofd=3 -ofilename=/tmp/recovered_file -oflags=O_CREAT $PID_OF_CAT

but it does nothing.

Answer 1

If /home is NFS, there will be a .nfsNNNNNNNNNN file in /home/vi that you can access/copy. If home is a local filesystem, you should be able to do the same thing via the /proc/PID/fd/3 link:

cp /proc/PID/fd/3 /tmp/recovered_file

If you want to actually undelete the file, here's a blog post on the subject.

Answer 2

... better than copying at a given time (and gathering only that time's snapshot of the file's content) is to "tail -f" that file into a new file:

tail -c +0 -f /proc/PIDofProgram>/fd/# > /new/path/to/file

(thanks to tail's cautious programmers, that will even work with binary output.)

During its runtime, the tail -f itself keeps the file open, safely preventing it from being purged off disk when the original program ends. Thus, don't stop the tail -f immediately after your original program ends - check the tail'ed /new/path/to/file first whether it is what you want. If it isn't (or is unsatisfying for any other reason), you can copy the original file again, but this time after all writing to it has finished by "Program" and from the still-running tail -f's /proc/PIDoftail/fd/ directory.

Answer 3

Use lsof to find the inode number, and debugfs to recreate a hard link to it. For example:

# lsof -p 12345 | grep /var/log/messages
syslogd 12345 root    3w   REG                8,3    3000    987654 /var/log/messages (deleted)
# mount | grep var
/dev/sda2 on /var type ext3 (rw)
# debugfs -w /dev/sda2
debugfs: cd log
debugfs: ln <987654> tmp
debugfs: mi tmp
                      Mode    [0100600] 
                   User ID    [0] 
                  Group ID    [0] 
                      Size    [3181271] 
             Creation time    [1375916400] 
         Modification time    [1375916322] 
               Access time    [1375939901]
             Deletion time    [9601027] 0
                Link count    [0] 1
               Block count    [6232] 
                File flags    [0x0] 
...snip...
debugfs:  q
# mv /var/log/tmp /var/log/messages
# ls -al /var/log/messages
-rw------- 0 root root 3301 Aug  8 10:10 /var/log/messages

---

Before you complain, I faked the above transcript as I don't have a deleted file to hand right now ;-)

I use mi to reset the delete time and link count to sensible values (0 and 1 respectively), but it doesn't work properly - you can see the link count remains at zero in ls. I think the kernel might be caching the inode data. You should probably fsck at the earliest opportunity after using debugfs, to be on the safe side.

In my experience, you should create the link using a temporary file name and then rename to the proper name. Linking it directly to the original file name seems to cause directory corruption. YMMV!

• http://glandium.org/blog/?p=87
• http://www.cyberciti.biz/tips/linux-ext3-ext4-deleted-files-recovery-howto.html

Answer 4

You can just cp the file, i.e:

cp /proc/<pid>/fd/<fdno> /new/path/to/file

Of course, if the file is still being modified, you'll run into trouble with this approach.