3

What is my situation?

I am working in a Dev Ops capacity for a service that manages jpeg files online. We had an unfortunate deploy and our media files (jpegs) are completely gone. I anticipate that our loss is probably simple and may be recoverable. I think somehow that the directory that contains the sub-directories that have our jpeg files was unlinked. If this is the case, we should be able to recover them.

What I have done so far and where we are hosted -- details

I realized the loss almost right away and fortunately we did not have any users online at that moment. I stopped our service and brought down our server. I did that to prevent any more writes to the filesystem figuring that avoiding writes was essential to file recovery.

We are running Ubuntu 16.04 in DigitalOcean. I have brought the server back up using DigitalOcean's recovery mode. This permits one to mount the filesystem of the given virtual host without running the virtual host and without running the services one has on the virtual host. This should be sufficient and correct for performing any form of recovery.

I need some where to write data for recovery. To that end, I have another server in DigitalOcean in the same data center (SFO1 unfortunately). I have mounted that host's filesystem using sshfs. I should be able to write any recovery data from my virtual host's filesystem (which is in recovery mode) to this other host via sshfs.

I selected the following utility to execute my recovery: PhotoRec

That utility is actually two utilities -- PhotoRec and TestDisk.

The filesystem of the host we wish to recover is ext4. PhotoRec supports ext4. TestDisk may not support ext4. That's okay, according to the documentation if the data is still there and largely uncorrupted, then we should be able to recover it with PhotoRec.

Here is the output of when I run df -Th -- as you can see the filesystem I wish to recover is /dev/vda1 it is of type ext4 and mounted via /mnt . I installed photorec in /lib/live/mount/overlay which is the tmpfs . I have mounted another host via sshfs within the same datacenter to put any recovered data on:

root@xxxx-xxxxxx-xxxxxxxxx:~# df -Th
Filesystem             Type        Size  Used Avail Use% Mounted on
udev                   devtmpfs    7.9G     0  7.9G   0% /dev
tmpfs                  tmpfs       1.6G  6.2M  1.6G   1% /run
/dev/sr0               iso9660     251M  251M     0 100% /lib/live/mount/medium
/dev/loop0             squashfs    220M  220M     0 100% /lib/live/mount/rootfs/rescue_rootfs.squashfs
tmpfs                  tmpfs       7.9G   14M  7.9G   1% /lib/live/mount/overlay
overlay                overlay     7.9G   78M  7.8G   1% /
tmpfs                  tmpfs       7.9G     0  7.9G   0% /dev/shm
tmpfs                  tmpfs       5.0M     0  5.0M   0% /run/lock
tmpfs                  tmpfs       7.9G     0  7.9G   0% /sys/fs/cgroup
tmpfs                  tmpfs       1.6G     0  1.6G   0% /run/user/0
root@xxx.xxx.xxx.xxx:/ fuse.sshfs  311G   13G  298G   5% /mnt2/xxxxxx-xxxxxx-xxxxxx
/dev/vda1              ext4        311G   41G  270G  14% /mnt

When I run photorec it only sees:

>Disk /dev/sr0 - 252 MB / 250 MiB (RO) - QEMU DVD-ROM

It does not see my filesystem that I want to execute recovery on at all. That is:

/dev/vda1              ext4        311G   41G  270G  14% /mnt

I have tried this with my filesystem mounted because that seems right to me. However, we did find in some online documentation that some file recovery tools require file systems to not be mounted (which seems weird to me - how is that supposed to work). So I tried executing it unmounted but same thing: it only sees:

>Disk /dev/sr0 - 252 MB / 250 MiB (RO) - QEMU DVD-ROM

Does anyone have any suggestions regarding getting photorec to see my filesystem:

/dev/vda1              ext4        311G   41G  270G  14% /mnt

I do have some backups, but unfortunately, I have about seven days worth of unbacked up photos. We could in theory live without them and reach out to our clients and get data from them and reprocess and repost it. But it would be ideal, if I could with just a few clicks of some buttons, get back this data that is likely still un the filesystem just unreachable.

Help using photorec for this purpose wouold be ideal as would any other suggestions regarding how to recover my lost/missing files.

Thanks!

Peter Jirak Eldritch
  • 399
  • 3
  • 5
  • 1
    Testdisk is for whole disks you oopsied. Photorec is pretty easy - and does individual files if you deleted them or such. – Journeyman Geek Mar 23 '19 at 13:25
  • 1
    @JourneymanGeek - thanks I appreciate it, Michael Hampton also included some additional information. Anticipate that I will provide an update in about a week -- something like, what I learned while trying to recover a business's production environment. What happened was that I had a deploy go wrong and the sub-directory of ~120 GB of JPEGs from dozens and dozens and dozens of users got destroyed. After I restore= the service, I started to investigate how to recover the files. We had backups but they were seven days old, we want to recover all of the data. – Peter Jirak Eldritch Mar 24 '19 at 02:03
  • 1
    @JourneymanGeek - PhotoRec is working but it appears that it is helping me get back the files, but without their file names. Trying to correctly restore the files without file names is going to be difficult. Also while the JPEGs may contain useful metadata, the file meta data -- user name, group, permissions, and date/times (created, accessed, modified) are missing from the data being recovered by PhotoRec. – Peter Jirak Eldritch Mar 24 '19 at 02:04
  • I vaguely recall there's a python – Journeyman Geek Mar 24 '19 at 03:38

1 Answers1

8

You can tell photorec explicitly which block device to work with, e.g. photorec /dev/vda1. It must not be mounted.

Of course, before photorec, you should try using extundelete, which may undelete your files on an ext* filesystem more quickly. Again, it must not be mounted.

And of course you should be prepared to go to your backup.

Michael Hampton
  • 237,123
  • 42
  • 477
  • 940
  • 1
    Hi @Michael-Hampton - Thanks for your help. We tried `extundelete` and we tried `ext4magic`. `extundelete` and `ext4magic` failed to find our missing inodes. We then tried running it with some additional options and it started to recover everything, but seemingly in random order so we were not finding our missing jpegs. – Peter Jirak Eldritch Mar 24 '19 at 02:36
  • 1
    Hi @Michael-Hampton - Your answer `photorec /dev/vda1` was exactly right and what we needed to proceed with `photorec`. Thank you! The issue we are having w\ respect to recovery is that `photorec` is doing an awesome job recoverying the jpeg file contents, bit it is not restoring the filenames nor any inode data. So we we are getting our jpeg files back, but not their filenames. Thousands of files w\out names or metadata. We need to associate the recovered jpegs with their former filenames because the filenames are in the database. Any suggestions on getting the filenames back as well? – Peter Jirak Eldritch Mar 24 '19 at 02:43
  • 1
    @PeterJirak photorec cannot recover filenames, only the files themselves. That's why you should be trying tools like extundelete first. – Michael Hampton Mar 24 '19 at 04:26
  • 1
    @PeterJirak, don't forget to mark Michael's answer as the correct one. – Paul Gear Mar 28 '19 at 22:22
  • @PaulGear - Thanks for the reminder. Michael Hampton's input was very valuable and useful. I am waiting to mark one or add more information until I get through this event. I learned a lot, mostly the things one would expect someone to learn under these circumstances. So, figure in a week or two, I will add more to this post and then mark something. Obviously if you or Michael are eager to contact me and work through this recovery, well, I won't say no, but I imagine the two of you have obligations od\f your own. – Peter Jirak Eldritch Mar 29 '19 at 02:48
  • @PaulGear and Michael Hampton - Recovering out of ext4 is difficult. ;-) What did I do, have I done? We had a backup that was unfortunately seven days old meaning that we were out seven days worth of photos -- that is not great. I had taken the service offline as soon as the photos had been destroyed. We are hosted in DigitalOcean. I created a snapshot of the service, brought it up, restored the backed up photos. I still have the original server that had the fault occur. I also made a DD from that original server. – Peter Jirak Eldritch Mar 29 '19 at 19:09
  • During this time we moved within DigitalOcean from SFO1 to SFO2. SFO1 has no features. SFO2 has virtual external volumes (DO's EBS), Spaces (s3 compliant s3 storage), kubernetes, and more. One of the things that got us in this mess was DOs lack of features in SFO1. We took this service outage as an opportunity to move to SFO2. Our next plan is to move to managed services within AWS. – Peter Jirak Eldritch Mar 29 '19 at 19:15