5

The previous administrator of a server that is now under my supervision made a mistake. He accidentally created a LVM volume (no more than pvcreate, I think, though not sure) on a disk that actually contained an Ext4 partition with data. How do I recover data from a mistake like this? I'm ready to read ext4 documentation and roll out my own, but perhaps I don't need to? A few tools that I tried were unable to find an Ext4 filesystem on it, so I guess I need something more serious.

Michael Hampton
  • 237,123
  • 42
  • 477
  • 940
Vilx-
  • 791
  • 4
  • 13
  • 25
  • 1
    Has the previous administrator actually written any more data than the `pvcreate` headers to the disk/partition? Like reformatted over the top of it? – Matthew Ife Sep 08 '14 at 01:09
  • @MatthewIfe - I don't know. Most likely not. He says it happened during a reinstall of the OS and he realized his mistake immediately. All the system data is on other disks, so I don't think that anything else but pvcreate has been done to it. But I don't have any proof. – Vilx- Sep 08 '14 at 08:02
  • Recover if you can, or restore from latest backup (... you do have backups, don't you? ^^) – Olivier Dulac Sep 08 '14 at 21:08
  • @OlivierDulac - I *will* have backups. :) – Vilx- Sep 08 '14 at 21:10

3 Answers3

6

If you run mkfs.ext4 -n /the/partition it will print out what a EXT4 formatted drive would look like on that system.

# mkfs.ext4 -n /dev/dm-3
mke2fs 1.42.8 (20-Jun-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
3276800 inodes, 13107200 blocks
655360 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
400 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
    4096000, 7962624, 11239424

Of note is that it will tell you where the superblock locations are.

Using this information, attempt to mount the drive using an alternative superblock..

mkdir /tmp/mntpnt
mount -o ro,sb=163840 /dev/dm-3 /tmp/mntpnt

Providing only the headers of the partition were destroyed this may work.

If that doesn't work however, you can try to fix the filesystem using fsck.ext4 by specifying the superblock address. Backup the data with dd or something before you do this.

fsck.ext4 -b 163840 /dev/dm-3

This may just end up overwriting the bad superblock with one of the known good ones, which could be enough to get the entire disk remounted. Then again you may lose key inodes (like your root filesystem inode). Mileage may vary.

Matthew Ife
  • 22,927
  • 2
  • 54
  • 71
  • 4
    Don't do any of this on the actual disk, but on a copy of it. – Michael Hampton Sep 08 '14 at 01:28
  • @MichaelHampton - Of course. That goes without saying. It's not the first time I'm dealing with data recovery, although this is the trickiest case so far. – Vilx- Sep 08 '14 at 08:07
3

I would give the UFS Explorer demo a try to see what it can detect... It's my go-to utility for filesystem recovery. I once had an XFS partition with 4 million files accidentally deleted and used this utility to recover the data.

But outside of that, it's a learning experience and a chance to test your backup routine. Sorry for the loss.

ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • Interesting! It DID detect the filesystem and it's label. But the scan will take over 2h (it's a 1TB drive), so final results are still pending. But will see what comes of it! Thanks! – Vilx- Sep 07 '14 at 22:26
  • The software is cheap (compared to a real data recovery service), and the scan will give you an idea of what's recoverable. Be sure to recover to another hard disk. – ewwhite Sep 07 '14 at 22:27
  • Ahh, nevermind, that was just the scanning for filesystems part. Once I found it, I could abort and just continue with the FS found. Now it's reconstructing and tells me that it will take only six minutes. Keeping fingers crossed! – Vilx- Sep 07 '14 at 22:28
  • ...aaaand it didn't find anything. Empty partition. Bummer. :( – Vilx- Sep 07 '14 at 22:34
  • Try all scanning options. I'm not sure if there is a lengthier or more comprehensive scan available. – ewwhite Sep 07 '14 at 22:34
  • Well, there's something called "IntelliRAW" that I didn't check before. That's the only other option. Will see if that does anything. Seems to take the same 6 minutes as before. – Vilx- Sep 07 '14 at 22:37
  • Oh gods. After the 6-minute initial scan now there's a full scan or something with an ETA of 20h! Though it's dropping rapidly. Not sure how long it will actually take. – Vilx- Sep 07 '14 at 22:43
  • I aborted to see the incomplete results. Seems that the IntelliRAW just dumps the inode contents or something. I'm getting files named "00002601.txt" and inside there are man pages or logfiles or whatever. I'm afraid this won't do. :( – Vilx- Sep 07 '14 at 22:46
  • 1
    That's about right. What were you expecting? :) – ewwhite Sep 07 '14 at 22:48
  • I seriously doubt that there _was_ a backup routine. Which may be one reason why the previous sysadmin is _previous_. @Vilx- Make sure that, whatever else you do, you get backups in place. – Michael Hampton Sep 07 '14 at 22:51
  • If there was a backup routine, I wouldn't HAVE this question. :D Luckily, the data there isn't THAT important, so if it's lost - well, meh, but we start a new life. Still, I want to give it my best. Looks like it's time to start pretending that I'm a digital forensics expert and write my own tool. :) – Vilx- Sep 07 '14 at 22:56
  • @ewwhite - A nice tree with filenames and folder names and everything? :) – Vilx- Sep 07 '14 at 22:56
  • @Vilx-You can write a tool to associate the file contents with names... That's what I ended up doing with the recovered data in my earlier example. – ewwhite Sep 07 '14 at 23:00
  • @ewwhite - I don't follow. How could you guess filename & location by file contents? – Vilx- Sep 07 '14 at 23:04
  • 1
    @Vilx- Context. That's really up to you? DO you KNOW what was on the partition before? – ewwhite Sep 07 '14 at 23:12
  • 2
    @Vilx- without a file system, the files themselves are orphans and don't know where they belong. They don't have file names or locations, because that stuff was stored in the file system headers. But they *do* have data which is the most important part. – Mark Henderson Sep 07 '14 at 23:42
  • @MarkHenderson - I know that. The problem is that the disk contained lots of private data - photos, videos, MP3, some websites... Tens of thousands of files. Manually going through each of them would take years. – Vilx- Sep 08 '14 at 08:04
3

The first step in any recovery operation is to make a copy of the data, and perform the recovery on the copy. Once you've done that, you can attempt to recover the data.

Depending on what exactly the admin did, the most likely damage is that the partition table has been corrupted, the volume's primary superblock has been corrupted, or both. You can re-build the partition table using fdisk: simply create a new partition table with the same setup as the original. Make sure you get the type (MBR or GPT) right. e2fsck -b will let you perform a filesystem repair using one of the secondary copies of the superblock, or in the unlikely event that they've all been corrupted, mke2fs -S will re-create the metadata structure without touching the data.

Mark
  • 649
  • 4
  • 10
  • More specifically, I'd suggest imaging the drive, and working off a copy of that. It takes up space *but* it prevents further data loss. – Journeyman Geek Sep 08 '14 at 01:15