-1

I had a mountable fs (pretty sure it was ext3), fsck.ext4 was run with -y and ended in a segmentation fault. Now it is unmountable (dmesg says it's corrupt) and identifies as a ext4 fs via the "blkid" command. The fs happens to sit on a raid 0 array mad up of 3 partitions.

As I am not 100% positive that is was originally ext3 I don't want to try running fsck.ext3. I also have no idea if that would work even if it was the case.

I would hope that fsck is intelligent enough to check the fs type and at least provide a warning.

Any suggestions on how to recover from this would be greatly appreciated.

FeelTheBurns
  • 111
  • 1
  • 4
    Just cut your losses and restore from backup. – EEAA Jan 05 '14 at 17:59
  • There is no backup – FeelTheBurns Jan 05 '14 at 18:05
  • 2
    Depending on your OS the fsck.ext* programs are all hard/soft links to the same binary so yes it aught to be smart enough to figure out the fs type, you'd have to read the source to confirm that though. – user9517 Jan 05 '14 at 18:10
  • 1
    If there are no backups, then this was not a production server, correct? – MDMarra Jan 05 '14 at 18:16
  • It's in a research lab, still vital. – FeelTheBurns Jan 05 '14 at 18:18
  • is it not an alias to fsck -T ext*? – FeelTheBurns Jan 05 '14 at 18:18
  • It must not have been **that** vital. Count this as a lesson... – EEAA Jan 05 '14 at 18:36
  • I did not design the system, I inherited it. – FeelTheBurns Jan 05 '14 at 18:37
  • No backups is bad enough but RAID 0 as well is just this kind of disaster waiting to happen. I know a researcher who almost had their career wrecked before it began because their lab did (pretty much) exactly this. Fortunately they were able to gather together all the data they needed to complete their PhD and fortunately their lab learned from it and put in place the right backups etc. – user9517 Jan 05 '14 at 18:44
  • 1
    Inheriting a system is *not* an excuse not to improve it, that's rather unprofessional behaviour. – Dennis Kaarsemaker Jan 05 '14 at 19:06
  • Have you checked whether or not the real cause behind this problem is a failed disk on your raid 0? – Matthew Ife Jan 05 '14 at 19:08
  • 1
    The snarky criticisms here do not help in answering the question about `fsck`. The comment area should be used to "ask for more information or suggest improvements", not for wandering into meta-topics about the company's bad practices. – Stefan Lasiewski Jan 06 '14 at 18:15

3 Answers3

3
$ ls -la /sbin/fsck.ext?
lrwxrwxrwx 1 root root 6 jun 24  2013 /sbin/fsck.ext2 -> e2fsck
lrwxrwxrwx 1 root root 6 jun 24  2013 /sbin/fsck.ext3 -> e2fsck
lrwxrwxrwx 1 root root 6 jun 24  2013 /sbin/fsck.ext4 -> e2fsck

So the fsck you ran is the correct one. If your filesystem is now so corrupt it won't mount you have two options:

  • Restore from backup
  • Pay a lot of money to a data recovery company.
Dennis Kaarsemaker
  • 18,793
  • 2
  • 43
  • 69
  • Thank you for the reply. It is still corrupt as fsck (version 2.19) segfaults. I am currently trying fsck (version 2.24) as some google results suggested segfaults weren't uncommon in older versions. – FeelTheBurns Jan 05 '14 at 18:35
  • @The_Pacifist What OS distribution and version is this? – ewwhite Jan 05 '14 at 18:41
  • The fsck.ext4 was run on a liveboot archlinux with version 2.19 of fschk, it was an older version of archlinux 3.6ish kernel. The newest run with fsck 2.24 is being run on the most recent archlinux liveboot. fsck.ext? has independent binaries. – FeelTheBurns Jan 05 '14 at 18:42
  • Arch Linux, no backup, RAID 0. Are you sure this isn't a home system? It sounds like the opposite of everything you want in a professional environment. – MDMarra Jan 05 '14 at 18:51
  • Archlinux liveboot, ubuntu 10.04 native. – FeelTheBurns Jan 05 '14 at 18:51
1

This is probably software RAID, but assuming you can see the block device, my recommendation for data recovery on ext2/3/4, XFS and other Linux filesystem is UFS Explorer. It's commercial software, but relatively inexpensive.

Seeing as blkid returns the right filesystem type, running a UFS Explorer scan against the device (and redirected to another disk) may be a clean approach to seeing what's recoverable or not.

ewwhite
  • 194,921
  • 91
  • 434
  • 799
0

Update: So those curious the last fsck (version 2.24) was able to complete. All of the resulting data was in (lost+found). I am currently backing up this directory before extracting anything. At first glance though, the majority of the data does seem to be there. So it's just a matter of traversing inodes instead of directory trees.

Thank you all for your help.

FeelTheBurns
  • 111
  • 1
  • on redhat 6 the fsck.ext? aren't link " ls -la /sbin/fsck.ext? -rwxr-xr-x. 5 root root 190344 Jun 25 2013 /sbin/fsck.ext2 -rwxr-xr-x. 5 root root 190344 Jun 25 2013 /sbin/fsck.ext3 -rwxr-xr-x. 5 root root 190344 Jun 25 2013 /sbin/fsck.ext4 " – c4f4t0r Jan 06 '14 at 17:54
  • 1
    I hope you will be able to recover at least some of the data. When you are done with this crisis, schedule some time with your manager or the stakeholders and propose a plan to fix this system that you inherited. Make sure you are able to schedule time and resources to get the system to a point where it can be properly backed up and restored. While this may not be easy, it is the management's responsibility to see that you have the time and resources required to do the job properly. – Stefan Lasiewski Jan 06 '14 at 18:06
  • @c4f4t0r: They are hard links. The 5 after the r-x is the link count. The binaries are all the same size, they are the same binary. – user9517 Jan 07 '14 at 08:57
  • @Iain yes that true :) – c4f4t0r Jan 07 '14 at 11:09
  • For the record I don't have a manager. I am a graduate student trying to graduate and play sysadmin. I know it's not the best scenario and there were bound to be mistakes/bad designs both inherited and designed. Thank you all for the help. – FeelTheBurns Jan 07 '14 at 15:22
  • don't forget to mount the filesystem read-only (`ro` flag) while you are trying to recover any files. The less you write to it, the higher the chances you can recover your files. – Renan Jan 07 '14 at 22:20