Recover from failed recovery of software RAID5 failure

Question

I have the pleasure ;-) to have been handed the task to try to get data off a server that had a RAID5 array problem and a failed attempt to recover it.

Background: the computer in question is some server from IBM with a 6-port SATA2 backplane. Four identical harddrives with 1 TB each are attached and configured as a RAID5 with 4 devices. In the past the server reportedly lost "contact" with one or more drives every once in a while and the RAID5 would go down. After rebooting the server the RAID5 would resync and all would be well. The guy maintaining this server believes that this is a hardware issue.

A few days ago the desaster happened. At first it supposedly looked the same as the previous times, with the RAID5 going down because of some drives being offline. But a reboot did not fix the problem, instead the RAID5 was "broken" (no information what this means). The maintainer tried to recreate the RAID5 using mdadm --create and he said it would show the RAID as "recovering" (not resyncing!). After 5-6 hours of work, this process was complete and the RAID5 was shown as being active again. However the filesystem was not mountable (no superblocks).

Using data recovery tools like testdisk we were able to recover files, however they seem to be corrupt (esp. noticable in raw WAV files).

That's the state I am taking over. I don't really know what went wrong, but my guess is that if it wasn't broken right after the reboot, the recovering caused data to be lost. Sequence of commands issued are not available either.

I believe there is nothing to recover, but wanted to check with the experts here before starting over from scratch (and introducing an actual backup strategy, cough).

Answer 1

Hopefully you have backups, as I highly doubt that you will recover all (if any) of your data. If you have no backups, and the data recovery is life or death, you can contact a Disaster Recovery specialist that may be able to assist you in retreiving the data.

Answer 2

I had something similar few weeks ago., i will make the long story short :)
First of all, about the server sometime see the HDD, then after reboot or shutdown can't see it. Its a hardware issue with the raid controller, just replace it.

I have an HP server with 6 HDDs, one of the disks failed and the server could not recognize the new HDD, after testing extra 2 new disks with no success, we decided to shutdown the server and power it up again, then BOOM! another disk failed and lost the data in the raid.

After days of looking for solutions with no success, i decided to replace the raid controller, and the server did recognized one of the disks and we back online.

During my searching on the internet for solutions i found this site: https://www.runtime.org/raid.htm
Maybe it can fix your problem, try the bootable CD