4

I have a RAID 5 array with 3 Disks (sdb sdc sdd)

Over last weekend I was prompted that sdd had failed. so replaced the drive, added it back into the array and let it rebuild (1.5TB)

It stopped at about 64% with an error, found out that sdb is failing as well.

I imaged sdd onto a new drive (ddrescue) and sdb onto a new drive (ddrescue).

The copy of sdb went well only like 3MB it couldn't copy. sdb had alot more problems ( Please note I couldn't get a NEW drive so my images drives are actually physically larger then 1.5TB)

Trying to re-assemble the drive as it was before with:

mdadm -A /dev/md0 /dev/sdb /dev/sdc /dev/sdd

Gave an error:

mdadm: no recogniseable superblock on /dev/sdb

I also tried --force, same result

i also did some reading about recovering the array by completely re-building it so i tried:

mdadm --verbose --create /dev/md0 --level=5 --raid-devices=3 /dev/sdc missing missing

(sdc is the only drive that didn't fail, I was going to get the array started and add the other 2 drives in)

This resulted in:

mdadm: RUN_ARRAY failed: Input/output error

I am really in a bad spot hear. I have a lot data I need, about 1.2TB of stuff, this is a worst case scenario!

MikeyB
  • 38,725
  • 10
  • 102
  • 186
phil
  • 41
  • 1
  • 2
  • 6
    Restore from backup. This is one of the primary reasons to *never* use RAID5 with modern hard drives. Rebuild time is too long. RAID6 is slightly better in this regard. – EEAA Aug 10 '12 at 15:19
  • I think that **never* is a bit to strong. But yes, there are technical reasons to avoid RAID 5 and it should only be used when you really have to (e.g. you absolutely need the disk space and can not afford to buy another drive, or can not wait for another drive to arrive). – Hennes Aug 10 '12 at 15:26
  • There is no backup.. This is the problem. If i could get some recommendations on what to try. Thanks for your input.. but really telling me what i should have done now, isn't going to help me out ;) – phil Aug 10 '12 at 15:27
  • 3
    I'd recommend sending your drives to Kroll Ontrack or somesuch. It's going to be expensive, but it's also highly likely that you'll get your data back. Any mucking around you do at this point is more likely to cause issues. – EEAA Aug 10 '12 at 15:30

3 Answers3

5

There is no backup.. This is the problem.

Storing important data (on ANY system, no matter how reliable) without a backup is indeed the problem!

Having no backup, and having experienced a failure mode for RAID 5 for which there is no proper recovery path, you are now what our British friends would refer to as "Right Royally Rogered" (actually they would probably use more colorful language).

You're down to two options at this point:

  1. Cry.
  2. Contact a data recovery company, explain what happened, and deliver to them a princely ransom to attempt a dark magic ritual to extract your data from the clutches of your failed disks.

(1 and 2 are not mutually exclusive - In fact when you see the price for (2) you will probably do (1)...)

You can consider this a learning experience, and an expensive object lesson in the importance of regular backups and restore testing...

voretaq7
  • 79,345
  • 17
  • 128
  • 213
5

I just wanted to update everyone on the solution.

End result i got all my data back except for 2 files and here is what i did

  • Installed the good drive and the drive that failed during the rebuild
  • Forced mdadm to create the array with 1 drive missing and set the flag so that all disks are clean
  • used MC (midnight commander) to start copying files folder by folder. Once mdadm detected an error on the array it spit out some errors, mdadm would removed the drive from the array, thus making the array unable. MC would then give me a read error ( Perfect because i didn't want it to keep coping corrupt data ). Take note of the file it got hung up on
  • Unmounted the array, stop mdadm, and then restart it again with 2 drives, mark them as clean
  • Skipped OVER the file it got stuck on.

Anyways i was able to recover all data except for 2 files. I hope this helps anyone that gets into this situation. I have also went back to a RAID-1. At least with RAID-1 if 1 drive has bad sectors i can still recover the data without having to use mdadm.

PS - I have learned my lesson and also added in backups to another drive.

Thanks to everyone.

Phil
  • 51
  • 1
3

Never use raid 5 on slow disks, and never use it on software raid. Also never use it unless you have a decent raid card with its own cache and pre-failure analysis.

Basil
  • 8,811
  • 3
  • 37
  • 73
  • 1
    You forgot: never use RAID 5 on very large disks, never use RAID 5 with consumer drives, never use RAID 5 in the northern hemisphere, never use RAID 5 in the southern hemishpere... – Ward - Reinstate Monica Aug 10 '12 at 20:01
  • I manage almost a petabyte of mainframe and open system storage, mostly databases, and it's mostly raid 5. It's fine when you have the right environment. Small fast disks are expensive, and the extra protection from raid 6 or 10 doesn't justify the cost when rebuilds are 4 hours. – Basil Aug 10 '12 at 20:15
  • Meh. I got RAID 5 on four consumer drives. But... I got up to date backups and my consumer drives are on the RAIOD cards tested compatibility list. --- I still agree in general principles though, but it is not a black vs white choice and RAID5 with no spares can be perfectly fine if you want to use it to keep a file server alive till 5PM. (with subsequent emergency actions at 5PM) – Hennes Aug 11 '12 at 18:17