-4

I have server on 2 ssd disks connected by Raid 1

Today I got 500 internal error on opening website. I have requested server management to reboot the computer.

After reboot to my surprise all my directories, files, sql databases after July 2013 went missing. So effectively all data from July 2013-May 2014 can't be seen.

Please suggest what are my options, do you think my data is wiped out and if there are ways to recover it.

Cristian Ciupitu
  • 6,226
  • 2
  • 41
  • 55
CanIndia
  • 1
  • 1
  • 2
    This is not a question for Serverfault. The answer, of-course, is to restore from backups. If you made the newbie mistake of not having backups, to take your data to a data recovery specialist if its high value, or use software tools if its not worth that kind of cost. Superuser has a lot of information about data recovery. (And as its a RAID1, you can probably just operate on a single disk as both should have identical info). It sounds like your filesystem is corrupted. – davidgo May 21 '14 at 03:28
  • Hi davidgo, Since I had RAID 1, I thought I am covered for hardware failure and data loss. But realized that's not the case now. If data on one disk got corrupted, what are the chances it will also corrupt other volume. – CanIndia May 21 '14 at 03:48
  • 1
    [Why is RAID not a backup?](http://serverfault.com/q/2888/4276) – Cristian Ciupitu May 21 '14 at 03:52
  • You haven't mentioned what operating system you are using. – Cristian Ciupitu May 21 '14 at 03:53
  • Cristian, I am using Cent OS 6.2 x64 – CanIndia May 21 '14 at 04:05
  • 2
    Chances that data is corrupt on both disks is extremely high - particularly if your RAID is not degraded. I guess you could always try a DIFF (maybe bsdiff ?) on the 2 disks and see how much they differ. (Make sure your disks are in Read-Only mode, or better, not mounted when you do this). While I would not expect the disks to be identical, I would expect them to be very similar, and possibly have identical copies of data in the partitions. – davidgo May 21 '14 at 04:26
  • 1
    What happens if you boot off a USB / other disk with only half the raid connected and try and mount the filesystem - try this for each device. If its all files after a certain point that have gone missing, maybe your RAID has not been working, and data has been written to only 1 of the 2 drives - and when your system booted it found the the other. Its a long shot, but not impossible. – davidgo May 21 '14 at 04:27
  • 1
    You're also running an end-of-life'd OS. If you ever manage to restore and get your server up and running again, you need to come up to C6.5 immediately. – MadHatter May 21 '14 at 06:54
  • 1
    RAID-1 does not help you at all in this case. Imagine (well, I guess you don't have to imagine now...) what happens if the RAID controller itself is broken and everything that gets written to disks through it is corrupted data. Fun fun fun! I truly hope you have backups somewhere. I'm not even mentioning other possibilities that can go wrong and where having backups is the only rescue you have. – Janne Pikkarainen May 21 '14 at 07:09

1 Answers1

4

Something must have happened in July 2013, which caused the RAID to become degraded and continue operating on only one drive. You did not notice this and take action, so ever since then the RAID was at risk.

The bad drive was not totally broken, but it also wasn't suitable to be in the RAID.

Then the second drive failed. At that point the RAID was simply not working at all, because it had no disks. Any reads from the RAID would likely fail with an I/O error. It is sort of surprising it was even able to produce a 500, but if the webserver is still able to process requests, a 500 is the proper error code to reply with on an I/O error.

You have to make a decision. Is it acceptable to perform a restore from backup and thus lose the data created between the most recent backup and the failure of the second drive? If that is acceptable, then you can get started doing that. But you should definitely not be doing the restore to the same drives.

So the next step for you is to get a new pair of drives and configure a new RAID on those, and then perform a restore to this new RAID.

The two faulty drives you can take to a data recovery specialist. Don't make the situation any worse by writing something to those drives. Having a copy of the most recent backup will make the recovery task a bit easier for the specialist, so you'll need to buy three new media. Two media for the new RAID, those would probably be two SSD. And one new drive to put a copy of the most recent backup on. That one could be a harddisk, since you don't need the extra performance offered by an SSD.

kasperd
  • 29,894
  • 16
  • 72
  • 122