0

A Debian Server stopped responding earlier today. After hardware reset, the data on server was rolled back several months ago to the state it was in february. How is this possible?Everything, logs, webdata, databases, are as if the computer has been turned off on 03. February and turnd on today.

Does anyone know what could cause this? Hacked? How can I prevent this from happening again?

Thanks a lot!

EDIT:

Here is the update what really happened.

On the Server there is a Mirror RAID configuration. One of the HDDs stopped writing data on February 3rd. When second HDD broke earlier today, the system could only boot from the first one. That is why all the data was that old.

Apparently, this is one in the million situation.

EDIT:

So you don't get into same situation as me:
http://www.tcpdump.com/kb/os/linux/raid-alerting.html

Bojan Hrnkas
  • 115
  • 11
  • Two clarIfying questions. 1) is this a VM? 2) is it mounting storage from elsewhere (NFS or similar)? – EEAA Jul 16 '12 at 11:42
  • That's bizzare, no hack could do that. The only thing I can think is that you have been running this in a virtual environment and a snapshot was rolled back. Where is this server situated? Please provide as much info as possible to get a proper answer, in it's current form your answer will not be answered and will most likely be closed. – Alex Berry Jul 16 '12 at 11:34
  • It is a Hetzner EQ6 Server. (www.hetzner.de). Thanks for the info, I will contact the hosting company. – Bojan Hrnkas Jul 16 '12 at 11:51
  • It's much more common than you think. If you are leasing a server with built-in RAID, you should monitor it yourself, and ticket the provider if you ever see the slightest hiccup with the RAID array or an individual drive. – Michael Hampton Jul 16 '12 at 14:20
  • @MichaelHampton what kind of monitoring tool would you suggest for debian? – Bojan Hrnkas Jul 16 '12 at 16:25
  • That depends on the hardware you're using, and you didn't specify this. – Michael Hampton Jul 16 '12 at 16:26
  • To answer my own question: I found this: http://www.tcpdump.com/kb/os/linux/raid-alerting.html – Bojan Hrnkas Jul 16 '12 at 20:21
  • @BojanHrnkas, you can post your solution as an answer below, and accept it in a couple of days; this keeps the question from showing up in the list of unanswered questions – Mike Pennington Jul 16 '12 at 20:26

3 Answers3

2

Here is the update what really happened.

On the Server there is a Mirror RAID configuration. One of the HDDs stopped writing data on February 3rd. When second HDD broke earlier today, the system could only boot from the first one. That is why all the data was that old.

So you don't get into same situation as me: http://www.tcpdump.com/kb/os/linux/raid-alerting.html

Bojan Hrnkas
  • 115
  • 11
0

if you're using unionfs to overlay changes and that overlay died or was reset, you'd end up "winding the clock back", but that would be quite unusual.

Paul M
  • 553
  • 5
  • 10
0

The mdadm package comes with its own scripts to monitor the raid status daily and send an email upon failure:

/etc/cron.daily/mdadm

# cron.daily/mdadm -- daily check that MD devices are functional
(..)
# As recommended by the manpage, run
#      mdadm --monitor --scan --oneshot
# every day to ensure that any degraded MD devices don't go unnoticed.
# Email will go to the address specified in /etc/mdadm/mdadm.conf

Normally the user that gets the emails is configured as root in /etc/mdadm/mdadm.conf so in order to send the emails to a different address, instead of the local root account, add an entry like this to /etc/aliases:

root: name@example.org

Suffice it to say your MTA such as exim should be configure to be able to send out email.

aseq
  • 4,550
  • 1
  • 22
  • 46
  • There is a more detailed tutorial under the link I provided in my initial post. It also uses mdadm. – Bojan Hrnkas Aug 14 '12 at 06:16
  • Yes, but since the debian mdadm package comes with such functionality "out of the box" I figured to mention it. It may save some time. – aseq Aug 15 '12 at 04:53