4

I have a linux machine dedicated to MySQL server with a pretty high load. Today I woke up and was terrified to see that database server is down. I could connect to it via SSH, but it was responding with bus error to each and every command:

[root@r1304 home]# ls
Bus error
[root@r1304 home]# tail /var/log/messages
Bus error
[root@r1304 home]# reboot
Bus error
[root@r1304 home]# free -m
Bus error
[root@r1304 home]# chkdisk
Bus error

I went to Data Center and did a hard reset, which seemed to help, but after a half an hour situation reapeated and now I can't even connet via SSH anymore.

Any ideas what this could be? how to diagnose such a problem and what are possible fixes?

Server has 32 GB RAM, 2xSSD drives with software RAID

UPDATE According to Zabbix, when MySQL died, number of processes stated to increase drammaticaly, until I did a hard reset. What could those be?

screenshot Number of processes

msanford
  • 1,427
  • 15
  • 27
Temnovit
  • 1,107
  • 6
  • 19
  • 27

1 Answers1

6

Your storage system seems to have failed. You can't read from disk at this point.

Can you see the messages in the kernel ring buffer? Look at the output of dmesg if the command even works at this point.

Do you have any backups or replication enabled?

ewwhite
  • 194,921
  • 91
  • 434
  • 799
  • Thank you for your answer. Yes, I have cold backups and replication on another server. Does this mean I have replace the hard drives? Unfortunatly, nothing works for me at this point. – Temnovit Dec 08 '12 at 13:50
  • If you have data elsewhere, I would reboot and try to examine the logs in `/var/log/messages` as soon at the system comes up. Look for indicators of filesystem or hardware failure. – ewwhite Dec 08 '12 at 13:52