I'm a programmer and a total newbie as a sysadmin...
Today I started receiving error emails from our production site:
Incorrect key file for table 'xxx'; try to repair it
I found the server with 2 out of 4 cores pegged at 100% CPU by MySQL. Took down the site, the crons, everything, and about 10 minutes later CPU went to 0.
Trying to backup the DB (mysqldump) resulted in CPU at 100% again, for several minutes, without any real output to log file.
I tried REPAIR TABLE, got this:
+----------------------------+--------+----------+----------+
| Table | Op | Msg_type | Msg_text |
+----------------------------+--------+----------+----------+
| db.table | repair | error | Corrupt |
+----------------------------+--------+----------+----------+
So that didn't work...
I thought of creating a new table (InnoDB) and "INSERT SELECT" from this one to try and recover some records, but SELECTING from the table (SELECT * FROM table LIMIT 1) also sends the CPU to 100% for a couple minutes, and again gives me the "Incorrect key file" error...
I got this error for 3 of the 4 tables in this DB, all 3 of them use the ARCHIVE engine and have several million records. (Largest is 46M records)
This started about an hour ago, all of a sudden.
Any ideas what I should do next? I would ideally like to lose as few records as possible, but at the same time I need to get the site back up ASAP.
Any help will be greatly appreciated!
Thank you!
Daniel