My client has a basic LAMP web-server running on Ubuntu 14.x.x at Digital Ocean used to host a WordPress blog site.
Very unfortunately for the past year there has been a random MySQL server crash 2-4 times a month.
Generally it is about 1 time per week. When it happens I simply have to login to the box with SSH and do a quick service mysql restart command which boots up the MySQL server and all is great again for the next week or so!
The problem is, it's been over a year and I am not always available right when it is detected which can result in extended downtime all because a single command isn't ran =(
Now I need to find a solid proper solution to simply avoid this weekly random crash. I say random because it is not on a set day every week and it is not even every week, some weeks it skips crashing and runs fine for like 2 weeks!
Latest log entries
So the MySQL server crashed again tonight and withing 10 minutes I was able to grab the contents of the MySQL error log file which is posted below.
I am hoping with this log file below and the community to assist that someone might be able to help me finally put this year old nightmare out of misery for good!?
So can anyone make sense of what might be the fatal error below that is happening almost weekly and better as to how I can fix and present it?
I really would appreciate any help as I feel like giving up after all this time with no success
Thanks for any assistance please
/var/log/mysql/error.log
160330 2:33:02 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
160330 2:33:02 [Note] Plugin 'FEDERATED' is disabled.
160330 2:33:02 InnoDB: The InnoDB memory heap is disabled
160330 2:33:02 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160330 2:33:02 InnoDB: Compressed tables use zlib 1.2.8
160330 2:33:02 InnoDB: Using Linux native AIO
160330 2:33:02 InnoDB: Initializing buffer pool, size = 128.0M
160330 2:33:02 InnoDB: Completed initialization of buffer pool
160330 2:33:02 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 56012194225
160330 2:33:02 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 56012359513
160330 2:33:02 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
160330 2:33:02 InnoDB: Waiting for the background threads to start
160330 2:33:03 InnoDB: 5.5.38 started; log sequence number 56012359513
160330 2:33:03 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
160330 2:33:03 [Note] - '127.0.0.1' resolves to '127.0.0.1';
160330 2:33:03 [Note] Server socket created on IP: '127.0.0.1'.
160330 2:33:03 [Note] Event Scheduler: Loaded 0 events
160330 2:33:03 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.38-0ubuntu0.14.04.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)
160330 2:33:04 [ERROR] /usr/sbin/mysqld: Table './wordpress/wp_aiowps_failed_logins' is marked as crashed and should be repaired
160330 2:33:04 [Warning] Checking table: './wordpress/wp_aiowps_failed_logins'
160330 2:33:11 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
160330 2:33:11 [Note] Plugin 'FEDERATED' is disabled.
160330 2:33:11 InnoDB: The InnoDB memory heap is disabled
160330 2:33:11 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160330 2:33:11 InnoDB: Compressed tables use zlib 1.2.8
160330 2:33:11 InnoDB: Using Linux native AIO
160330 2:33:11 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
160330 2:33:11 InnoDB: Completed initialization of buffer pool
160330 2:33:11 InnoDB: Fatal error: cannot allocate memory for the buffer pool
160330 2:33:11 [ERROR] Plugin 'InnoDB' init function returned error.
160330 2:33:11 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160330 2:33:11 [ERROR] Unknown/unsupported storage engine: InnoDB
160330 2:33:11 [ERROR] Aborting
160330 2:33:11 [Note] /usr/sbin/mysqld: Shutdown complete
160330 2:33:12 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
160330 2:33:12 [Note] Plugin 'FEDERATED' is disabled.
160330 2:33:12 InnoDB: The InnoDB memory heap is disabled
160330 2:33:12 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160330 2:33:12 InnoDB: Compressed tables use zlib 1.2.8
160330 2:33:12 InnoDB: Using Linux native AIO
160330 2:33:12 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
160330 2:33:12 InnoDB: Completed initialization of buffer pool
160330 2:33:12 InnoDB: Fatal error: cannot allocate memory for the buffer pool
160330 2:33:12 [ERROR] Plugin 'InnoDB' init function returned error.
160330 2:33:12 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160330 2:33:12 [ERROR] Unknown/unsupported storage engine: InnoDB
160330 2:33:12 [ERROR] Aborting
160330 2:33:12 [Note] /usr/sbin/mysqld: Shutdown complete
