0

This Amazon Linux server is running a single WordPress site that is constantly crashing. The way to resolve it seems to be to run service mysqld restart and it resumes operation until it crashes again. 

160131 21:17:24 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
160201 15:30:10 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
160201 15:30:10 [Note] /usr/libexec/mysql55/mysqld (mysqld 5.5.46) starting as process 24089 ...
160201 15:30:10 [Note] Plugin 'FEDERATED' is disabled.
160201 15:30:10 InnoDB: The InnoDB memory heap is disabled
160201 15:30:10 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160201 15:30:10 InnoDB: Compressed tables use zlib 1.2.8
160201 15:30:10 InnoDB: Using Linux native AIO
160201 15:30:10 InnoDB: Initializing buffer pool, size = 128.0M
160201 15:30:10 InnoDB: Completed initialization of buffer pool
160201 15:30:10 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 4698648262
160201 15:30:10  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 4698648272
160201 15:30:10  InnoDB: Waiting for the background threads to start
160201 15:30:11 InnoDB: 5.5.46 started; log sequence number 4698648272
160201 15:30:11 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
160201 15:30:11 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
160201 15:30:11 [Note] Server socket created on IP: '0.0.0.0'.
160201 15:30:11 [Note] Event Scheduler: Loaded 0 events
160201 15:30:11 [Note] /usr/libexec/mysql55/mysqld: ready for connections.
Version: '5.5.46'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)
160202 04:42:41 mysqld_safe Number of processes running now: 0
160202 04:42:41 mysqld_safe mysqld restarted
160202  4:42:41 [Note] /usr/libexec/mysql55/mysqld (mysqld 5.5.46) starting as process 22903 ...
160202  4:42:41 [Note] Plugin 'FEDERATED' is disabled.
160202  4:42:41 InnoDB: The InnoDB memory heap is disabled
160202  4:42:41 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160202  4:42:41 InnoDB: Compressed tables use zlib 1.2.8
160202  4:42:41 InnoDB: Using Linux native AIO
160202  4:42:42 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
160202  4:42:42 InnoDB: Completed initialization of buffer pool
160202  4:42:42 InnoDB: Fatal error: cannot allocate memory for the buffer pool
160202  4:42:42 [ERROR] Plugin 'InnoDB' init function returned error.
160202  4:42:42 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160202  4:42:42 [ERROR] Unknown/unsupported storage engine: InnoDB
160202  4:42:42 [ERROR] Aborting

On AWS, the system log shows the following error:

[5613656.990910] [16575] 48 16575 92137 2204 177 4 0 0 httpd 
[5613656.994560] Out of memory: Kill process 15486 (mysqld) score 31 or sacrifice child 
[5613656.997693] Killed process 15486 (mysqld) total-vm:892860kB, anon-rss:64220kB, file-rss:0kB 
/dev/fd/11: line 1: /sbin/plymouthd: No such file or directory 
initctl: Event failed Stopping atd: [ OK ]

Any idea what this could be?

Dominick
  • 101
  • 2
  • 1
    Take a look in http://dba.stackexchange.com/questions/25077/mysql-innodb-crash-post-mortem/25083#25083 – Federico Sierra Feb 02 '16 at 16:04
  • Updated the question with information from the system log. I'm getting a "Out of memory: Kill process 15486 (mysqld) score 31 or sacrifice child" error. – Dominick Feb 02 '16 at 18:12
  • Looks like something on your machine (likely mysql) is eating up all available memory and the system is closing stuff to try and save itself. You need to either do some DB tuning or get a machine with more memory. – Catherine MacInnes Feb 02 '16 at 18:55
  • 1
    Both logs say out of memory... have you considered that the system may be out of memory? – Tim Feb 02 '16 at 19:11
  • Possible duplicate of [Ubuntu 12.04 Server - MySQL keeps crashing - Possible InnoDB problems](http://serverfault.com/questions/560541/ubuntu-12-04-server-mysql-keeps-crashing-possible-innodb-problems) – Michael - sqlbot Feb 02 '16 at 22:45
  • This is almost always the web server (particularly Apache) making demands for system memory that trigger the system to kill MySQL to ward off the otherwise-inevitable total system crash that could occur under extreme memory pressure. – Michael - sqlbot Feb 02 '16 at 22:48
  • How do I configure Apache and MySQL to not use more memory than my server has? I can't increase the memory from 2GB. – Dominick Feb 03 '16 at 16:01
  • Analysing your /etc/my.cnf might help explain why you are out of memory. – bbaassssiiee Feb 03 '16 at 22:00
  • I couldn't figure this out but I ended up moving my MySQL database to AWS RDS, so mysqld crashing should no longer be an issue. – Dominick Feb 04 '16 at 18:37

0 Answers0