-1

I have a fresh instance of Ubuntu Server 12.04 installed and configured as a basic web server using the base lamp configuration. All settings for apache and mysql are at their defaults here, but I am seeing random services crash under a heavy apache load.

Apache will spin up many www-data threads under load, which is fine, and memory will slowly start to get consumed. When the server reaches the max memory load, it starts killing processes, the most common of which is mysql.

I looked in the mysql logs and a portion of them is below. Everything runs fine until I am short on ram here. The only thing I am running in mysql is a default wordpress database with all tables being on the MyISAM engine. These are the mysql logs right at the time of crash:

130716 14:41:12 InnoDB: Fatal error: cannot allocate memory for the buffer pool
130716 14:41:12 [ERROR] Plugin 'InnoDB' init function returned error.
130716 14:41:12 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
130716 14:41:12 [ERROR] Unknown/unsupported storage engine: InnoDB
130716 14:41:12 [ERROR] Aborting

Here are server logs at the time of crash:

Jul 16 16:47:36 ip-172-31-36-128 kernel: [728613.183276] apache2 invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jul 16 16:47:36 ip-172-31-36-128 kernel: [728613.183283] apache2 cpuset=/ mems_allowed=0
Jul 16 16:47:36 ip-172-31-36-128 kernel: [728613.183287] Pid: 29348, comm: apache2 Not tainted 3.2.0-49-virtual #75-Ubuntu
Jul 16 16:47:36 ip-172-31-36-128 kernel: [728613.183290] Call Trace:
Jul 16 16:47:36 ip-172-31-36-128 kernel: [728613.183301]  [<ffffffff81119b31>] dump_header+0x91/0xe0
Jul 16 16:47:36 ip-172-31-36-128 kernel: [728613.183305]  [<ffffffff81119eb5>] oom_kill_process+0x85/0xb0
Jul 16 16:47:36 ip-172-31-36-128 kernel: [728613.183309]  [<ffffffff8111a25a>] out_of_memory+0xfa/0x220
Travis Stoll
  • 341
  • 2
  • 12
  • And your question being? How to fix? You basically solved your own problem: ADD more RAM. – mdpc Jul 16 '13 at 19:58
  • Is there a way to limit apache and/or mysql to only consume up to a certain amount of ram?...or disable automatically killing processes at peak usage? Kind of weird to just kill processes if a server runs out of ram... – Travis Stoll Jul 16 '13 at 20:05

1 Answers1

2

Either add more RAM to cope with your increased load, or limit the number of apache processes to make sure that MySQL isn't killed. Also, make sure your VPS has swap space (many VPS providers don't provide this)

pauska
  • 19,532
  • 4
  • 55
  • 75
  • Thanks. Running this on AWS and by default t1.micro instances don't have swap. Created a swap file using the method here: http://serverfault.com/questions/218750/why-dont-ec2-ubuntu-images-have-swap .... All seems to be working much better now. – Travis Stoll Jul 16 '13 at 20:32
  • @TravisStoll good to hear. Just note that active processes should never ever be in swap, it's just there to safeguard you from crashing. I highly doubt that you want critical services like the database or the webserver residing in swap. You really should tune your limits according to how much RAM you have. – pauska Jul 17 '13 at 10:15