Repeatedly facing mysqld out of memory. Lots of PHP processes spawned, taking up all the swap.
[562074.326710] mysqld invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 [562074.329646] mysqld cpuset=/ mems_allowed=0 [562074.330882] Pid: 21351, comm: mysqld Not tainted 2.6.32-642.1.1.el6.x86_64 #1 [562074.333171] Call Trace: [562074.334870] [] ? dump_header+0x90/0x1b0 [562074.336812] [] ? security_real_capable_noaudit+0x3c/0x70 [562074.338692] [] ? oom_kill_process+0x82/0x2a0 [562074.340270] [] ? select_bad_process+0xe1/0x120 [562074.341582] [] ? out_of_memory+0x220/0x3c0 [562074.343229] [] ? __alloc_pages_nodemask+0x93c/0x950 [562074.344930] [] ? ext4_get_block+0x0/0x120 [ext4] [562074.346436] [] ? alloc_pages_current+0xaa/0x110 [562074.347604] [] ? __page_cache_alloc+0x87/0x90 [562074.348889] [] ? find_get_page+0x1e/0xa0 [562074.350660] [] ? filemap_fault+0x1a7/0x500 [562074.352207] [] ? __do_fault+0x54/0x530 [562074.353872] [] ? handle_pte_fault+0xf7/0xb20 [562074.355966] [] ? rwsem_down_failed_common+0x95/0x1d0 [562074.357531] [] ? rwsem_down_read_failed+0x26/0x30 [562074.359503] [] ? handle_mm_fault+0x299/0x3d0 [562074.361105] [] ? call_rwsem_down_read_failed+0x14/0x30 [562074.362522] [] ? __do_page_fault+0x146/0x500 [562074.364590] [] ? lock_sock_nested+0xac/0xc0 [562074.387996] [] ? _spin_unlock_bh+0x1b/0x20 [562074.389187] [] ? pvclock_clocksource_read+0x58/0xd0 [562074.390457] [] ? kvm_clock_read+0x1c/0x20 [562074.391586] [] ? kvm_clock_get_cycles+0x9/0x10 [562074.392673] [] ? getnstimeofday+0x58/0xf0 [562074.394697] [] ? do_page_fault+0x3e/0xa0 [562074.395974] [] ? page_fault+0x25/0x30 [562074.397218] Mem-Info: [562074.399103] Node 0 DMA per-cpu: [562074.400605] CPU 0: hi: 0, btch: 1 usd: 0 [562074.402129] CPU 1: hi: 0, btch: 1 usd: 0 [562074.403506] Node 0 DMA32 per-cpu: [562074.405629] CPU 0: hi: 186, btch: 31 usd: 0 [562074.406749] CPU 1: hi: 186, btch: 31 usd: 30 [562074.407914] Node 0 Normal per-cpu: [562074.409325] CPU 0: hi: 186, btch: 31 usd: 0 [562074.410787] CPU 1: hi: 186, btch: 31 usd: 30 [562074.411941] active_anon:717855 inactive_anon:182000 isolated_anon:5083 [562074.411943] active_file:32 inactive_file:192 isolated_file:64 [562074.411944] unevictable:0 dirty:0 writeback:259 unstable:0 [562074.411944] free:21353 slab_reclaimable:5762 slab_unreclaimable:10906 [562074.411945] mapped:63 shmem:43 pagetables:21560 bounce:0 [562074.418218] Node 0 DMA free:15724kB min:248kB low:308kB high:372kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15312kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [562074.424382] lowmem_reserve[]: 0 3512 4017 4017 [562074.426602] Node 0 DMA32 free:61060kB min:58868kB low:73584kB high:88300kB active_anon:2667432kB inactive_anon:524160kB active_file:4kB inactive_file:396kB unevictable:0kB isolated(anon):15980kB isolated(file):384kB present:3596500kB mlocked:0kB dirty:0kB writeback:560kB mapped:232kB shmem:172kB slab_reclaimable:15184kB slab_unreclaimable:17264kB kernel_stack:6736kB pagetables:66680kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:64 all_unreclaimable? no [562074.432920] lowmem_reserve[]: 0 0 505 505 [562074.434773] Node 0 Normal free:8392kB min:8464kB low:10580kB high:12696kB active_anon:203604kB inactive_anon:203616kB active_file:124kB inactive_file:320kB unevictable:0kB isolated(anon):4992kB isolated(file):128kB present:517120kB mlocked:0kB dirty:0kB writeback:476kB mapped:20kB shmem:0kB slab_reclaimable:7864kB slab_unreclaimable:26360kB kernel_stack:6832kB pagetables:19560kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no [562074.441562] lowmem_reserve[]: 0 0 0 0 [562074.443251] Node 0 DMA: 3*4kB 0*8kB 0*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15724kB [562074.449972] Node 0 DMA32: 492*4kB 4530*8kB 511*16kB 140*32kB 89*64kB 3*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 60784kB [562074.455851] Node 0 Normal: 1455*4kB 68*8kB 3*16kB 2*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 8396kB [562074.461844] 22462 total pagecache pages [562074.463320] 21989 pages in swap cache [562074.464571] Swap cache stats: add 34726956, delete 34704967, find 25405257/28809084 [562074.466666] Free swap = 0kB [562074.467793] Total swap = 4194252kB [562074.508867] 1048575 pages RAM [562074.510066] 67960 pages reserved [562074.511918] 7658 pages shared [562074.513082] 947230 pages non-shared [562074.514178] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [562074.516236] [ 490] 0 490 2680 0 1 -17 -1000 udevd [562074.518561] [ 790] 0 790 2662 0 1 -17 -1000 udevd [562074.520752] [ 1342] 0 1342 62368 50 1 0 0 rsyslogd [562074.522842] [ 1357] 0 1357 4563 23 0 0 0 irqbalance [562074.525130] [ 1376] 25 1376 59678 129 0 0 0 named [562074.527174] [ 1394] 81 1394 5359 1 0 0 0 dbus-daemon [562074.529698] [ 1432] 0 1432 1020 0 0 0 0 acpid [562074.532021] [ 1451] 28 1451 336791 177 0 0 0 nscd [562074.534053] [ 1569] 0 1569 16086 24 1 -17 -1000 sshd [562074.536171] [ 1580] 38 1580 6262 27 1 0 0 ntpd [562074.539282] [ 2053] 0 2053 4662 1 0 0 0 dovecot [562074.542072] [ 2065] 401 2065 10639 1 0 0 0 pop3-login [562074.545036] [ 2066] 401 2066 10637 1 0 0 0 imap-login [562074.547647] [ 2067] 97 2067 3368 1 0 0 0 anvil [562074.550029] [ 2068] 0 2068 3401 1 1 0 0 log [562074.552190] [ 2070] 401 2070 10655 1 0 0 0 pop3-login [562074.554224] [ 2071] 401 2071 10670 1 0 0 0 imap-login [562074.556886] [ 2072] 0 2072 3659 1 0 0 0 config [562074.560112] [ 2082] 47 2082 18238 1 1 0 0 exim [562074.562160] [ 2100] 0 2100 49742 335 1 0 0 /usr/local/cpan [562074.564197] [ 2102] 0 2102 53119 223 0 0 0 spamd child [562074.566209] [ 2129] 0 2129 45352 1 0 0 0 abrtd [562074.568273] [ 2139] 0 2139 45197 1 0 0 0 abrt-dump-oops [562074.570489] [ 2150] 0 2150 19189 124 0 0 0 httpd [562074.572607] [ 2165] 0 2165 34191 0 0 0 0 pure-ftpd [562074.574719] [ 2167] 0 2167 34036 1 0 0 0 pure-authd [562074.577609] [ 2260] 0 2260 29271 27 1 0 0 crond [562074.580212] [ 2275] 0 2275 4857 5 1 0 0 atd [562074.582301] [ 2375] 0 2375 26952 125 0 0 0 cpsrvd (SSL) - [562074.584483] [ 2381] 0 2381 8172 69 0 0 0 queueprocd - wa [562074.586500] [ 2391] 0 2391 27506 53 0 0 0 dnsadmin - serv [562074.589509] [ 2419] 0 2419 20822 258 0 0 0 tailwatchd [562074.592045] [ 2433] 0 2433 19586 1 0 0 0 cPhulkd - proce [562074.594295] [ 2476] 0 2476 33028 60 0 0 0 cpdavd - accept [562074.597210] [ 2512] 0 2512 5915 102 1 0 0 cpanellogd - sl [562074.600503] [ 2540] 0 2540 1016 1 1 0 0 mingetty [562074.603689] [ 2542] 0 2542 1016 1 0 0 0 mingetty [562074.610302] [ 2545] 0 2545 1016 1 1 0 0 mingetty [562074.615154] [ 2547] 0 2547 2679 0 0 -17 -1000 udevd [562074.617989] [ 2548] 0 2548 1016 1 1 0 0 mingetty [562074.622920] [ 2550] 0 2550 1016 1 1 0 0 mingetty [562074.625853] [ 2552] 0 2552 1016 1 1 0 0 mingetty [562074.628530] [30499] 0 30499 147473 23 0 0 0 clamd [562074.631934] [ 8712] 0 8712 2394 1 1 0 0 mysqld_safe [562074.634865] [ 9408] 402 9408 564397 42943 0 0 0 mysqld [562074.637620] [24553] 0 24553 27283 1 1 0 0 sshd [562074.641062] [24562] 0 24562 13893 1 0 0 0 sftp-server [562074.643785] [25296] 0 25296 27283 1 1 0 0 sshd [562074.646513] [25312] 0 25312 26649 1 0 0 0 bash [562074.649571] [25954] 99 25954 19222 60 0 0 0 httpd [562074.652579] [25955] 0 25955 19061 1 0 0 0 leechprotect [562074.655407] [25958] 99 25958 334134 795 0 0 0 httpd [562074.657912] [25959] 99 25959 334193 806 1 0 0 httpd [562074.660551] [25960] 99 25960 334438 802 0 0 0 httpd [562074.663110] [26044] 99 26044 334439 803 0 0 0 httpd [562074.665618] [26206] 99 26206 334191 844 1 0 0 httpd [562074.668698] [26361] 99 26361 334134 886 0 0 0 httpd [562074.671241] [28324] 0 28324 27283 1 1 0 0 sshd [562074.673759] [28333] 0 28333 26649 1 0 0 0 bash [562074.676257] [28546] 99 28546 334167 821 0 0 0 httpd [562074.702428] [ 1091] 0 1091 27283 1 1 0 0 sshd [562074.704538] [ 1130] 0 1130 26649 1 0 0 0 bash [562074.706652] [ 5276] 500 5276 77474 7801 1 0 0 php [562074.708866] [ 5278] 500 5278 77582 7649 0 0 0 php [562074.711662] [ 5281] 500 5281 77580 7259 1 0 0 php [562074.714235] [ 5290] 500 5290 77601 7474 1 0 0 php [562074.716490] [ 5295] 500 5295 77538 7540 1 0 0 php [562074.718824] [ 5299] 500 5299 77414 7551 1 0 0 php [562074.721384] [ 5302] 500 5302 77518 8104 1 0 0 php [562074.724114] [ 5304] 500 5304 77414 7522 1 0 0 php [562074.726653] [ 5308] 500 5308 77582 7241 0 0 0 php [562074.729482] [ 5309] 500 5309 77542 7294 0 0 0 php [562074.731758] [ 5367] 500 5367 79215 8223 0 0 0 php [562074.734948] [ 5369] 500 5369 79230 7073 1 0 0 php [562074.738000] [ 5389] 500 5389 74670 6863 1 0 0 php [562074.740609] [ 5407] 500 5407 74996 5832 0 0 0 php [562074.742809] [ 5415] 500 5415 75850 6233 1 0 0 php [562074.745813] [ 5416] 500 5416 75061 5948 1 0 0 php [562074.748265] [ 5418] 500 5418 74992 5796 1 0 0 php [562074.750694] [ 5420] 500 5420 76190 7332 0 0 0 php [562074.753004] [ 5430] 500 5430 74932 6113 1 0 0 php [562074.755384] [ 5434] 500 5434 75460 6480 1 0 0 php [562074.757639] [ 5439] 500 5439 74928 5879 1 0 0 php [562074.761562] [ 5440] 500 5440 75125 5701 1 0 0 php [562074.764306] [ 5442] 500 5442 75914 6693 1 0 0 php [562074.766600] [ 5446] 500 5446 75214 5475 1 0 0 php [562074.768955] [ 5448] 500 5448 74705 6309 1 0 0 php [562074.771372] [ 5450] 500 5450 73976 6564 0 0 0 php [562074.773778] [ 5451] 500 5451 73831 6459 1 0 0 php [562074.776178] [ 5458] 500 5458 74611 7603 0 0 0 php [562074.778666] [ 5459] 500 5459 75781 8240 1 0 0 php [562074.781649] [ 5465] 500 5465 73772 7310 1 0 0 php [562074.784412] [ 5470] 500 5470 74414 7797 1 0 0 php [562074.788349] [ 5476] 500 5476 74864 8214 1 0 0 php [562074.790625] [ 5480] 500 5480 73510 8366 0 0 0 php [562074.793613] [ 5482] 500 5482 73708 8698 0 0 0 php [562074.796689] [ 5492] 500 5492 72444 11027 1 0 0 php [562074.799184] [ 5496] 500 5496 72784 12415 1 0 0 php [562074.802301] [ 5499] 500 5499 73768 11388 0 0 0 php [562074.804993] [ 5500] 500 5500 72582 11302 0 0 0 php [562074.808597] [ 5503] 500 5503 70752 9687 0 0 0 php [562074.810941] [ 5513] 500 5513 73447 10543 0 0 0 php [562074.813474] [ 5519] 500 5519 72728 12005 1 0 0 php [562074.815596] [ 5520] 500 5520 72710 11134 1 0 0 php [562074.818435] [ 5523] 500 5523 72246 11313 0 0 0 php [562074.821278] [ 5533] 500 5533 73576 12705 0 0 0 php [562074.824227] [ 5536] 500 5536 70305 11674 1 0 0 php [562074.826292] [ 5541] 500 5541 73378 13119 1 0 0 php [562074.828638] [ 5545] 500 5545 69769 10932 1 0 0 php [562074.831461] [ 5548] 500 5548 69382 10296 0 0 0 php [562074.833803] [ 5549] 500 5549 73458 13978 1 0 0 php [562074.836589] [ 5564] 500 5564 69708 11383 1 0 0 php [562074.839962] [ 5568] 500 5568 69380 12267 0 0 0 php [562074.842304] [ 5570] 500 5570 69580 13324 1 0 0 php [562074.844533] [ 5574] 500 5574 69057 13993 0 0 0 php [562074.847082] [ 5577] 500 5577 69450 13875 1 0 0 php [562074.849309] [ 5578] 500 5578 70048 14829 1 0 0 php [562074.851653] [ 5587] 500 5587 69514 15274 1 0 0 php [562074.854140] [ 5589] 500 5589 69382 16196 1 0 0 php [562074.856537] [ 5593] 500 5593 68012 14763 0 0 0 php [562074.858820] [ 5597] 500 5597 71343 18344 0 0 0 php [562074.861408] [ 5598] 500 5598 66115 12838 1 0 0 php [562074.864285] [ 5603] 500 5603 69319 16732 1 0 0 php [562074.866510] [ 5612] 500 5612 63573 11969 0 0 0 php [562074.868753] [ 5613] 500 5613 61574 9964 1 0 0 php [562074.871009] [ 5616] 500 5616 63637 12150 0 0 0 php [562074.874032] [ 5624] 500 5624 63726 12707 1 0 0 php [562074.876563] [ 5625] 500 5625 55276 4013 1 0 0 php [562074.879156] [ 5630] 500 5630 52951 2211 0 0 0 php [562074.881476] [ 5631] 500 5631 53736 3070 1 0 0 php [562074.883968] [ 5633] 500 5633 53245 2474 0 0 0 php [562074.886305] [ 5634] 500 5634 54533 3868 1 0 0 php [562074.888585] [ 5635] 500 5635 54650 3839 0 0 0 php [562074.890872] [ 5636] 500 5636 56802 6104 0 0 0 php [562074.893228] [ 5637] 500 5637 54189 3375 1 0 0 php [562074.895530] [ 5638] 500 5638 54563 3901 0 0 0 php [562074.897784] [ 5639] 500 5639 52667 1958 1 0 0 php [562074.900122] [ 5640] 500 5640 54533 3711 0 0 0 php [562074.902475] [ 5641] 500 5641 52922 2197 0 0 0 php [562074.904697] [ 5642] 500 5642 52667 1954 1 0 0 php [562074.907013] [ 5643] 500 5643 52958 2226 0 0 0 php [562074.909381] [ 5644] 500 5644 52667 1927 0 0 0 php [562074.911628] [ 5645] 500 5645 54015 3402 1 0 0 php [562074.913914] [ 5646] 500 5646 52951 2165 0 0 0 php [562074.916614] [ 5647] 500 5647 54635 3863 1 0 0 php [562074.919109] [ 5648] 99 5648 219408 400 1 0 0 httpd [562074.921510] [ 5649] 500 5649 53928 3331 1 0 0 php [562074.924011] [ 5650] 500 5650 56653 5958 0 0 0 php [562074.926651] [ 5651] 500 5651 52951 2223 0 0 0 php [562074.929157] [ 5652] 500 5652 52735 1997 1 0 0 php [562074.954567] [ 5653] 500 5653 52457 1707 1 0 0 php [562074.957251] [ 5654] 99 5654 235792 481 1 0 0 httpd [562074.959480] [ 5660] 500 5660 53928 3321 0 0 0 php [562074.961682] [ 5686] 500 5686 54417 3563 0 0 0 php [562074.964573] [ 5712] 500 5712 52958 2216 0 0 0 php [562074.966804] [ 5713] 500 5713 53887 3247 0 0 0 php [562074.969044] [ 5714] 500 5714 52951 2206 0 0 0 php [562074.971840] [ 5715] 500 5715 53736 3103 1 0 0 php [562074.974827] [ 5716] 500 5716 52663 1938 1 0 0 php [562074.977527] [ 5717] 500 5717 56118 5385 0 0 0 php [562074.980867] [ 5726] 500 5726 57172 6396 1 0 0 php [562074.984658] [ 5727] 500 5727 52523 1760 0 0 0 php [562074.987267] [ 5728] 500 5728 52587 1869 1 0 0 php [562074.989641] [ 5730] 500 5730 52430 1711 1 0 0 php [562074.992423] [ 5731] 500 5731 53672 3061 0 0 0 php [562074.995140] [ 5732] 500 5732 52587 1860 1 0 0 php [562074.998136] [ 5733] 500 5733 52770 2016 0 0 0 php [562075.002104] [ 5734] 500 5734 53216 2537 1 0 0 php [562075.004987] [ 5735] 500 5735 54351 3615 1 0 0 php [562075.007912] [ 5737] 500 5737 54329 3665 0 0 0 php [562075.010646] [ 5738] 500 5738 53864 3231 1 0 0 php [562075.013388] [ 5750] 500 5750 53992 3332 1 0 0 php [562075.016259] [ 5751] 500 5751 53992 3356 1 0 0 php [562075.018239] [ 5754] 500 5754 52457 1707 0 0 0 php [562075.020361] [ 5755] 500 5755 53672 3008 0 0 0 php [562075.022769] [ 5758] 500 5758 53928 3334 1 0 0 php [562075.024717] [ 5759] 500 5759 53800 3153 0 0 0 php [562075.026820] [ 5760] 500 5760 52457 1715 1 0 0 php [562075.029743] [ 5761] 500 5761 53672 2999 1 0 0 php [562075.031868] [ 5767] 500 5767 52430 1709 0 0 0 php [562075.033922] [ 5768] 500 5768 53672 3015 1 0 0 php [562075.035960] [ 5772] 500 5772 52457 1708 1 0 0 php [562075.038017] [ 5774] 500 5774 53928 3307 0 0 0 php [562075.040342] [ 5777] 500 5777 52430 1709 0 0 0 php [562075.042585] [ 5780] 500 5780 52922 2200 1 0 0 php [562075.044900] [ 5781] 500 5781 29874 419 0 0 0 php [562075.046949] [ 5782] 500 5782 52430 1710 0 0 0 php [562075.049163] [ 5787] 500 5787 53736 3111 0 0 0 php [562075.051348] [ 5788] 500 5788 52428 1681 0 0 0 php [562075.053411] [ 5789] 500 5789 16226 45 1 0 0 php [562075.055455] [ 5790] 500 5790 53992 3361 0 0 0 php [562075.057385] [ 5796] 500 5796 53800 3145 0 0 0 php [562075.059354] [ 5802] 500 5802 51933 1164 0 0 0 php [562075.061403] [ 5803] 500 5803 53736 3088 1 0 0 php [562075.063319] [ 5828] 500 5828 52457 1712 1 0 0 php [562075.065328] [ 5830] 500 5830 53992 3362 1 0 0 php [562075.067304] [ 5833] 500 5833 52457 1711 0 0 0 php [562075.069272] [ 5834] 500 5834 52457 1710 0 0 0 php [562075.071342] [ 5838] 500 5838 25576 65 0 0 0 php [562075.073811] [ 5839] 99 5839 121104 248 1 0 0 httpd [562075.075876] [ 5848] 99 5848 121104 279 1 0 0 httpd [562075.078224] [ 5849] 500 5849 29230 282 1 0 0 php [562075.080532] [ 5850] 500 5850 29118 217 1 0 0 php [562075.082861] [ 5851] 500 5851 53352 2643 0 0 0 php [562075.084933] [ 5852] 99 5852 121104 281 1 0 0 httpd [562075.087091] [ 5854] 500 5854 52430 1708 0 0 0 php [562075.089180] [ 5856] 500 5856 51703 882 0 0 0 php [562075.091154] [ 5863] 99 5863 121104 286 1 0 0 httpd [562075.093112] [ 5970] 500 5970 28797 186 1 0 0 php [562075.095140] [ 5986] 0 5986 5651 60 0 0 0 upcp-running [562075.097396] [ 5988] 500 5988 29118 223 0 0 0 php [562075.099621] [ 5991] 99 5991 104720 241 0 0 0 httpd [562075.101929] [ 5993] 500 5993 3116 43 0 0 0 suphp [562075.104055] [ 5994] 500 5994 29329 408 0 0 0 php [562075.106361] [ 6026] 500 6026 29118 223 0 0 0 php [562075.108660] [ 6027] 500 6027 29118 231 1 0 0 php [562075.111327] [ 6028] 500 6028 7365 23 1 0 0 php [562075.113563] [ 6032] 99 6032 121104 279 1 0 0 httpd [562075.115682] Out of memory: Kill process 9408 (mysqld) score 44 or sacrifice child [562075.117786] Killed process 9408, UID 402, (mysqld) total-vm:2257588kB, anon-rss:171860kB, file-rss:544kB
This occurs only once or twice in the day. Otherwise the load is well within limits (<2.00). The load reaches >100 when this error occurs. As per the access logs I wasn't getting more traffic when this happened. What should I do to fix this?
