One of my server (Ubuntu 12.04, 16CPU, 32GB RAM) runs the app section of my website (Nginx 1.1.19, PHP5-FPM, Php 5.3.10, Symfony2 Web Framework)
Suddenly, users started getting HTTP 5** Errors. When I went on the server, there was huge disk IO and the culprit was [flush].
When I went through the kernel logs using dmesg, it showed the reason for [flush] was php5-fpm invoked oom-killer
Kernel Log entry from the time of Incident:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295736] php5-fpm invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295740] php5-fpm cpuset=/ mems_allowed=0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295743] Pid: 13310, comm: php5-fpm Not tainted 3.2.0-69-virtual #103-Ubuntu
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295744] Call Trace:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295751] [<ffffffff8111b971>] dump_header+0x91/0xe0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295754] [<ffffffff8111bd25>] oom_kill_process+0x85/0xb0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295757] [<ffffffff8111c0ca>] out_of_memory+0xfa/0x220
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295761] [<ffffffff81121aa3>] __alloc_pages_nodemask+0x8c3/0x8e0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295764] [<ffffffff8115a853>] alloc_pages_vma+0xb3/0x190
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295768] [<ffffffff8113ae9c>] do_anonymous_page.isra.38+0x7c/0x2f0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295772] [<ffffffff8113e921>] handle_pte_fault+0x1e1/0x200
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295777] [<ffffffff8165d9be>] ? _raw_spin_lock+0xe/0x20
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295780] [<ffffffff8113c089>] ? __pte_alloc+0xa9/0x160
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295782] [<ffffffff8113fb19>] handle_mm_fault+0x269/0x370
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295785] [<ffffffff81661504>] do_page_fault+0x184/0x550
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295789] [<ffffffff81013728>] ? __switch_to+0x138/0x360
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295793] [<ffffffff8105668d>] ? set_next_entity+0xad/0xd0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295796] [<ffffffff8105682a>] ? finish_task_switch+0x4a/0xf0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295799] [<ffffffff8165b6ec>] ? __schedule+0x3cc/0x6f0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295802] [<ffffffff8165e135>] page_fault+0x25/0x30
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295804] Mem-Info:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295805] Node 0 DMA per-cpu:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295808] CPU 0: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295809] CPU 1: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295810] CPU 2: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295812] CPU 3: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295813] CPU 4: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295814] CPU 5: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295816] CPU 6: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295817] CPU 7: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295819] CPU 8: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295820] CPU 9: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295822] CPU 10: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295823] CPU 11: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295824] CPU 12: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295826] CPU 13: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295827] CPU 14: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295828] CPU 15: hi: 0, btch: 1 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295830] Node 0 DMA32 per-cpu:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295831] CPU 0: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295833] CPU 1: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295834] CPU 2: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295836] CPU 3: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295837] CPU 4: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295838] CPU 5: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295840] CPU 6: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295841] CPU 7: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295842] CPU 8: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295844] CPU 9: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295845] CPU 10: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295846] CPU 11: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295848] CPU 12: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295850] CPU 13: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295851] CPU 14: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295853] CPU 15: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295854] Node 0 Normal per-cpu:
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295856] CPU 0: hi: 186, btch: 31 usd: 15
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295857] CPU 1: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295859] CPU 2: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295860] CPU 3: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295861] CPU 4: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295863] CPU 5: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295864] CPU 6: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295865] CPU 7: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295867] CPU 8: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295868] CPU 9: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295870] CPU 10: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295871] CPU 11: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295872] CPU 12: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295874] CPU 13: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295875] CPU 14: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295877] CPU 15: hi: 186, btch: 31 usd: 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295880] active_anon:7408050 inactive_anon:436 isolated_anon:0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295881] active_file:21159 inactive_file:33077 isolated_file:419
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295882] unevictable:0 dirty:36283 writeback:0 unstable:0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295882] free:47243 slab_reclaimable:20065 slab_unreclaimable:15299
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295883] mapped:15592 shmem:138 pagetables:28362 bounce:0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295885] Node 0 DMA free:15908kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295892] lowmem_reserve[]: 0 3760 30220 30220
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295895] Node 0 DMA32 free:114216kB min:8404kB low:10504kB high:12604kB active_anon:3651120kB inactive_anon:284kB active_file:7848kB inactive_file:38464kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3850496kB mlocked:0kB dirty:38664kB writeback:0kB mapped:272kB shmem:0kB slab_reclaimable:8444kB slab_unreclaimable:7164kB kernel_stack:208kB pagetables:10056kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:116761 all_unreclaimable? yes
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295902] lowmem_reserve[]: 0 0 26460 26460
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295905] Node 0 Normal free:58848kB min:59144kB low:73928kB high:88716kB active_anon:25981080kB inactive_anon:1460kB active_file:76788kB inactive_file:93844kB unevictable:0kB isolated(anon):0kB isolated(file):1676kB present:27095040kB mlocked:0kB dirty:106468kB writeback:0kB mapped:62096kB shmem:552kB slab_reclaimable:71816kB slab_unreclaimable:54032kB kernel_stack:2352kB pagetables:103392kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:388409 all_unreclaimable? yes
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295912] lowmem_reserve[]: 0 0 0 0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295915] Node 0 DMA: 1*4kB 0*8kB 0*16kB 1*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15908kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295922] Node 0 DMA32: 268*4kB 544*8kB 367*16kB 391*32kB 271*64kB 165*128kB 103*256kB 32*512kB 5*1024kB 0*2048kB 1*4096kB = 114240kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295929] Node 0 Normal: 13697*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 58884kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295937] 54555 total pagecache pages
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295938] 0 pages in swap cache
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295939] Swap cache stats: add 0, delete 0, find 0/0
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295940] Free swap = 0kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.295941] Total swap = 0kB
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359778] 7864304 pages RAM
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359780] 144336 pages reserved
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359781] 654931 pages shared
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359782] 7618451 pages non-shared
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359783] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359798] [ 445] 0 445 4800 221 4 0 0 rpcbind
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359801] [ 452] 108 452 5376 279 2 0 0 rpc.statd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359804] [ 482] 0 482 4308 94 0 0 0 upstart-udev-br
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359807] [ 494] 0 494 5431 327 1 -17 -1000 udevd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359809] [ 608] 0 608 5430 231 1 -17 -1000 udevd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359812] [ 624] 0 624 5430 233 3 -17 -1000 udevd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359815] [ 706] 0 706 3797 49 3 0 0 upstart-socket-
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359818] [ 882] 0 882 1816 213 6 0 0 dhclient3
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359821] [ 993] 0 993 12509 499 2 -17 -1000 sshd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359824] [ 1018] 101 1018 63430 480 4 0 0 rsyslogd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359827] [ 1040] 102 1040 5954 155 13 0 0 dbus-daemon
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359830] [ 1048] 0 1048 7445 153 9 0 0 rpc.idmapd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359832] [ 1151] 0 1151 3626 190 3 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359835] [ 1157] 0 1157 3626 189 1 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359837] [ 1162] 0 1162 3626 189 5 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359840] [ 1163] 0 1163 3626 188 1 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359842] [ 1165] 0 1165 3626 188 6 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359845] [ 1170] 0 1170 4778 211 4 0 0 cron
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359847] [ 1171] 0 1171 4227 89 12 0 0 atd
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359850] [ 1172] 0 1172 1082 149 10 0 0 acpid
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359852] [ 1178] 0 1178 3995 173 0 0 0 irqbalance
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359855] [ 1179] 103 1179 46916 650 10 0 0 whoopsie
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359858] [ 1195] 106 1195 80805 236 5 0 0 memcached
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359861] [ 1228] 109 1228 4199 99 0 0 0 nrsysmond
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359863] [ 1230] 109 1230 62851 1327 0 0 0 nrsysmond
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359866] [ 1340] 0 1340 4164 105 9 0 0 newrelic-daemon
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359869] [ 1341] 0 1341 61114 1948 13 0 0 newrelic-daemon
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359871] [ 1381] 0 1381 3310 100 6 0 0 mdadm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359874] [ 1444] 0 1444 3626 189 0 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359877] [ 1445] 0 1445 3188 177 3 0 0 getty
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359879] [ 1890] 0 1890 369259 2517 3 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359882] [ 1916] 0 1916 20056 1040 8 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359885] [ 1917] 33 1917 20473 1994 8 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359887] [ 1918] 33 1918 20399 1927 0 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359890] [ 1919] 33 1919 20454 2020 7 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359893] [ 1920] 33 1920 20201 1775 5 0 0 nginx
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359896] [ 6544] 33 6544 624267 32944 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359898] [ 7293] 33 7293 618265 25932 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359901] [ 7381] 33 7381 618555 26851 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359904] [ 8280] 33 8280 618008 27712 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359906] [ 8737] 33 8737 619699 27793 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359909] [ 8908] 33 8908 618037 26037 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359912] [ 9093] 33 9093 618471 25848 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359914] [ 9479] 33 9479 688805 75927 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359917] [ 9503] 33 9503 5636713 4312476 15 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359919] [10392] 33 10392 621196 28967 13 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359922] [10509] 33 10509 618070 26222 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359925] [10510] 33 10510 622084 30147 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359927] [10540] 33 10540 620672 29171 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359930] [10799] 33 10799 670237 78421 4 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359932] [11066] 33 11066 616125 23836 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359935] [11250] 33 11250 619613 27729 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359937] [11254] 33 11254 619605 27515 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359940] [11426] 33 11426 620759 28353 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359943] [11428] 33 11428 618500 24942 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359945] [11429] 33 11429 832300 131639 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359948] [11444] 33 11444 390293 26564 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359951] [12119] 33 12119 389579 26295 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359953] [12542] 33 12542 390336 27483 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359956] [12577] 33 12577 1770706 1177669 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359958] [13310] 33 13310 1559195 1166076 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359961] [13481] 33 13481 729336 121410 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359963] [14074] 33 14074 613987 21729 15 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359966] [14686] 33 14686 405943 20594 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359968] [14771] 33 14771 1341333 17036 10 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359971] [14814] 33 14814 375034 16151 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359973] [14888] 33 14888 374759 15618 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359976] [14917] 33 14917 382071 16366 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359979] [14919] 33 14919 373976 14146 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359981] [14982] 33 14982 373012 11466 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359984] [14983] 33 14983 372940 10379 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359986] [14984] 33 14984 374522 12917 14 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359989] [14985] 33 14985 372898 9890 7 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359991] [14986] 33 14986 373733 11996 15 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359994] [14987] 33 14987 373203 12061 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359996] [14988] 33 14988 373368 12404 13 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.359999] [14989] 33 14989 372407 8838 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360002] [14990] 33 14990 373491 11722 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360004] [14991] 33 14991 371987 8545 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360007] [14992] 33 14992 372184 9768 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360010] [14993] 33 14993 375221 12528 6 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360012] [15011] 33 15011 372181 8558 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360015] [15012] 33 15012 374863 13426 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360018] [15013] 33 15013 372497 10278 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360020] [15014] 33 15014 372489 9905 8 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360023] [15015] 33 15015 373200 11024 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360026] [15017] 33 15017 372621 9203 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360028] [15018] 33 15018 371993 9152 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360031] [15019] 33 15019 374082 10949 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360033] [15020] 33 15020 373137 10112 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360036] [15021] 33 15021 377055 14606 0 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360038] [15027] 33 15027 372243 9148 9 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360041] [15028] 33 15028 373802 11787 12 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360044] [15029] 33 15029 372094 7847 11 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360046] [15030] 33 15030 372093 7475 2 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360049] [15031] 33 15031 372606 9650 15 0 0 php5-fpm
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.360071] Out of memory: Kill process 9503 (php5-fpm) score 564 or sacrifice child
Jan 31 12:39:01 ip-10-0-0-160 kernel: [78817.363179] Killed process 9503 (php5-fpm) total-vm:22546852kB, anon-rss:17203684kB, file-rss:46220kB
Seems like the issue is with php5-fpm overcommiting the memory for some cases.
How do I ensure that such cases, doesn't create HIGH DISK IO / Server Unstable?
