This morning my server went down for about an hour I'm using ubuntu 16.04LTS on AWS EC2.
some info: it's a t2.medium instance, I recently upgraded from ubuntu 14.04 to 16.04, updated lots of packages, this is a production instance, I'm using PHP Version 5.6.40-10+ubuntu16.04.1+deb.sury.org+1 and for some sites I'm using the latest php via fast-cgi php modules, my device memory /xvda1 is at 49% storage remaining.
From syslog:
Oct 11 05:09:10 ip-10-0-x-xxx systemd[1]: Starting Clean php session files...
Oct 11 05:09:10 ip-10-0-x-xxx systemd[1]: Stopping User Manager for UID 110...
Oct 11 05:09:10 ip-10-0-x-xxx systemd[19573]: Reached target Shutdown.
Oct 11 05:09:10 ip-10-0-x-xxx systemd[19573]: Starting Exit the Session...
Oct 11 05:09:10 ip-10-0-x-xxx systemd[19573]: Stopped target Default.
Oct 11 05:09:10 ip-10-0-x-xxx systemd[19573]: Stopped target Basic System.
Oct 11 05:09:10 ip-10-0-x-xxx systemd[19573]: Stopped target Timers.
Oct 11 05:09:10 ip-10-0-x-xxx systemd[19573]: Stopped target Sockets.
Oct 11 05:09:10 ip-10-0-x-xxx systemd[19573]: Stopped target Paths.
Oct 11 05:09:10 ip-10-0-x-xxx CRON[28971]: (root) CMD ( [ -
x/usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))
Oct 11 05:09:10 ip-10-0-x-xxx CRON[28972]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Oct 11 05:09:10 ip-10-0-x-xxx systemd[19573]: Received SIGRTMIN+24 from PID 28967 (kill).
Oct 11 05:09:10 ip-10-0-x-xxx systemd[1]: Stopped User Manager for UID 110.
Oct 11 05:09:10 ip-10-0-x-xxx systemd[1]: Removed slice User Slice of solr.
Oct 11 05:10:32 ip-10-0-x-xxx kernel: [488908.756151] apache2 invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
Oct 11 05:10:32 ip-10-0-x-xxx kernel: [488908.756155] apache2 cpuset=/ mems_allowed=0
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756160] CPU: 1 PID: 29346 Comm: apache2 Not tainted 4.4.0-157-generic #185-Ubuntu
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756162] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756163] 0000000000000286 3f04de0b6dc470df ffff88005a2739e8 ffffffff8140c9a1
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756166] ffff88005a273ba0 ffff8800e84e4600 ffff88005a273a58 ffffffff81219bde
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756169] ffff8800eaf61c00 ffff88005a273ba0 ffff88005a273ab0 ffff8800e86a5400
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756171] Call Trace:
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756180] [<ffffffff8140c9a1>] dump_stack+0x63/0x82
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756184] [<ffffffff81219bde>] dump_header+0x5a/0x1c3
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756189] [<ffffffff8119db2b>] oom_kill_process+0x20b/0x3d0
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756191] [<ffffffff8119df38>] out_of_memory+0x1f8/0x460
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756196] [<ffffffff811a3f73>] __alloc_pages_slowpath.constprop.88+0x943/0xaf0
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756199] [<ffffffff811a43a8>] __alloc_pages_nodemask+0x288/0x2a0
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756203] [<ffffffff811efc6c>] alloc_pages_current+0x8c/0x110
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756205] [<ffffffff81199c5b>] __page_cache_alloc+0xab/0xc0
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756208] [<ffffffff8119c580>] filemap_fault+0x160/0x440
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756211] [<ffffffff812b3016>] ext4_filemap_fault+0x36/0x50
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756215] [<ffffffff811c9d37>] __do_fault+0x77/0x110
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756218] [<ffffffff811cdcdc>] handle_mm_fault+0x125c/0x1b80
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756222] [<ffffffff81860041>] ? __schedule+0x301/0x810
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756224] [<ffffffff8186004d>] ? __schedule+0x30d/0x810
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756225] [<ffffffff81860041>] ? __schedule+0x301/0x810
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756227] [<ffffffff8186004d>] ? __schedule+0x30d/0x810
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756229] [<ffffffff81860041>] ? __schedule+0x301/0x810
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756231] [<ffffffff81860041>] ? __schedule+0x301/0x810
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756235] [<ffffffff8106edd4>] __do_page_fault+0x1a4/0x410
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756237] [<ffffffff8106f062>] do_page_fault+0x22/0x30
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756241] [<ffffffff81867ae8>] page_fault+0x28/0x30
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756242] Mem-Info:
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756247] active_anon:919500 inactive_anon:18067 isolated_anon:0
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756247] active_file:620 inactive_file:764 isolated_file:0
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756247] unevictable:0 dirty:0 writeback:0 unstable:0
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756247] slab_reclaimable:8517 slab_unreclaimable:10839
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756247] mapped:25419 shmem:36238 pagetables:21622 bounce:0
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756247] free:21367 free_pcp:140 free_cma:0
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756251] Node 0 DMA free:15820kB min:268kB low:332kB high:400kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:28kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:8kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756256] lowmem_reserve[]: 0 3704 3893 3893 3893
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756260] Node 0 DMA32 free:64756kB min:64040kB low:80048kB high:96060kB active_anon:3513132kB inactive_anon:69560kB active_file:2472kB inactive_file:3028kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3915776kB managed:3834888kB mlocked:0kB dirty:0kB writeback:0kB mapped:95428kB shmem:137276kB slab_reclaimable:32652kB slab_unreclaimable:38928kB kernel_stack:4192kB pagetables:83352kB unstable:0kB bounce:0kB free_pcp:560kB local_pcp:400kB free_cma:0kB writeback_tmp:0kB pages_scanned:37356 all_unreclaimable? yes
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756265] lowmem_reserve[]: 0 0 189 189 189
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756268] Node 0 Normal free:4892kB min:3268kB low:4084kB high:4900kB active_anon:164868kB inactive_anon:2708kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:262144kB managed:193808kB mlocked:0kB dirty:0kB writeback:0kB mapped:6240kB shmem:7676kB slab_reclaimable:1416kB slab_unreclaimable:4428kB kernel_stack:720kB pagetables:3136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Then I get an huge list of this:
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174008] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174009] 37420 total pagecache pages
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174011] 0 pages in swap cache
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174012] Swap cache stats: add 0, delete 0, find 0/0
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174013] Free swap = 0kB
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174014] Total swap = 0kB
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174015] 1048477 pages RAM
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174016] 0 pages HighMem/MovableOnly
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174017] 37328 pages reserved
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174018] 0 pages cma reserved
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174019] 0 pages hwpoisoned
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174020] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174025] [ 275] 0 275 10189 689 21 3 0 0 systemd-journal
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174028] [ 303] 0 303 11224 309 23 3 0 -1000 systemd-udevd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174030] [ 532] 0 532 7240 179 18 3 0 0 systemd-logind
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174032] [ 534] 0 534 7470 52 20 3 0 0 cgmanager
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174034] [ 536] 0 536 6512 49 18 3 0 0 atd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174037] [ 544] 0 544 6933 71 18 3 0 0 cron
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174039] [ 550] 102 550 10762 176 25 3 0 -900 dbus-daemon
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174041] [ 600] 0 600 1099 16 7 3 0 0 acpid
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174043] [ 621] 0 621 68655 244 36 3 0 0 accounts-daemon
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174045] [ 626] 101 626 65157 313 29 3 0 0 rsyslogd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174047] [ 670] 0 670 4869 65 15 3 0 0 irqbalance
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174050] [ 689] 0 689 13193 2548 29 3 0 0 munin-node
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174052] [ 691] 0 691 4031 216 11 3 0 0 dhclient
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174054] [ 698] 0 698 69861 342 39 3 0 0 polkitd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174056] [ 735] 0 735 9667 180 22 3 0 0 monit
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174058] [ 1228] 0 1228 133721 6781 85 5 0 -500 dockerd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174060] [ 1248] 0 1248 109978 1466 139 4 0 0 php-fpm7.2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174063] [ 1253] 0 1253 43332 2030 52 3 0 0 unattended-upgr
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174065] [ 1255] 0 1255 91709 3642 139 3 0 0 php-fpm7.0
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174067] [ 1321] 107 1321 28735 291 53 3 0 0 opendkim
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174069] [ 1324] 0 1324 16379 177 35 3 0 -1000 sshd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174072] [ 1327] 107 1327 88165 827 62 3 0 0 opendkim
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174074] [ 1352] 0 1352 43196 733 52 5 0 0 nginx
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174076] [ 1353] 33 1353 43196 733 54 5 0 0 nginx
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174078] [ 1354] 33 1354 43196 733 54 5 0 0 nginx
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174080] [ 1355] 33 1355 43196 733 54 5 0 0 nginx
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174087] [ 1362] 0 1362 3664 33 12 3 0 0 agetty
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174089] [ 1371] 33 1371 109978 1327 125 4 0 0 php-fpm7.2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174091] [ 1372] 33 1372 109978 1327 125 4 0 0 php-fpm7.2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174093] [ 1373] 33 1373 112297 9698 150 3 0 0 php-fpm7.0
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174095] [ 1374] 33 1374 112315 10265 151 3 0 0 php-fpm7.0
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174098] [ 1492] 0 1492 93681 2281 40 5 0 -500 docker-containe
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174100] [ 1506] 0 1506 16352 120 24 3 0 0 master
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174102] [ 1974] 111 1974 27509 166 26 3 0 0 ntpd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174105] [27112] 33 27112 111777 8536 150 3 0 0 php-fpm7.0
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174107] [30935] 0 30935 146764 2750 54 4 0 0 fail2ban-server
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174109] [ 6574] 0 6574 122194 3926 179 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174111] [27782] 33 27782 44694 2331 70 3 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174114] [27811] 108 27811 23037 213 45 3 0 0 zabbix_agentd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174116] [27812] 108 27812 23037 212 45 3 0 0 zabbix_agentd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174118] [27813] 108 27813 23037 211 45 3 0 0 zabbix_agentd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174120] [27814] 108 27814 23037 211 45 3 0 0 zabbix_agentd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174122] [27815] 108 27815 23037 211 45 3 0 0 zabbix_agentd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174124] [27816] 108 27816 23037 218 45 3 0 0 zabbix_agentd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174127] [27291] 106 27291 16911 131 25 3 0 0 qmgr
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174129] [27413] 106 27413 20177 184 30 3 0 0 tlsmgr
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174131] [26417] 106 26417 16869 112 25 3 0 0 pickup
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174133] [27850] 33 27850 163843 26103 224 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174136] [27856] 33 27856 161347 23458 215 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174138] [27895] 33 27895 162572 24791 221 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174140] [28374] 33 28374 163657 33629 236 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174142] [28377] 33 28377 142967 31279 231 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174144] [28379] 33 28379 144883 32799 235 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174146] [28423] 33 28423 162756 24751 222 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174149] [28429] 33 28429 161587 31448 232 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174151] [28434] 33 28434 161233 23432 219 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174153] [28463] 33 28463 163658 33523 236 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174155] [28464] 33 28464 122401 2826 177 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174157] [28467] 33 28467 143221 31432 231 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174159] [28468] 33 28468 122473 2965 177 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174161] [28469] 33 28469 129756 26267 220 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174164] [28482] 33 28482 161283 22980 219 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174166] [28483] 33 28483 163618 25643 223 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174168] [28484] 33 28484 163367 25141 223 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174170] [28486] 33 28486 143209 31129 231 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174172] [28488] 33 28488 153425 24734 220 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174174] [28492] 33 28492 161351 23526 215 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174176] [28500] 33 28500 163574 25769 221 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174178] [28501] 33 28501 163577 26837 224 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174180] [28503] 33 28503 142863 23211 214 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174182] [28865] 33 28865 161160 23608 218 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174184] [28873] 33 28873 161345 23207 215 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174186] [28890] 33 28890 163621 25344 223 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174188] [28891] 33 28891 122250 2748 166 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174190] [28892] 33 28892 162593 24527 217 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174193] [28894] 33 28894 127346 7830 186 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174195] [28899] 33 28899 163557 25511 223 4 0 0 apache2
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.936086] INFO: task apache2:28423 blocked for more than 120 seconds.
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.943206] Not tainted 4.4.0-157-generic #185-Ubuntu
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.948135] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955958] apache2 D ffff8800e674fc58 0 28423 6574 0x00000000
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955963] ffff8800e674fc58 0000000000000001 ffff880021adc600 ffff88001dac8000
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955966] ffff8800e6750000 ffff880034195984 ffff88001dac8000 00000000ffffffff
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955969] ffff880034195988 ffff8800e674fc70 ffffffff81860585 ffff880034195980
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955971] Call Trace:
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955981] [<ffffffff81860585>] schedule+0x35/0x80
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955989] [<ffffffff818608de>] schedule_preempt_disabled+0xe/0x10
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955992] [<ffffffff81862777>] __mutex_lock_slowpath+0xb7/0x130
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955994] [<ffffffff8186280f>] mutex_lock+0x1f/0x30
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955998] [<ffffffff8122bbf2>] path_openat+0x512/0x1360
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.956001] [<ffffffff8122e611>] do_filp_open+0x91/0x100
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.956004] [<ffffffff8123c2e7>] ? __alloc_fd+0xc7/0x190
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.956007] [<ffffffff8121c8c8>] do_sys_open+0x138/0x2a0
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.956009] [<ffffffff8121ca4e>] SyS_open+0x1e/0x20
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.956012] [<ffffffff81864f1b>] entry_SYSCALL_64_fastpath+0x22/0xcb
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.956026] INFO: task apache2:29254 blocked for more than 120 seconds.
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.962952] Not tainted 4.4.0-157-generic #185-Ubuntu
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.977383] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.992125] apache2 D ffff880013db3bf8 0 29254 6574 0x00000000
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.992131] ffff880013db3bf8 0000000000000000 ffffffff81e13500 ffff8800e8701c00
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.992135] ffff880013db4000 ffff880034195984 ffff8800e8701c00 00000000ffffffff
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.992137] ffff880034195988 ffff880013db3c10 ffffffff81860585 ffff880034195980
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.992140] Call Trace:
Possible reasons: Cron is running sessionclean at the same time as systemd starts clean php session files? It's simply not cleaning php sessions correctly? My RAM is too low?
free -m #see memory info
total used free shared buff/cache available
Mem: 3949 1710 1397 141 842 1793
Swap: 0 0 0
Also since I updated the OS my sites are not using php5 anymore but that's an cron job from before the update that migrated over, maybe that could be an issue? Thanks!
