My server is CentOS7 64bit going under kernel panic
issue and rebooted after everyday around 7 to 8am.
Here is part of dmesg
[ 536.606448] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[82341.807403] INFO: task sync:6493 blocked for more than 300 seconds.
[82341.807406] Tainted: P O 3.14.39ltsi-ayan-130.0.33 #1
[82341.807407] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[82341.807408] sync D ffffffff81a0b3c0 0 6493 1 0x00000000
[82341.807410] ffff8801ed563d80 0000000000000086 0000000000009000 ffff880308cb4800
[82341.807412] ffff8803f962b000 ffff8801ed563fd8 ffff880308cb4800 ffff8803f922f600
[82341.807413] ffff8801ed563d60 ffff8801ed563e98 7fffffffffffffff ffff8801ed563e90
[82341.807415] Call Trace:
[82341.807421] [<ffffffff81802029>] schedule+0x29/0x70
[82341.807422] [<ffffffff81801169>] schedule_timeout+0x199/0x280
[82341.807426] [<ffffffff81085d42>] ? __queue_delayed_work+0xa2/0x1b0
.
.
.
[82341.807867] Code: 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 65 48 8b 04 25 70 a8 00 00 48 8b 80 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <65> 48 8b 04 25 70 a8 00 00 83 a0 3c c0 ff ff fb 0f ae f0 65 48
[82341.808450] Kernel panic - not syncing: hung_task: blocked tasks
[82341.814824] CPU: 0 PID: 75 Comm: khungtaskd Tainted: P O 3.14.39ltsi-ayan-130.0.33 #1
What I understood from dmesg, sync is a process with pid-6490 waiting for DISK operation and blocked for 300s because DISK is hogging by other process. sync goes under process state D (uninterruptible sleep).
Year back when first time I saw panic issue resolved by tweaking sysctl.conf
file then issue went away. Guided by website https://www.blackmoreops.com/2014/09/22/linux-kernel-panic-issue-fix-hung_task_timeout_secs-blocked-120-seconds-problem/
Here is current config of /etc/sysctl.conf
kernel.hung_task_timeout_secs = 300
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
I have configured daily cron job task for data backup of pgsql
DB. Data size around 200GB and some other task.
Server details:
Filesystem Size Used Avail Use% Mounted on
total 3.7T 1.1T 2.6T 30%
RAM: 16GB