From the man page of top:
sy: time running kernel processes
The sy-value has peaks with values above 50, and then goes down to about 5 again.
What could be the root of it?
Here are two snapshots of top and the output of vmstat 1 5
top
Sy-value 72.2:
top - 15:05:31 up 73 days, 1:41, 4 users, load average: 5,64, 4,89, 4,36
Tasks: 222 total, 23 running, 199 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0,1 us, *74,9 sy*, 0,0 ni, 13,7 id, 11,1 wa, 0,0 hi, 0,2 si, 0,0 st
KiB Mem: 20492080 total, 19555680 used, 936400 free, 3243624 buffers
KiB Swap: 2095100 total, 1770960 used, 324140 free, 14142880 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14292 wwwrun 20 0 124m 6136 1332 R 78,0 0,0 0:02.49 httpd2-prefork
14293 wwwrun 20 0 124m 6136 1332 R 75,7 0,0 0:02.30 httpd2-prefork
9121 root 20 0 0 0 0 S 67,1 0,0 0:05.86 kworker/8:0
12477 root 20 0 0 0 0 S 56,8 0,0 0:05.69 kworker/0:2
14296 postgres 20 0 7226m 1112 664 R 31,4 0,0 0:00.95 postgres
14285 postgres 20 0 7233m 7344 5564 S 24,1 0,0 0:01.91 postgres
49 root rt 0 0 0 0 S 23,8 0,0 32:26.38 migration/8
26159 root 20 0 0 0 0 S 23,8 0,0 0:35.23 kworker/10:0
10590 modcron 30 10 84372 8384 1380 S 19,8 0,0 3:06.98 modcron-cmd.py
14291 postgres 20 0 7233m 5036 3528 R 19,2 0,0 0:00.74 postgres
1829 postgres 20 0 7225m 16m 16m S 13,5 0,1 9:41.58 postgres
24583 root 20 0 0 0 0 S 13,2 0,0 1:21.04 kworker/2:0
13582 modwork+ 39 19 313m 99m 7740 S 12,6 0,5 0:25.06 archiviere_bele
1760 rabbitmq 20 0 2886m 36m 2024 S 10,9 0,2 1118:14 beam.smp
8 root rt 0 0 0 0 R 10,6 0,0 116:55.84 migration/0
14289 modwork+ 39 19 42116 3688 2464 R 10,2 0,0 0:00.99 ssh
12 root rt 0 0 0 0 S 9,3 0,0 17:01.30 watchdog/1
32089 root 20 0 135m 33m 1292 R 8,6 0,2 13:46.11 rsync
28100 modwork+ 20 0 698m 159m 16m S 7,9 0,8 1:51.36 httpd2-prefork
48 root 20 0 0 0 0 S 6,6 0,0 7:01.48 ksoftirqd/8
288 root 20 0 0 0 0 S 6,3 0,0 1:57.83 flush-253:0
1828 postgres 20 0 7225m 82m 81m S 5,9 0,4 2:36.61 postgres
47 root rt 0 0 0 0 S 5,6 0,0 15:33.23 watchdog/8
1415 root 20 0 0 0 0 S 4,0 0,0 0:39.58 kworker/4:0
14287 postgres 20 0 7233m 5052 3544 R 4,0 0,0 0:01.01 postgres
13967 postgres 20 0 7273m 271m 233m R 3,6 1,4 0:39.91 postgres
5856 root 20 0 0 0 0 S 3,0 0,0 0:09.37 kworker/1:1
19049 root 20 0 0 0 0 S 2,3 0,0 0:25.17 kworker/3:1
1494 root 20 0 0 0 0 S 2,0 0,0 5:56.14 flush-253:16
478 root 20 0 0 0 0 S 1,7 0,0 9:09.84 jbd2/vdb1-8
17830 modwork+ 20 0 268m 20m 6912 S 1,3 0,1 52:06.14 celery
17861 root 20 0 123m 7116 4024 S 1,0 0,0 5:39.92 httpd2-prefork
3 root 20 0 0 0 0 S 0,3 0,0 52:22.43 ksoftirqd/0
10 root 20 0 0 0 0 S 0,3 0,0 83:14.45 rcu_sched
203 root 0 -20 0 0 0 R 0,3 0,0 67:48.68 kworker/0:1H
509 root 20 0 7304 1264 660 S 0,3 0,0 18:06.80 haveged
13529 modwork+ 20 0 19520 1844 1196 R 0,3 0,0 0:00.71 top
Sy-value 41.2:
top - 15:09:09 up 73 days, 1:45, 4 users, load average: 3,03, 3,98, 4,12
Tasks: 221 total, 6 running, 214 sleeping, 0 stopped, 1 zombie
%Cpu(s): 0,2 us, *41,2 sy*, 0,1 ni, 51,7 id, 6,5 wa, 0,0 hi, 0,2 si, 0,1 st
KiB Mem: 20492080 total, 19548000 used, 944080 free, 3257416 buffers
KiB Swap: 2095100 total, 1771524 used, 323576 free, 14183352 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15716 modwork+ 39 19 85704 20m 2324 R 86,6 0,1 0:04.08 convert
14 root rt 0 0 0 0 S 68,8 0,0 51:04.96 migration/1
49 root rt 0 0 0 0 S 62,9 0,0 32:29.92 migration/8
14364 root 20 0 0 0 0 S 61,3 0,0 0:06.00 kworker/0:0
15719 modcron 30 10 36900 5444 2332 R 60,3 0,0 0:01.86 check_load
10590 modcron 30 10 84372 8384 1380 S 37,3 0,0 3:09.08 modcron-cmd.py
15718 postgres 20 0 7233m 7600 5792 R 30,8 0,0 0:01.68 postgres
15717 postgres 20 0 7233m 9764 7480 S 17,5 0,0 0:01.92 postgres
19 root rt 0 0 0 0 S 16,5 0,0 57:53.36 migration/2
30709 root 20 0 0 0 0 S 13,9 0,0 0:11.11 kworker/5:0
19049 root 20 0 0 0 0 S 13,3 0,0 0:25.90 kworker/3:1
8 root rt 0 0 0 0 R 13,0 0,0 117:09.89 migration/0
17 root rt 0 0 0 0 S 10,4 0,0 15:07.58 watchdog/2
1519 ntp 20 0 27092 1616 1388 S 6,8 0,0 6:47.32 ntpd
1415 root 20 0 0 0 0 S 5,2 0,0 0:41.00 kworker/4:0
1524 rabbitmq 20 0 7344 316 316 S 5,2 0,0 1:08.64 epmd
32089 root 20 0 135m 35m 1292 D 5,2 0,2 15:05.85 rsync
15026 wwwrun 20 0 0 0 0 Z 4,2 0,0 0:00.17 httpd2-prefork
26159 root 20 0 0 0 0 S 4,2 0,0 0:36.43 kworker/10:0
27992 modwork+ 20 0 698m 151m 16m S 4,2 0,8 1:57.30 httpd2-prefork
478 root 20 0 0 0 0 S 3,2 0,0 9:10.07 jbd2/vdb1-8
15432 modwork+ 20 0 19392 1792 1184 R 3,2 0,0 0:00.20 top
13 root 20 0 0 0 0 S 2,3 0,0 13:07.37 ksoftirqd/1
203 root 0 -20 0 0 0 S 1,9 0,0 67:51.00 kworker/0:1H
33 root 20 0 0 0 0 S 0,6 0,0 7:56.53 ksoftirqd/5
48 root 20 0 0 0 0 S 0,6 0,0 7:01.55 ksoftirqd/8
509 root 20 0 7304 1264 660 S 0,6 0,0 18:07.12 haveged
1760 rabbitmq 20 0 2886m 36m 2024 S 0,6 0,2 1118:17 beam.smp
1826 postgres 20 0 7226m 3,3g 3,3g D 0,6 16,7 32:19.05 postgres
27950 modwork+ 20 0 699m 159m 16m S 0,6 0,8 2:17.94 httpd2-prefork
3 root 20 0 0 0 0 S 0,3 0,0 52:22.69 ksoftirqd/0
10 root 20 0 0 0 0 S 0,3 0,0 83:16.95 rcu_sched
1453 modwork+ 20 0 223m 16m 5228 S 0,3 0,1 79:09.48 celery
17830 modwork+ 20 0 268m 20m 6912 S 0,3 0,1 52:08.47 celery
1 root 20 0 45752 2964 1804 S 0,0 0,0 5:47.74 systemd
vmstat
vmstat output:
procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
29 1 1786384 550668 3326544 14359188 0 0 22 11 0 0 2 1 97 0 0
35 0 1786384 558988 3323304 14355956 0 0 72 60 3134 1542 3 97 0 0 0
29 0 1786384 556404 3323300 14354380 0 0 8 0 3580 1066 1 99 0 0 0
19 1 1786384 579688 3323528 14340016 0 0 228 28 4495 5194 4 90 4 2 0
22 1 1786384 559376 3323724 14338364 0 0 188 252 3240 2890 9 85 4 2 0
I read it like this: the system is not swapping.
Linux version
foo-work:~ # uname -a
Linux foo-work.example.com 3.7.10-1.45-default #1 SMP
Tue Dec 16 20:27:58 UTC 2014 (4c885a1) x86_64 x86_64 x86_64 GNU/Linux
Reasons for high sy?
Here are some reasons I know why sy could be high in general. All don't apply to my situation.
Spawning a lot of new processes
Up to now I have only seen high sy-value if shell scripts start several new processes per second like grep, cut, sed, ...
But on this system I think it is not the case.
/dev/urandom
AFAIK no process reads it.
...
What is causing high sy here?
Solution: Hypervisor with no RAM for himself.
I finally found the root of the problem. The host was a virtual machine running on kvm/qemu. Some sysadmin had added new virtual hosts to the hypervisor and did some foolish calculation: I have 32GByte RAM on the hypervisor, let's give the one machine 20GByte and the other 12GByte. Result: The hypervisor itself had not RAM left!
