On a freshly-rebooted server why would the CPU running time of processes be huge?

Question

I rebooted our server this morning, yet there are umpteen processes that appear to have been running for 600+ days?

Could someone please shed some light?

The datetime of the machine is correct:

[root@abc youdev]# hwclock
Wed 23 Jul 2014 15:50:35 BST  -0.828434 seconds
[root@abc youdev]# date
Wed Jul 23 15:50:35 BST 2014
[root@abc youdev]#

Here's the output of 'top' and 'uptime'

[youdev@abc ~]$ top

top - 15:13:40 up  6:52,  4 users,  load average: 22.18, 21.86, 21.23
Tasks: 452 total,  11 running, 441 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  32829408k total,  4504280k used, 28325128k free,   317572k buffers
Swap: 16482296k total,        0k used, 16482296k free,   574688k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                 
  113 root      20   0     0    0    0 S  0.3  0.0 300194:22 events/14                                                                                                               
    1 root      20   0 19356 1540 1224 S  0.0  0.0  9712065h init                                                                                                                    
    2 root      20   0     0    0    0 S  0.0  0.0  4788099h kthreadd                                                                                                                
    3 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0                                                                                                             
    4 root      20   0     0    0    0 S  0.0  0.0 10237405h ksoftirqd/0

 ... snip ...

 55 root      RT   0     0    0    0 R  0.0  0.0 300194:20 migration/13                                                                                                            
 56 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/13                                                                                                             
[youdev@abc ~]$ uptime
 15:13:47 up  6:52,  4 users,  load average: 22.16, 21.86, 21.24
[youdev@abc ~]$

Running CentOS release 6.4 (Final)

Edited to include the hwclock output too – ChrisW Jul 23 '14 at 14:52 — ChrisW, Jul 23 '14 at 14:52

score 1 · Answer 1 · answered Jul 23 '14 at 14:32

1

Time+ represents CPU time, or more specifically, "Cumulative CPU time which the process and children of the process have used".

Total CPU time the task has used since it started. When 'Cumulative mode' is On, each process is listed with the cpu time that it and its dead children has used. You toggle 'Cumulative mode' with 'S', which is a command-line option and an interactive command. See the 'S' interactive command for additional information regarding this mode.

Source

Here's a link explaining CPU time if you're curious.

answered Jul 23 '14 at 14:32

jski

911
1
7
20

1

Thanks @jski, but how is the total CPU significantly higher then the total uptime of the machine? When the machine was rebooted (using the reboot command), shouldn't all processes have been killed, meaning that all processes should be 'new'? – ChrisW Jul 23 '14 at 14:36
2

Fun fact, I totally glazed over and didn't look too hard at the times. Apologies! I was able to find this bug report, which seems to indicate issues with CentOS 6.x, and the only resolution they seemed to find was a cold boot, ie. powering the machine fully down, then fully back up. Hopefully this is helpful to you: http://bugs.centos.org/view.php?id=6421 – jski Jul 23 '14 at 14:44
1

happens to us all! – ChrisW Jul 23 '14 at 14:45
Will look at that link and work out when I can get the machine cold-rebooted. Thanks for your help – ChrisW Jul 23 '14 at 14:46

score 1 · Answer 2 · answered Jul 24 '14 at 08:20

1

As suggested by @jski, a full cold reboot of the machine has worked.

The Time+ values of top have returned to (pratically) zero.

answered Jul 24 '14 at 08:20

ChrisW

283
2
9

What type of hardware is this? – ewwhite Jul 24 '14 at 08:22
@ewwhite it is a dedicated server, as an actual machine not a virtual one – ChrisW Jul 24 '14 at 08:27

score 0 · Answer 3 · answered Sep 24 '14 at 13:40

0

We faced similar problem: * SLES 11sp2 * uname -a Linux admin 3.0.26-0.7-default #1 SMP Tue Apr 17 10:27:57 UTC 2012 (3829766) x86_64 x86_64 x86_64 GNU/Linux * Dual Socket Xeon E52670

Power off and power on solved the problem.

answered Sep 24 '14 at 13:40

Wei

179
4

On a freshly-rebooted server why would the CPU running time of processes be huge?

3 Answers3

Linked