6

I am monitoring a Cpanel (centos) server which have a 2-core CPU (4 virtual CPU cores) and it seems to be overloaded because I got this values using top:

load average: 11.80, 13.30, 13.02
Cpu(s): 42.2%us, 11.7%sy,  0.0%ni, 35.6%id, 10.1%wa,  0.1%hi,  0.3%si,  0.0%st

But if I look at the proccess list (using top or ps) no proccess is using more that 1%

Also, the sum of the proccess CPU usage (%) equals 4 and if I even assume that 0% values are rounded numbers, and change it to 0.04 (which rounded using 1 decimal digit is 0) the sum is 11 (still less than 100%).

How can I correctly interpret this data?, are there some hidden proccesses which are overloading my cpu.

Falcon Momot
  • 24,975
  • 13
  • 61
  • 92
Pablo
  • 99
  • 1
  • 3

3 Answers3

8

On Linux, blocked processes also contribute to the load averages. The command ps -Al lists all processes. In the second column (S for State) of its output you will find the process states. Most often I have processes waiting for disk "D", which are counted towards the load averages.

The full list of states from the ps man page is

   D    Uninterruptible sleep (usually IO)
   R    Running or runnable (on run queue)
   S    Interruptible sleep (waiting for an event to complete)
   T    Stopped, either by a job control signal or because it is being
        traced.
   W    paging (not valid since the 2.6.xx kernel)
   X    dead (should never be seen)
   Z    Defunct ("zombie") process, terminated but not reaped by its
        parent.

Sample output

F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
4 S     0     1     0  0  80   0 -  4906 poll_s ?        00:00:23 init
1 S     0     2     0  0  80   0 -     0 kthrea ?        00:00:02 kthreadd
1 R     0     3     0 99  80   0 -     0        ?        01:00:02 runner
1 D     0     4     0  1  80   0 -     0        ?        01:00:02 loader

If these were your only processes you we see a load of about 2, 1 for the CPU hog "runner" and another 1 for the loader which is waiting for disk.

Very precise is the information available on Wikipedia

An idle computer has a load number of 0. Each process using or waiting for CPU (the ready queue or run queue) increments the load number by 1. Most UNIX systems count only processes in the running (on CPU) or runnable (waiting for CPU) states. However, Linux also includes processes in uninterruptible sleep states (usually waiting for disk activity), which can lead to markedly different results if many processes remain blocked in I/O due to a busy or stalled I/O system.1 This, for example, includes processes blocking due to an NFS server failure or to slow media (e.g., USB 1.x storage devices). Such circumstances can result in an elevated load average, which does not reflect an actual increase in CPU use (but still gives an idea on how long users have to wait).

Peter G.
  • 333
  • 5
  • 12
1

The top info you provided doesn't necessarily mean overloading:

  • the CPU is 35% idle
  • the load averages aren't necessarily too big (depends on the server's intended usage)
  • the RAM and swap info are missing

Or rather, if by overloading you mean hitting some limitation, there can be various aspects of it: CPU limitation, network and/or disk I/O limitations, memory usage limitations, etc.

You shouldn't try to match various CPU load/usage views - they typically mean different things and the views are also collected at different timestamps (stats collection is not atomic):

  • load average means number of jobs in the running queue, not CPU usage: https://stackoverflow.com/questions/21617500/understanding-load-average-vs-cpu-usage
  • the CPU % usage numbers in the process context don't have to add up to 100% for various reasons, here are just a few:
    • the CPU doesn't spend all its cycles in process space
    • various cycles spent in a process context is counted differently on the overall CPU % usage line (the same process could have been in both running or waiting on I/O states during the accounting interval, thus contributing to both %us and %wa numbers on the overall CPU % usage line)
    • the CPU could have spent cycles on processes nu longer running, those would be counted in the overall CPU % usage line, but not present on any of the process lines
Community
  • 1
Dan Cornilescu
  • 327
  • 4
  • 10
-1

Peter is right. But that did not answer your questions. Give it 12 logical CPUs so the load drops below the number of CPUs. That way no process or thread has to wait additional CPU cycles to get its execution time.

Turn on the thread view in top, too.

I suspect you have some multi threaded processes there.

11% System time might indicate a network bottleneck.

Nils
  • 7,657
  • 3
  • 31
  • 71
  • 1
    My theory is that the CPUs aren't overloaded or saturated. Instead processes are waiting for "fast" IO, e.g. disk or NFS, which is also reported as load. Adding more CPU power won't help in this case. – Peter G. Aug 11 '15 at 13:40
  • @Peter G. NFS suits. But there the load can be caused by the reassebly process for large transfers, wich profits from more (not faster) CPUs, too. I tried to saturate a 10gb line with an nfs-server. This topped out at 16 cpus. Then the interrupt handling became the bottleneck. – Nils Aug 11 '15 at 21:09