4

I understand load averages on Linux, but am a bit mystified by the load averages on a legacy Solaris 10 machine my app runs on. The load averages seem impossibly high. Here's the output.

[netcool1 (root)/]$ uptime
 11:49am  up 580 day(s), 10:51,  3 users,  load average: 35.50, 38.54, 39.03
[netcool1 (root)/]$ uname -a
SunOS netcool1 5.10 Generic_139555-08 sun4u sparc SUNW,Sun-Fire-V245
[netcool1 (root)/]$ psrinfo -v
Status of virtual processor 0 as of: 01/11/2012 11:52:52
  on-line since 06/10/2010 01:58:29.
  The sparcv9 processor operates at 1504 MHz,
        and has a sparcv9 floating point processor.
Status of virtual processor 1 as of: 01/11/2012 11:52:52
  on-line since 06/10/2010 01:58:27.
  The sparcv9 processor operates at 1504 MHz,
        and has a sparcv9 floating point processor.
[netcool1 (root)/]$

I don't see how you can have a load average of 35 on a two-processor system. That seems incredibly high to me. When I view the processes with top, the system is about 60-70% idle. Could someone help explain this?

vmstat 10 6

kthr      memory            page            disk          faults      cpu
r b w   swap  free  re  mf pi po fr de sr rm s0 s2 --   in   sy   cs us sy id
3 0 0 8747008 5562704 865 1866 188 63 63 0 0 -0 9 40 0 762 8588 1495 26  8 66
0 0 0 7715256 5068016 73 23 5 17 17  0  0  0 110 66 0 1135 3888 9855 59 12 30
0 0 0 7717936 5069128 0  5  0  6  6  0  0  0 100 4  0 1071 3273 4191 62  6 32
0 0 0 7717952 5027912 0 11649 0 5 5  0  0  0 115 21 0 1017 26370 3260 32 15 53
102 1 0 7717952 4979088 0 1 0  0  0  0  0  0 112 4  0  900 3464 7683 15  9 76
0 0 0 7717952 4978936 0  1  0  0  0  0  0  0 105 4  0  886 3379 8698 19  9 72
coding_hero
  • 221
  • 3
  • 5
  • 11

4 Answers4

4

On older-solaris, the load-average is the average number of runnable and running threads. In other words, it is the number of threads running on the CPUs, plus the number of threads in the run queue, waiting for CPUs, averaged over time.

So... a CPU that completed processing 10 threads for the last second... and had 5 more waiting to be processed would show 15.

In contrast...

Linux load averages are calculated as "overload" of a CPU... i.e. during the last period of time, how many threads were waiting for CPU time over how many were completed. (as a percentage)

So... a CPU that completed processing 10 threads for the last second... and had 5 more waiting to be processed would show 0.5

In Solaris 10... they changed the formula a bit... and I'm not 100% sure what it entails, but it should be pretty-close.

TheCompWiz
  • 7,349
  • 16
  • 23
  • 1
    In Solaris 10... the "exact" method for calculating load is explained as "The calculation is made by summing high-resolution user time, system time, and thread wait time, then processing this total to generate averages with exponential decay." – TheCompWiz Jan 11 '12 at 17:32
  • Hehe, yah, I read that too. Clear as mud. – coding_hero Jan 11 '12 at 17:47
  • So, is a 35 bad on Solaris 10? Does that mean that each processor is handling 17.5 threads each? How do I interpret this? – coding_hero Jan 11 '12 at 17:56
  • Wikipedia does not confirm that Linux only counts "overload". Linux counts running+runnable as any Unix (Solaris). The only difference is that Linux also counts uninterruptibly sleeping threads, but this does not pertain to this question. http://en.wikipedia.org/wiki/Load_%28computing%29 – kubanczyk Jan 11 '12 at 19:57
  • The "should be pretty-close" in this answer made me think that Solaris 10 has changed its definition of load average to include tasks waiting for things other than CPU, to closer match Linux. But given that [Brendan Gregg (who I consider an expert in this area) doesn't mention it in a 2011 article](http://dtrace.org/blogs/brendan/2011/06/24/load-average-video/), and the [other answer here](https://serverfault.com/a/839769/58957) I think you actually meant to say that the formula didn't change significantly in Solaris 10... – Nickolay Sep 12 '18 at 11:45
3

Quite a late reply but the accepted answer still has incorrect statements, is missing parts of the point, and suggest statistics lies while there is no reason here not to trust the ones reported by the OS.

Here is an in-depth explanation of the statistics observed.

The load average reported by uptime and other commands is a floating average of 1, 5, and 15 minutes of the average number of threads waiting for a CPU (run queue) plus the average number of threads actually running on a CPU.

The idea is to smooth the display of the run queue size and running processes count which is often very irregular.

The run queue size is the first column of vmstat output (r). Any non zero value here means that your system would have run faster should it had more CPUs available.

vmstat first data line shows the average since last boot. An average of 3 threads were waiting on your machine before you launched vmstat. This value is generally meaningless being biased by long inactivity periods like week-ends and other non working hours:

r b w   swap  free  re  mf pi po fr de sr rm s0 s2 --   in   sy   cs us sy id
3 0 0 8747008 5562704 865 1866 188 63 63 0 0 -0 9 40 0 762 8588 1495 26  8 66
↑

All other samples show an empty run queue except the second last one which shows a whopping average number of 102 threads:

102 1 0 7717952 4979088 0 1 0  0  0  0  0  0 112 4  0  900 3464 7683 15  9 76
 ↑                                                                         ↑

The CPU is nevertheless 76% idle during this 10 seconds sample which is what puzzles you.

To understand the apparent discrepancy, you need to understand 102 is the average value for this sample. One way to get it is to assume the run queue was holding 1020 threads during one second, then was empty during the remaining 9 seconds. Any other combination leading to that 102 number is also conceivable, like 204 threads during 5 seconds and none during the other 5, and so on.

However, from vmstat last column, we know your system was 76% idle during this period. A plausible value accommodating the average run queue and the idle CPU would be 408 threads competing during 2.4 seconds (100% busy CPUs) and no thread active during 7.6 seconds leading (0% busy CPU).

Now we know there was definitely a CPU contention. Should you have had more than 408 CPUs available instead of 2 and assuming all thread would have been able to run full speed in parallel, you would have reduced these 2.5 seconds to around 6 ms. This would have had a dramatic effect on interactive application but not that much on a batch job as the remaining time wouldn't have benefit from the extra CPUs anyway.

Bottom line:

If your application is interactive, your system is seriously overloaded, if not, it is between slightly overloaded and just "regular".

There is a tradeoff to consider, 6 ms is likely "too good" for a response time and 408 CPU too expensive. Assuming 60 ms is a more reasonable goal, around 40 CPUs might do the job and of course if 2.5 s is fine, your system is behaving correctly.

Generally, a best practice is to assume there is a contention when the overall average run queue size exceeds the number of CPUs, here ~37 vs 2. Figuring out whether it is a problem or not cannot be done without analyzing what applications and threads are affected and how it impacts the platform operation.

jlliagre
  • 8,691
  • 16
  • 36
1

The "load" is normally an average of the first column of vmstat (column r, the run queue). The first load is averaged over 1 minute, second over 5 minutes, and the last over 15 minutes. As you can see, in your system vmstat at one point reported no less than 102 threads woken up to use the processor (probably some massively multi-threaded application).

But no worries, as certainly this burst of workload has been handled, and run queue went back to zero on the next probe and continuing. The V245 has two processors, each single-core and single-thread, so it can run two threads at the same time (i.e. r=2 means no thread needed to wait for processor time).

Statistically this could translate to an average of 35, but as you can see this value says very very little about actual system usage. Adage says "there are three kinds of lies: lies, damned lies, and statistics", and I think this serves well as a conclusion.

kubanczyk
  • 13,502
  • 5
  • 40
  • 55
  • That actually makes a lot of sense, considering what this server does. I have a multithreaded app that connects to network devices, another that waits for and interprets snmp traffic, and a third that handles user connections. – coding_hero Jan 11 '12 at 20:13
  • 1
    Your definition of the load average is incorrect, not only processes waiting in the run queue but also processes effectively running are taken into account in the calculation. You should also remove the incorrect statement about the UltraSPARC IIIi decribed as a four-way processor. It is actually a single core, single thread CPU so the V245 can only run two threads in parallel, not eight. Also, you write r=8 means no thread need to wait. This is incorrect. As soon as r != 0, a thread is waiting for a CPU. – jlliagre Jan 12 '12 at 00:39
  • It seems to me that traditionally run queue (vmstat `r` column) includes both running and runnable, isn't it? The confusion stems from the name "queue" which suggests "runnable but not actually running". You are right about IIIi and I am correcting that right away - it supports four-way systems (4 processors, not 4 threads). – kubanczyk Jan 12 '12 at 08:15
  • The r queue is only containing runnable processes waiting. Running processes aren't in queue(s), they are on CPUs. Your statement "r=2 means no thread needed to wait for processor time" is still incorrect. – jlliagre Jan 12 '12 at 13:04
-1

Load averages >>1 and high idle percentage are usually a sign of heavy disk i/o. This might be useful to find out why.

Community
  • 1
wallenborn
  • 257
  • 1
  • 2
  • Next time read the question. – TheCompWiz Jan 11 '12 at 17:20
  • Why so rude? coding_hero asks how his system can have load avg of 35 with idle% of 60-70, i answer "probably heavy io", and point to a related question, what's wrong with that? – wallenborn Jan 11 '12 at 17:31
  • He was asking the difference between linux load averages vs solaris load averages. (the definitions and how they compare) Not what might cause high-load. – TheCompWiz Jan 11 '12 at 17:35
  • While not directly what I asked, it is still useful info. Adding the vmstat output above. – coding_hero Jan 11 '12 at 17:59