1

I'm seeing strange behavior on one of my servers (running RHEL 6). There seems to be something wrong with the scheduler.

Here's the test program I'm using:

#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>

void RunClient(int i) {
  printf("Starting client %d\n", i);
  while (true) {
  }
}

int main(int argc, char** argv) {
  for (int i = 0; i < 4; ++i) {
    pid_t p_id = fork();
    if (p_id == -1) {
      perror("fork");
    } else if (p_id == 0) {
      RunClient(i);
      exit(0);
    }
  }

  return 0;
}

This machine has a lot more than 4 cores so we'd expect all processes to be running at 100%. When I check on top, the cpu usage varies. Sometimes it's split (100%, 33%, 33%, 33%), other times it's split (100%, 100%, 50%, 50%).

When I try this test on another server of ours (running RHEL 5), there are no issues (it's 100%, 100%, 100%, 100%) as expected. What's causing this and how can I fix it?

Thanks

User512
  • 11
  • 1
  • 2
  • Oh, this should be good... (grabs chair and popcorn...) – Avery Payne Apr 05 '12 at 01:25
  • Could you update your question to give some details on the underlying hardware? Is this just a typical Xeon in a box server? Or a NUMA style system? Maybe this is related to the type of architecture you have running (I realise this may be clutching at straws; kind of at the extent of my knowledge in this area). – webtoe Apr 05 '12 at 14:12
  • Yes, it's a typical Xeon server. 2 physical xeon processors (hyper-threaded). – User512 Apr 05 '12 at 14:26

2 Answers2

3

You seem to be reinventing the wheel, there are various CPU torturing programs available already, such as stress. I bet the C compiler optimizes your program in such way it does not have to constantly burn CPU during the test run.

Try with 

stress -c 4 -t 60

That would start 4 CPU intensive processes, and would run the test for 60 seconds. I bet you will see four cores running at 100%. Though if you have MUCH more cores than 4 (say, 16), then the result may vary, unless you pin the stress command to use specific cores with taskset.

Janne Pikkarainen
  • 31,454
  • 4
  • 56
  • 78
  • Why would results vary if I have much more than 4 cores (and yes, this server has a lot of cores). When I run stress, I also see some bizarre behavior. For example, running "stress -c 2 -t 60" three times it sometimes works correctly (6 processes running at 100%). But sometimes they're clustered (100%, 100%, 100%, 33%, 33%, 33%, for example). When I do "stress -c 4 -t 60" so far every time it's been 4 processes at 100%. But "stress -c 6 -t 60" has never worked (it's always split somehow with other CPUs sitting idle)... – User512 Apr 05 '12 at 12:38
  • Because the CPU scheduler has changed between RHEL 5 and 6, and even if it overall performs better, there are corner cases where you see some weird things. Did you try to pin your processes to some specific cores with `taskset` when running `stress -c 6 -t 60`? – Janne Pikkarainen Apr 05 '12 at 12:44
  • I didn't pin the processes at all. I can manually fix the problem by using taskset and then pinning the processes to different CPUs (then I see each process running at 100%), but that's obviously not a solution. The problem seems to be with the CPU load balancer, where it's not moving processes to less busy CPUs. I've run all kinds of other test variants (where each process also sleeps every once in a while between it's spins, but it still seems to be stuck on the CPU it was given at the start..) – User512 Apr 05 '12 at 13:00
1

Well there are large differences between Red Hat 5 and 6. I think the largest one that would affect your use case is the switch to the Completely Fair Scheduler. It isn't broken but will probably exhibit different behaviour in top compared to the scheduler in RHEL 5. It will allow processes that were being totally ignored previously to be allowed some processor time.

I would also think that that empty while loop is mostly a no-op and will be optimised into oblivion if you had any sort of optimisation turned on for the compilation (I'm not that much of a C programmer but I believe that to be the case; it may even happen without explicit optimisation flags on as it is totally redudant). Try putting some actual computation in there and see what happens.

As a general rule, unless you know enough to write your own kernel scheduler I would assume that it isn't a bug, but a manifestation of how the system works. Or an issue with your code/instrumentation.

webtoe
  • 1,946
  • 11
  • 12
  • The problem started with actual programs doing actually work where many were all assigned the same CPU and they were stuck (so some CPUs were at 100% while many other CPUs were completely idle). I then simplified it to this test program. Even when I change the test program to do actual work I see the same problem. – User512 Apr 05 '12 at 12:36