17

To preface I'm using Debian Wheezy with kernel 3.2 on an AMD64 chipset. My machine has two Xeon E5-2690 cores. I set up the boot parameters so that all the cores on one CPU are dedicated to a single process. To do this I've set isolcpus=8,9,10,11,12,13,14,15 in grub.

So far, so good. Now let's say I want to use the isolated CPUs for a given command, to be simple I'll just use a simple infinite loop:

$ taskset -c 8-15 bash -c 'while true ; do echo hello >/dev/null; done' &

So far so good, top shows that core 8 spins up to near 100% utilization. Now let's say I launch that command again:

$ taskset -c 8-15 bash -c 'while true ; do echo hello >/dev/null; done' &

Now top shows that cores 9-15 remain idle and the two processes are sharing core 8. If instead I do this:

$ taskset -c 8 bash -c 'while true ; do echo hello >/dev/null; done' &

$ taskset -c 9 bash -c 'while true ; do echo hello >/dev/null; done' &

Cores 8 and 9 each get 100% utilization as they should. This only applies to isolcpus because the same taskset with cores 1-7 properly spreads the processes over the relevant cores. Furthermore "taskset -p" shows that the affinity mask for the 8-15 processes are set correctly. It appears the kernel scheduler refuses to use anything but the lowest core specified on an isolcpus affinity mask.

Now normally this wouldn't be a big deal with my above examples, just specify individual cores for each process. However I want to run a highly multithreaded application on the dedicated CPU. I want to specify the core set and have the thread pool automatically use, without having to individually reset the processor affinity for each individual thread that's spawned.

Does anyone have any idea how to get the scheduler to give me more than one core from the isolcpu set?

user79126
  • 449
  • 1
  • 4
  • 9
  • can you try using a multithread program? bash isn't multithread – c4f4t0r Feb 04 '14 at 09:46
  • 1
    Yes, that's originally what caused me to notice (my multithreaded program wasn't using more than one core). A simple python script that creates a lot of threads fails to utilize more than one core when run on the isolcpus set. (When run on the non-isolated cores it utilizes all available 8 cores). – user79126 Feb 04 '14 at 20:49
  • read this http://www.linuxtopia.org/online_books/linux_kernel/kernel_configuration/re46.html, this exclude a cpus from kernel scheduler, but after you exclude cpus you want to run process on the excluded cpus ? – c4f4t0r Feb 04 '14 at 22:39
  • 1
    The kernel will not schedule a thread or process on the isolcpu unless the processor affinity mask indicates that it should be used. Otherwise isolcpus would be the same as turning the core off, when the purpose is to reserve a core for a user-specified reason and make sure no unwanted process uses it. Taskset sets the affinity mask to use all the cores in the 8-15 range (which is set correctly when checked in /proc) so the kernel should be scheduling the process on the idle cores. – user79126 Feb 04 '14 at 23:19

1 Answers1

13

After a day of frustration I've determined a solution. This behavior seems to be an artifact of the default kernel scheduler algorithm (SCHED_OTHER for this distro/kernel). Changing the process to a different algorithm eliminates the problem, isolcpus are adequately utilized across the processes/threads.

I ended up using SCHED_RR, but I also tested SCHED_FIFO and SCHED_IDLE both of which seem to work. The process can be launched with the alternative algorithm by use of the chrt utility:

$ sudo chrt -r 1 [command]

(If you want to run as non-root you can use the setcap utility to enable CAP_SYS_NICE on the binary file related to the command)

user79126
  • 449
  • 1
  • 4
  • 9
  • 1
    Although tasksetting the affinity to cores 0,1 my java application only utilized the first core. 'sudo chrt -r 1 [command]' solved my problem too. – Barry NL Sep 19 '14 at 08:43
  • Faced same issue, however if the process using RR on isolated cores is perpetually running or requires additional process to run on same core it won't be possible. In this scenario only one process will occupy core and will free it only once it's completely done. Any idea on how to allow multiple processes to run on isolated cores? – Anton9988 May 03 '21 at 20:58