To preface I'm using Debian Wheezy with kernel 3.2 on an AMD64 chipset. My machine has two Xeon E5-2690 cores. I set up the boot parameters so that all the cores on one CPU are dedicated to a single process. To do this I've set isolcpus=8,9,10,11,12,13,14,15 in grub.
So far, so good. Now let's say I want to use the isolated CPUs for a given command, to be simple I'll just use a simple infinite loop:
$ taskset -c 8-15 bash -c 'while true ; do echo hello >/dev/null; done' &
So far so good, top shows that core 8 spins up to near 100% utilization. Now let's say I launch that command again:
$ taskset -c 8-15 bash -c 'while true ; do echo hello >/dev/null; done' &
Now top shows that cores 9-15 remain idle and the two processes are sharing core 8. If instead I do this:
$ taskset -c 8 bash -c 'while true ; do echo hello >/dev/null; done' &
$ taskset -c 9 bash -c 'while true ; do echo hello >/dev/null; done' &
Cores 8 and 9 each get 100% utilization as they should. This only applies to isolcpus because the same taskset with cores 1-7 properly spreads the processes over the relevant cores. Furthermore "taskset -p" shows that the affinity mask for the 8-15 processes are set correctly. It appears the kernel scheduler refuses to use anything but the lowest core specified on an isolcpus affinity mask.
Now normally this wouldn't be a big deal with my above examples, just specify individual cores for each process. However I want to run a highly multithreaded application on the dedicated CPU. I want to specify the core set and have the thread pool automatically use, without having to individually reset the processor affinity for each individual thread that's spawned.
Does anyone have any idea how to get the scheduler to give me more than one core from the isolcpu set?