I have used cpuset to shield some cpus for exclusive use by some realtime threads.
Displaying the cpuset config with the test app RealtimeTest1 running and its tasks moved into the cpusets:
$ cset set --list -r
cset:
Name CPUs-X MEMs-X Tasks Subs Path
------------ ---------- - ------- - ----- ---- ----------
root 0-23 y 0-1 y 279 2 /
system 0,2,4,6,8,10 n 0 n 202 0 /system
shield 1,3,5,7,9,11 n 1 n 0 2 /shield
RealtimeTest1 1,3,5,7 n 1 n 0 4 /shield/RealtimeTest1
thread1 3 n 1 n 1 0 /shield/RealtimeTest1/thread1
thread2 5 n 1 n 1 0 /shield/RealtimeTest1/thread2
main 1 n 1 n 1 0 /shield/RealtimeTest1/main
I can interrogate the cpuset filesystem to show that my tasks are supposedly pinned to the cpus I requested:
/cpusets/shield/RealtimeTest1 $ for i in `find -name tasks`; do echo $i; cat $i; echo "------------"; done
./thread1/tasks
17651
------------
./main/tasks
17649
------------
./thread2/tasks
17654
------------
Further, if I use sched_getaffinity, it reports what cpuset does - that thread1 is on cpu 3 and thread2 is on cpu 5.
However, if I run top -p 17649 -H with f,j to bring up the last used cpu, it shows that thread 1 is running on thread 2's cpu, and main thread is running on a cpu in the system cpuset
(Note that thread 17654 is running FIFO, hence thread 17651 is blocked)
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND
17654 root -2 0 54080 35m 7064 R 100 0.4 5:00.77 3 RealtimeTest
17649 root 20 0 54080 35m 7064 S 0 0.4 0:00.05 2 RealtimeTest
17651 root 20 0 54080 35m 7064 R 0 0.4 0:00.00 3 RealtimeTest
Also, looking at /proc/17649/task to find the last_cpu each of its tasks ran on:
/proc/17649/task $ for i in `ls -1`; do cat $i/stat | awk '{print $1 " is on " $(NF - 5)}'; done
17649 is on 2
17651 is on 3
17654 is on 3
So cpuset and sched_getaffinity reports one thing, but reality is another
I would say that cpuset is not working?
My machine configuration is:
$ cat /etc/SuSE-release
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 1
$ uname -a
Linux foobar 2.6.32.12-0.7-default #1 SMP 2010-05-20 11:14:20 +0200 x86_64 x86_64 x86_64 GNU/Linux
Update:
Additionally I am parsing /proc/pid/task/tid/stat and calling sched_getcpu() before and after the cset --move call, and I'm also doing a sched_yield() after the pinning to try get the thread to move...
This is an example output:
13:12:56 INFO before pinning thread 17508 reports lastCpu=0, currCpu=1
13:12:56 INFO pinning thread 17508 to cpu 3 (/shield/RealtimeTest1/thread1)
13:12:56 INFO SetAffinity cset response:
cset: moving following pidspec: 17508
cset: moving 1 userspace tasks to /shield/RealtimeTest1/thread1
cset: done
13:12:56 INFO after pinning thread 17508 reports lastCpu=0, currCpu=1
13:12:56 INFO after sch_yld thread 17508 reports lastCpu=0, currCpu=1
So the thread is not moving to its new cpuset immediately, or even after a sched_yield
Could this be a SLES 11 / SP 1 issue?
