3

I use the isolcpus kernel parameter to restrict the CPUs the Linux scheduler should use to place processes. 

cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-4.2.0-18-generic.efi.signed root=UUID=78e5b46f-6c1c-4e8f-b921-8c091c98209b ro isolcpus=4−23,28−47

From what I have read, this should place all processes where not explicitly specified by cpuset or similar on the cpus that are not in the isolcpus list.

However, the processes run on arbitrary CPUs, the supposedly isolated CPUs are still used. Here is a part if the output of ps -F ax:

UID        PID  PPID  C    SZ   RSS PSR STIME TTY      STAT   TIME CMD
message+  1185     1  0 10712  3804  11 20:36 ?        Ss     0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root      1208     1  0  4411  5364  20 20:36 ?        S<L    0:00 /usr/bin/atop -a -w /var/log/atop/atop_20151114 600
root      1214     1  0  7135  3008   2 20:36 ?        Ss     0:00 /lib/systemd/systemd-logind
daemon    1221     1  0  4795  1912  42 20:36 ?        Ss     0:00 /usr/sbin/atd -f

As you can see in the PSR column, CPUs 11,20, and 42 are used, which should not be the case.

I have tried the following Ubuntu Versions so far with the result described above in all cases:

  • Ubuntu 14.04 Server
  • Ubuntu 15.04 Server, with Ubuntu 3.19 kernel as well as mainline kernel
  • Ubuntu 15.10 Server

Any ideas what might be the problem?

Update

Isolcpus does work when running Ubuntu in a VM without NUMA (Core i7). However, it still does not work on a server with NUMA (2x Xeon E3 v3 2690).

jgjl
  • 31
  • 4
  • Chatter on the Internet indicates that they broke it. Are you absolutely 100% certain you need to use Ubuntu? – Michael Hampton Nov 14 '15 at 22:16
  • Well, we settled with Ubuntu for now. Are there alternatives that are known to work? Debian? – jgjl Nov 15 '15 at 15:12
  • Not finding a launchpad bug on the subject. If the kteam doesn't know its broken it can't fix it. – ppetraki Nov 15 '15 at 18:26
  • @ttry56. Is your application so sensitive that "everything" must be perfectly balanced or is this just in additional optimization that's nice to have? The last storage appliance we did, once the storage adapters were lined up with iscsi targets using cpuset, it really didn't mater where anything else went because they dominated the available resources on those numa nodes. Unless you got the performance numbers to back it up, you might be beating yourself up over nothing. Just trying to save you some headaches and unnecessary work. – ppetraki Nov 15 '15 at 18:46
  • Maybe a Debian kernel would work, but I'm surprised that you would seriously consider a Debian based distribution for this. They do not target enterprise use and a large number of design decisions over the years have made Debian particularly ill-suited overall for the enterprise. Ubuntu, of course, inherits most of those design issues. – Michael Hampton Nov 15 '15 at 21:43
  • @MichaelHampton, Well, before he up roots his platform and the stack that depends on it, perhaps he could try the mainline kernel ppa or even build upstream? Does this feature even work at all? Back to NUMA perf. we pushed 58 CPUs/2 nodes + 1TB DRAM to the platform's theoretical limit on RH7 without this; I didn't even know about it until today. We use Ubuntu now. I would never trade the massive leverage and agility Ubuntu gives me for one api endpoint I didn't even know existed until now. One obscure feature having a bug doesn't mean a platform isn't "enterprise". – ppetraki Nov 15 '15 at 23:36
  • @ppetraki we tried the mainline kernel ppa already. The feature works, even with Ubuntu but I got to work it in a VM only. – jgjl Nov 16 '15 at 14:50
  • Isolcpus is a feature that is typically used with low latency applications, e.g. Intel DPDK-based ones. Indeed, for these kind of applications CentOS seems to be the better choice. I get the impression, that the feature, although its usage is advised in many configuration guides does not work in our setting due to a Linux bug that was only fixed this year: [commit message](https://lkml.org/lkml/2015/2/23/557) – jgjl Nov 16 '15 at 14:52
  • @ttry56, If it works in a VM, it's probably because there's only one numa node. If you went into your bios and flattened your cpus into a single numa node it would probably work too. I looked at trusty and the patch is definitely not in there. It wouldn't be much effort to apply and rebuild, there's tons of documentation on the subject in the Ubuntu kteam wiki. The whole concept of the fix isn't that difficult at all, it's just routing the map to where it needs to be. – ppetraki Nov 17 '15 at 18:27
  • @ppetraki: thank you for the update, I guess this is the reason for the whole issue. – jgjl Apr 12 '16 at 14:43

0 Answers0