assume the following setup: system (Ubuntu 14.04 LTS) with 40 cores, lots of RAM. Additionally a program which is only partly multithreaded. That is, it constantly switches between running on multiple cores for hours and then on only one core (also for hours).
I want to maximise the CPU utilisation of that machine by running that program in parallel, say, 10 times, but I'd really like to prevent situations where either
- all 10 instances want to run on all 40 cores in parallel (terrible cache trashing)
- I tell each instance to use, say, a max of 4 cores, but then have situations where 9 of them are in single core use and the tenth could momentarily use all cores but cannot as it is allowed to use only 4.
Question is: are there utilities / OS extensions which allow that? Using "renice" is not entirely what would be needed, although if there was an automatic supervisor allowed to dynamically renice processes up or down that could come pretty close. Ideally it would work something like this: as long as a set of programs does not maximise out given CPU resources, they are allowed to get as much as they want. But if they use more, then the program(s) with the longest runtime (or whatever) gets highest priority for the CPU, eventually starving the other instances.
The closest I have found is Control Groups (cgexec and friends), but from what I can tell one can only define program groups with fixed CPU ratios, e.g., programs in a group can use at most 60% of the CPU (even if the other 40% are unused) and there is no way to prevent all processes from getting something (again, trashing CPU caches).
