difference between taskset and cpuset

Question

I'm trying to reduce the latency of my linux network application. I've learned that there are two tools for "binding" a program to particular CPU core: taskset and cpuset.

Which one should I prefer? Are they equivalent on a lower level?
(the disposition) My application has single thread and is supposed to process single tcp connection (no reconnect) over fast LAN network with the least possible latency. Am I on the right way?

Have you run detailed profiling so that you are sure that it is the Linux networking part that causes the latency, and not the application? — Tero Kilkanen, Aug 30 '14 at 10:07
Also, what type of hardware is involved? Server make/model, CPU specifications, networking infrastructure... — ewwhite, Aug 30 '14 at 10:17

score 14 · Answer 1 · edited May 23 '17 at 12:41

Taskset is for binding a process to one or more CPUs; essentially specifying where it can run at initial execution or while it's running. If using RHEL/CentOS on modern server equipment, numactl is recommended over taskset.

Cpuset/cset is for CPU shielding and is a framework build around Linux cgroups. Cset was never popular on certain distributions (like RHEL) because there are other tools available for process management.

The first command below creates a shield that would limit the operating system's tasks to CPU cores 0 and 8. The second would move your current shell session to the CPU shield specified, resulting in an isolation of system and user processes.

# cset shield --cpu 1-7,9-15 --kthread=on
# cset proc --move --pid=$$ --threads --toset=user

There are other things to possibly check for and tune before you go down the path of binding processes to CPUs; interrupts (irqbalance partial disablement), power-saving settings, system scheduler, I/O elevators, realtime policy (chrt).

See: Low latency TCP settings on Ubuntu

Here's a (convoluted) example of an application wrapper that selects a core, stops irqbalance, starts it and blacklists the selected core, then executes ./your_program with SCHED_FIFO and priority 99 on the selected core.

Core=5
CoreMask=`echo "16 o 2 $Core ^ p" | dc`
service irqbalance stop
  until [ "`service irqbalance status`" = "irqbalance is stopped" ] ; do sleep 1 ; done
IRQBALANCE_ONESHOT=1 IRQBALANCE_BANNED_CPUS=${CoreMask} irqbalance
sleep 1
  until [ "`service irqbalance status`" = "irqbalance is stopped" ] ; do sleep 1 ; done
numactl --physcpubind=${Core} --localalloc chrt -f 99 ./your_program

Note, its possible for an application to reset its affinities when using `taskset`. If you use cpuset its not possible to alter your affinities from what the cpuset grants you. — Matthew Ife, Jan 09 '15 at 23:07
Same, both programs call the same underlying system call `sched_setaffinity`. — Matthew Ife, Jan 09 '15 at 23:13

difference between taskset and cpuset

1 Answers1

Linked