37

I have a system running a financial trading application at a remote facility. I do not have access to the ILO/DRAC, but need to disable hyperthreading. The system runs Intel Westmere 3.33GHz X5680 hex-core CPUs. I can reboot, but want to make sure that the system does not enable hyperthreading due to performance problems. Is there a clean way to do this from within Linux?

Edit: The noht directive added to the kernel boot command line did not work. Same for RHEL.

See: https://bugzilla.redhat.com/show_bug.cgi?id=440321#c9

ewwhite
  • 194,921
  • 91
  • 434
  • 799

12 Answers12

28

Newer Kernels provide a Simultaneous Multithreading (SMT) control.

You can check the state of SMT with;

cat /sys/devices/system/cpu/smt/active

Change the state with

echo off > /sys/devices/system/cpu/smt/control

Options are;

  • on
  • off
  • forceoff

We have tested this with Linux Kernel 4.4.0

  • Hi Nick and welcome to the site. The info about tests (and version) is quite valuable. – kubanczyk May 16 '19 at 21:08
  • 1
    Excellent, Tested on Ubuntu 16.04.6 LTS – Elder Geek May 31 '19 at 01:20
  • I get : permisson denied when I turn it off with the commands? how can I fix it? – esra Sep 13 '21 at 19:20
  • [Kernel docs for the `/sys/devices/system/cpu` sysfs tree](https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-devices-system-cpu), search for `/sys/devices/system/cpu/smt`. And a `tee`-based `sudo` one-liner for good measure: `echo off | sudo tee /sys/devices/system/cpu/smt/control` – genpfault Nov 22 '21 at 16:53
  • @esra, you'll need to run this command as root – Nick Bascombe-Fox Nov 25 '21 at 11:00
24

You can do this at runtime if you want to. I found a nice solution described here: http://www.absolutelytech.com/2011/08/01/how-to-disable-cpu-cores-in-linux/

Step 1: Identify the linux CPUs you want to switch off:

cat /proc/cpuinfo

Look for the CPUs that have the same "core id", you want to switch off one of each pair.

Step 2: Switch off the hyperthreading CPUs (in my case the last four of the total 8 "CPUs" seen by Linux)

echo 0 > /sys/devices/system/cpu/cpu4/online
echo 0 > /sys/devices/system/cpu/cpu5/online
echo 0 > /sys/devices/system/cpu/cpu6/online
echo 0 > /sys/devices/system/cpu/cpu7/online

You could setup yourself a script that you run just after system start.

BenMorel
  • 4,215
  • 10
  • 53
  • 81
ahus1
  • 557
  • 4
  • 12
  • 1
    It works _almost_ as I expected. virtual cores are disabled, now when I execute one cpu-consuming thread it loads physical core by 100%. But using `sysbench --num-threads=1 --test=cpu run` with different num-threads and HT turned on and off says that disabling HT decreases perfomrance when there are many threads, and even if there's just one thread there's no benefit from turning HT off. So I suggest to leave it as it is: it's optimal. – Sergey P. aka azure Dec 31 '14 at 13:43
  • Would you know what the command to switch them back on is? The link at the beginning of your answer is dead~. Thanks! – user189035 Oct 30 '16 at 10:55
  • @user189035: `echo 1` instead of `echo 0` should turn them back on. – Peter Cordes May 16 '17 at 16:28
  • @SergeyP.akaazure, I think for a financial services application, the main reason to turn off HT isn't performance, but security. – Simon Richter Dec 15 '18 at 08:06
  • @SimonRichter At the time this question was originally written, it was indeed performance. SMT/HT wasn't nearly as good at some workloads on CPUs of that era. The Meltdown/Spectre thing, and the more recent Foreshadow attacks, happened years later. – Michael Hampton Jan 08 '19 at 14:34
14

A script to disable hyperthreading in the machine startup...

To disable hyperthreading I include a script on machine /etc/rc.local. It is not exaclty clean, but is easy to install, independent of cpu architecture and should work on any modern linux distribution.

nano /etc/rc.local

    # place this near the end before the "exit 0"

    for CPU in /sys/devices/system/cpu/cpu[0-9]*; do
        CPUID=$(basename $CPU)
        echo "CPU: $CPUID";
        if test -e $CPU/online; then
                echo "1" > $CPU/online; 
        fi;
        COREID="$(cat $CPU/topology/core_id)";
        eval "COREENABLE=\"\${core${COREID}enable}\"";
        if ${COREENABLE:-true}; then        
                echo "${CPU} core=${CORE} -> enable"
                eval "core${COREID}enable='false'";
        else
                echo "$CPU core=${CORE} -> disable"; 
                echo "0" > "$CPU/online"; 
        fi; 
    done;    

How this works?

Linux kernel information and controls can be accessed as files in /sys directory on modern linux distributions. For example:

/sys/devices/system/cpu/cpu3 contains the kernel information and controls for logical cpu 3.

cat /sys/devices/system/cpu/cpu3/topology/core_id will show the core number this logical cpu belongs to.

echo "0" > /sys/devices/system/cpu/cpu3/online allows to disable logical cpu 3.

Why it works?

I do not know exactly why... but the system become more responsive with hyperthreading off (on my i5 notebook and massive Xeon servers with 60+ cores). I guess that has to do with per-cpu caches, per-cpu memory allocation, cpu scheduler allocation and process priorities complex iterations. I think the benefits of hyperthreading is outweight by the complexity of making cpu schedulers that know how to use it.

For me, the problem with hyperthreading is: If I start as many cpu-intensive threads as I have logical cores, I will have fast context switches for the cpu intensive tasks, but expensive ones for the background tasks since the hyperthreading totally consumed by the cpu intensive tasks. On the other hand, if I start as many cpu-intensive threads as I have physical cores I will have no context switches to those tasks and fast context switches for the background tasks. Seems good, but the background tasks will found free logical processors and will run almost imediatedly. It is like they are realtime performace (nice -20).

In the first scenario the hyperthreading is uselles, the background tasks will use expensive context switches because I maxed out hyperthreading with the normal processing. The second is unaceptable because up to 50% of my cpu power gets prioritized to the background tasks.

The "cpu-intensive" tasks I am talking about are artificial intelligence data mining and authorization servers (my work). Blender rendering in cheap computers and clusters (to sketch my future house).

Also, this is guesswork.

I have the impression that is better, but it may not.

Lucas
  • 513
  • 3
  • 10
10

You can use the "thread_siblings_list" for each core to turn off the second core in the HT pair.

The following command pipeline is hacky, not optimised, and done this way hopefully to make it easier to understand.

cat /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | \
awk -F, '{print $2}' | \
sort -n | \
uniq | \
( while read X ; do echo $X ; echo 0 > /sys/devices/system/cpu/cpu$X/online ; done )

so, take all the thread siblings lists, extract the second CPU for each pair, get a unique list, and then turn them off.

does this make sense?

if I do "cat /proc/cpuinfo" after running the above, the number of cores halves.

Paul M
  • 553
  • 5
  • 10
  • 2
    This is a great answer. I had to modify it as follows to work for my purposes: `echo 0 > /sys/devices/system/cpu/cpu$X/online` becomes `echo 0 | sudo tee /sys/devices/system/cpu/cpu$X/online` – carbocation Nov 22 '18 at 19:44
9

For really old kernels (Linux 2.6.9 or so), append the noht parameter to the kernel on boot.

This kernel command-line option has been removed since at least Linux 2.6.18.


From http://www.faqs.org/docs/Linux-HOWTO/BootPrompt-HOWTO.html :

The `noht' Argument

This will disable hyper-threading on intel processors that have this feature. 

If using lilo edit you /etc/lilo.conf (and run lilo afterwards) or if using grub then edit your /boot/grub/menu.lst .

Peter Cordes
  • 457
  • 4
  • 10
rems
  • 2,240
  • 13
  • 11
  • Is this functionally equivalent to disabling HT in the BIOS? – ewwhite Feb 15 '11 at 18:26
  • I don't know that for sure, but yes, I would expect noht to be equivalent to disabling it on the BIOS. – rems Feb 15 '11 at 18:52
  • 2
    This is a Gentoo system. I tried the `noht` entry in the grub kernel command line. The system did not honor the `noht` command. Same for RHEL. See: https://bugzilla.redhat.com/show_bug.cgi?id=440321#c9 – ewwhite Feb 15 '11 at 19:41
  • 1
    **This is obsolete since at least Linux 2.6.18**. The `noht` kernel option was removed. This is unfortunate, because Linux enables a workaround for some Haswell perf-counter errata (BJ122, BV98, HSD29) [only if HT is on](https://lkml.org/lkml/2014/10/9/357), and this happens before initramfs is even loaded. – Peter Cordes May 16 '17 at 16:48
5

Disable SMT / HT at boot time using the kernel command line parameter nosmt:

        nosmt           [KNL,S390] Disable symmetric multithreading (SMT).
                        Equivalent to smt=1.

                        [KNL,x86] Disable symmetric multithreading (SMT).
                        nosmt=force: Force disable SMT, cannot be undone
                                     via the sysfs control file.

Disable SMT / HT at runtime using SMT control:

   /sys/devices/system/cpu/smt/control:

     This file allows to read out the SMT control state and provides the
     ability to disable or (re)enable SMT. The possible states are:

        ==============  ===================================================
        on              SMT is supported by the CPU and enabled. All
                        logical CPUs can be onlined and offlined without
                        restrictions.

        off             SMT is supported by the CPU and disabled. Only
                        the so called primary SMT threads can be onlined
                        and offlined without restrictions. An attempt to
                        online a non-primary sibling is rejected

        forceoff        Same as 'off' but the state cannot be controlled.
                        Attempts to write to the control file are rejected.

        notsupported    The processor does not support SMT. It's therefore
                        not affected by the SMT implications of L1TF.
                        Attempts to write to the control file are rejected.
        ==============  ===================================================

     The possible states which can be written into this file to control SMT
     state are:

     - on
     - off
     - forceoff

Erik
  • 151
  • 1
  • 2
4

Lukas' answer is nice but does not really work for disabling HT because core ID cannot serve for identification of HT siblings. This script works instead:

#!/bin/bash
for CPU in /sys/devices/system/cpu/cpu[0-9]*; do
    CPUID=`basename $CPU | cut -b4-`
    echo -en "CPU: $CPUID\t"
    [ -e $CPU/online ] && echo "1" > $CPU/online
    THREAD1=`cat $CPU/topology/thread_siblings_list | cut -f1 -d,`
    if [ $CPUID = $THREAD1 ]; then
        echo "-> enable"
        [ -e $CPU/online ] && echo "1" > $CPU/online
    else
        echo "-> disable"
        echo "0" > $CPU/online
    fi
done
Anton
  • 141
  • 3
  • your script is a variation of mine. we'd have to check what happens if you have multiple CPUs, just to be sure. – Paul M Sep 27 '17 at 13:07
  • @PaulM That's exactly where I tested and used it for my purposes: 2 socket Haswell system. – Anton Sep 27 '17 at 15:07
0

I had to wait until I could get into the ILO/Drac. The kernel boot parameters do not work on current Linux distributions.

ewwhite
  • 194,921
  • 91
  • 434
  • 799
0

In the libsmbios-bin package (Debian, Ubuntu, etc), you have the binaries isCmosTokenActive and activateCmosToken. Together with the token list, you can then try something like this:

# isCmosTokenActive 0x00d1 # CPU_Hyperthreading_Enable
[...] Type 0x00d1  Location 0x46 AND(fe) OR(0)  BITFIELD: 1
# isCmosTokenActive 0x00d2 # CPU_Hyperthreading_Disable
[....] Type 0x00d2  Location 0x46 AND(fe) OR(1)  BITFIELD: 0

Then activate the CPU_Hyperthreading_Disable token:

# activateCmosToken 0x00d2 # CPU_Hyperthreading_Disable
[...] Type 0x00d2  Location 0x46 AND(fe) OR(1)  BITFIELD: 1

Verify:

# isCmosTokenActive 0x00d1 # CPU_Hyperthreading_Enable
[...] Type 0x00d1  Location 0x46 AND(fe) OR(0)  BITFIELD: 0
# isCmosTokenActive 0x00d2 # CPU_Hyperthreading_Disable
[...] Type 0x00d2  Location 0x46 AND(fe) OR(1)  BITFIELD: 1

Now, the big question is whether or not you simply need a reboot for this to take effect, or if a full power cycle is required. Try it out and see how it goes!

svenx
  • 406
  • 3
  • 4
0

Based on info provided by Paul M here, I'd "script" it this way:

fgrep , /sys/devices/system/cpu/cpu*/topology/thread_siblings_list |
cut -d, -f2 | sort -u |
sudo xargs -rI, sh -c 'echo 0 > /sys/devices/system/cpu/cpu,/online'

Of course it's not turning off hyper-threading in same sense as tinkering with BIOS would do, basically it only tells kernel task scheduler not to use some cores cause we know they're fake ones.

Software that made its assumption based on previous state of /proc or /sys sub-system still might be running sub-optimal or even fail due to this run-time change so its restart might be required. For e. g., I've noticed irqbalance was prone to fail in that circumstances.

poige
  • 9,171
  • 2
  • 24
  • 50
0

Disable HT:

echo 0 |sudo tee /sys/devices/system/cpu/cpu{4..7}/online

Enable HT:

echo 1 |sudo tee /sys/devices/system/cpu/cpu{4..7}/online

Note: This do not really disable HyperThreading but they disable the "fake" cores obtaining almost the same result.

Zibri
  • 301
  • 2
  • 6
  • I like the way you are using `tee`, but this does still fall short of providing a real answer to the question. Those commands only apply to specific hardware configurations and may have unintended effects on other hardware configurations. And an explanation of what those commands do is completely absent. – kasperd Jan 08 '19 at 10:28
  • Since 0 means off and 1 means on, I thought it was easy to understand that the first switches off 4 cores (of the fake 8 on a quandocore with hyperthreading on) and the second switches them back on... If you have a DUAL core those number must be {3,4} instead of {4..7} If you use an octacore it must be {8..15} – Zibri Jan 09 '19 at 14:17
0

Old topic, but had reason to try this experiment. First, I am not at all certain that disabling the (slightly fake) CPUs at runtime is truly equivalent to disabling Hyperthreading at boot. That said, I did see a small performance boost in our application. (But not enough to keep.)

Used the thread_siblings value (common to hyperthreaded CPUs) as a key for enable/disable:

for i in /sys/devices/system/cpu/cpu[0-9]* 
do echo "$(cat $i/topology/thread_siblings) $i" 
done | 
awk '{v = (a[$1] ? 0 : 1); a[$1] = 1; print "echo " v " > " $2 "/online"}' | 
sudo sh 

Try the command w/o the final sudo sh to verify correct.