Questions tagged [numa]

Non-Uniform Memory Access is what it stands for. For x86 architectures this is the method used to handle memory architectures where each processor has local memory and accessing another processor's memory is appreciably more expensive.

Non-Uniform Memory Access describes a memory architecture in which RAM is partitioned into more than one locality. Localities are called Nodes, and in most commodity hardware correlates to CPU processor sockets. In such systems access times to RAM is dependent upon which CPU is calling the FETCH and which NUMA Node the requested RAM resides in. RAM that is local to the CPU node will be fetched faster than RAM local to another CPU node.

NUMA-enabled systems provide hints to the OS in the form of certain BIOS structures. One such structure is the System Locality Information Table, which describes the relative cost of certain nodes communicating. In a fully-connected system where each node can talk directly to every other node this table is likely to have the same values for each node. In a system where nodes do not have direct connection, such as a ring topology, this table tells the OS how much longer it takes for the distant nodes to communicate.

NUMA allows NUMA-aware operating systems and programs an additional optimization center. Such programs ( is one such) will keep process-local memory on the same NUMA-node, which in turn allows for faster memory response times. For NUMA-aware operating systems operating policy is usually set for processes to be served out of a specific NUMA node's memory for as long as possible, which also restricts execution to the cores associated with that node.

For systems that will not be running NUMA-aware programs the differential memory access times can cause seemingly undiagnosable performance differences. The severity of this disparity is very dependent upon the Operating System being used. Because of this, most server manufacturers have a BIOS option to interleave memory between NUMA nodes to create uniform access times.

Historically, older servers (before 2011) set this BIOS setting to interleave by default. However, advances in OS support of NUMA and CPU manufacturers inter-node connection architecture advances have change this, and such settings are increasingly set to let the OS handle memory interleaving.

For Linux operating systems the command numactl can be used to manage the memory policy for a NUMA-enabled system.

65 questions
43
votes
1 answer

Is there still a use for irqbalance on modern hardware?

This question has been asked before, but I believe that the world has changed enough for it to be asked again. Does irqbalance have any use on today’s systems where we have NUMA-capable CPUs with memory sharing between their cores? Running…
espenfjo
  • 1,676
  • 2
  • 13
  • 15
34
votes
1 answer

Unexpected and unexplained slow (and unusual) memory performance with Xeon Skylake SMP

We've been testing a server using 2x Xeon Gold 6154 CPUs with a Supermicro X11DPH-I motherboard, and 96GB RAM, and found some very strange performance issues surrounding memory when compared to running with only 1 CPU (one socket empty), similar…
Marc Micalizzi
  • 373
  • 3
  • 7
11
votes
2 answers

How to check if server is NUMA enabled?

My boss wants to know if the HP DL320 G5 is NUMA enabled. I've tried scouring the internet, but can't find any specific information. Does anyone know off hand if this server is suitable for running NUMA applications?
Eamorr
  • 596
  • 5
  • 13
  • 27
9
votes
1 answer

How would I know if I should be disabling virtual NUMA on VMs running older applications?

From VMware's docs: Virtual NUMA topology is available to hardware version 8 virtual machines and is enabled by default when the number of virtual CPUs is greater than eight. You can also manually influence virtual NUMA topology using…
red888
  • 4,069
  • 16
  • 58
  • 104
9
votes
1 answer

AMD 24 core server memory bandwidth

I need some help to determine whether the memory bandwidth I'm seeing under Linux on my server is normal or not. Here's the server spec: HP ProLiant DL165 G7 2x AMD Opteron 6164 HE 12-Core 40 GB RAM (10 x 4GB DDR1333) Debian 6.0 Using mbw on this…
ntherning
  • 265
  • 2
  • 7
6
votes
1 answer

DL380 Gen9 which processor and what impact on NUMA

We are about to build a new server and I was wondering what would the impact on NUMA with the options available. The system will be built using a Proliant DL380 Gen9 8SFF which has 2 sockets, and 12 memory banks per socket. The question I have is…
Tom V
  • 177
  • 10
6
votes
1 answer

numa and taskset

I am trying to use taskset to assign some processes to specific CPU's. When I run taskset, it appears that the command has worked as I get a reply stating what the new affinity of the process is set. However when i monitor the top output (the…
jumpercake
  • 61
  • 1
  • 4
5
votes
1 answer

Server computational slowdown when RAM is used extensively

I have problem with server slowdowns in very specific scenario. The facts are: 1) I use computational application WRF (Weather Research and Forecast) 2) I use Dual Xeon E5-2620 v3 with 128GB RAM (NUMA architecture - probably related to problem!) 3)…
Ivan Toman
  • 51
  • 2
4
votes
0 answers

Hyper-V 2016 DDA issue: a VM boot fails when a drive is connected to an opposite NUMA node

Brief description: the host is Windows Server 2016 Datacenter Build 14393 (UEFI) running Hyper-V role. SR-IOV and Intel VT are enabled. 4x Intel® Optane ™ SSD 900P 280GB PCIe NVMe 3.0 drives are connected to the host. To check DDA support, I execute…
4
votes
2 answers

NUMA CPU numbering in Linux

I have access to two NUMA servers. One of them is Dell R720 and has these CPUs: $ cat /proc/cpuinfo |grep Xeon|sort|uniq -c 24 model name : Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz The other is a HPE DL360 Gen8 and has these CPUs: $ cat…
juhist
  • 272
  • 1
  • 3
  • 10
4
votes
1 answer

Set default numa policy to "interleave" system wide

I know it is possible to set the numa mode to "interleave" (see NB below) for a specific process using numactrl --interleave, but I'd like to know if it is possible to make this the system wide default (aka change the "system policy"). For example,…
BeeOnRope
  • 553
  • 3
  • 6
  • 12
4
votes
0 answers

IIS Performance NUMA

I have been struggling with an IIS issue and can't seem to find any answers. Basically here is the setup. I have a Dell Server with 4 processors 10 cores each with hyper-threading enabled. Giving me 80 logical cores. Server has Windows 2012 R2…
user204579
4
votes
3 answers

HP DL360p with Intel E5-2630 NUMA Capable?

I couldn't find any information via search to determine this via the Intel or HP website. Can you enable NUMA nodes on this? How can you tell the side of each node if so? Is it always best practice to enable and size VMs based on this if it's…
Untalented
  • 323
  • 1
  • 4
  • 14
4
votes
2 answers

What does `numactl --localalloc` do?

What does numactl --localalloc do? Does it bind program allocation to only the single node that numactl --localalloc was run on? Or does it ensure that whenever the program allocates its given memory from its local memory?
oconnor0
  • 143
  • 1
  • 5
3
votes
0 answers

Kernel parameter isolcpus seems to have no effect on Ubuntu Server with NUMA

I use the isolcpus kernel parameter to restrict the CPUs the Linux scheduler should use to place processes. cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-4.2.0-18-generic.efi.signed root=UUID=78e5b46f-6c1c-4e8f-b921-8c091c98209b ro…
jgjl
  • 31
  • 4
1
2 3 4 5