NUMA node interleaving doesn't work for MariaDB

Question

Please help activating NUMA node interleaving for MariaDB.

Using MariaDB 10.1.21 on CentOS Linux release 7.3.1611.

Per instructions on https://mariadb.com/kb/en/mariadb/systemd/ added /etc/systemd/system/mariadb.service.d/custom.conf:

[Service]
ExecStart=/usr/bin/numactl --interleave=all /usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION

/var/log/messages shows that numactl is invoked:

Feb 23 10:40:46 host systemd: Starting MariaDB database server...
Feb 23 10:40:46 host numactl: 2017-02-23 10:40:46 140580844599552 [Note] /usr/sbin/mysqld (mysqld 10.1.21-MariaDB) starting as process 37349 ...
Feb 23 10:40:51 host systemd: Started MariaDB database server.

But numastat shows NUMA node interleaving doesn't work:

[root@host ~]# numastat -p $(pidof mysqld)

Per-node process memory usage (in MBs) for PID 37349 (mysqld)
                           Node 0          Node 1           Total
                  --------------- --------------- ---------------
Huge                         0.00            0.00            0.00
Heap                         0.00            0.00            0.00
Stack                     2711.53         1330.39         4041.92
Private                  23868.98         6126.55        29995.54
----------------  --------------- --------------- ---------------
Total                    26580.51         7456.95        34037.46

As you can see there is a huge imbalance between Node 0 and Node 1.

score 0 · Answer 1 · answered Dec 27 '18 at 16:23

I'd recommend starting first with checking if that policy had been applied for real. Check out if /proc/…PID-of-interest…/numa_maps has entries denoted as interleave:.

For e. g.:

7ffecdccd000 interleave:0-1 stack anon=8 dirty=8 active=1 N0=4 N1=4 kernelpagesize_kB=4

(I'd assume you won't find such.)

Second thing that I can think of is:

[mysqld]
…
innodb_numa_interleave = on
…
[mysqld_safe]
…
numa_interleave = on
…

— At least Percona's scripts are handling the former setting and it can be so that if you haven't explicitly specified them they would be set to default's.

NUMA node interleaving doesn't work for MariaDB

1 Answers1