I have two physical database servers (both Windows Server 2016):
test server (5 years old): DELL PowerEdge R730xd, 1x Intel(R) Xeon(R) CPU E5-2637 v4 @ 3.50GHz (4C/8T), 192 GB RAM (12x 16GB PC4-17000 - 36ASF2G72PZ-2G1A2) - one NUMA node
production server (half year old): DELL PowerEdge R740xd, 2x Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz (6C/12T), 512 GB RAM (16x 32GB PC4-21300 - 36ASF4G72PZ-2G6E1) - two NUMA nodes (256 GB RAM on each NUMA node)
Both servers has Performance profile selected in BIOS.
I am Oracle DBA and I noticed that my test server is faster in some queries that uses "only" RAM - not storage system. I am really disappointed, because my 5 years old test server is faster than the new one. I think that my problem is related to NUMA, because test server is one NUMA node system and production hase two NUMA nodes.
I made a lot of test in Oracle, but I made also one simple test outside Oracle to confirm my suggestion. One simple PHP script that allocates in loop cca 2GB memory and free it again:
<?php
for($n=0;$n<=10000;$n++){
$start = microtime(true);
for ($i = 0; $i < 50000000; ++$i) {
$arr[] = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ';
}
//echo (memory_get_usage()/1024/1024).PHP_EOL;
echo (microtime(true) - $start).PHP_EOL;
unset($arr);
}
On my test server one iteration run about 1.7 seconds On my production 2.0 to 2.6 seconds, if I set processor affinity to NUMA node 1 it is 1.9 seconds.
I am not hardware specialist, so could you help me to tune my memory subsystem? It comes to my mind - BIOS settings, node interleaving, Windows tuning etc. I can't believe that PC4-21300 is slower than PC4-17000 - could someone explain me this behavior? I can provide you some additional information if you want - my current BIOS setting etc.