4

I am trying to understand what I guess is called a balanced memory configuration. For the newest cpu's from Intel at this point in time, January 2018 which are for example Xeon 8180 and Xeon 6154, how many DIMMS per CPU socket should be used to get the best memory performance?

For a server board having say 4 Intel 6154 cpus, there is a choice of going with either RDIMM or LRDIMM... which I do not know if that warrants extra discussion but for now using RDIMMS because of pricing and I do not need more than 768GB of system RAM,

For the RDIMM choice there is 8gb or 16gb or 32gb per DIMM, there are 12 memory sockets per cpu socket so 48 sockets total on the board allowing for a maximum of 768 GB of RAM for the Xeon 6154 or 8180,

Note: maximum memory supported by 8180 or 6154 is 768gb per intel spec sheet, I am not sure if that is per CPU or total of the system. If more than that is needed then there is Xeon 8180m and 6154m which supports up to 1.5TB and again not sure if that is 1.5tb for just that cpu but anyway that is above and beyond what I need or am asking,

My question is, for the choice I make on getting a server(s) which I want to have N amount of RAM each which would be either at least 192GB, 256gb, 384gb, 512gb, 640gb, or 768gb...

  1. how would I know the minimum amount of RAM that should be in the system?
  2. choosing either 8gb DIMMs or 16 gb DIMMS affects (1.) obviously but for arguments sake let's say I want the server to have either 256gb or 384gb of system RAM? Why should I choose what size RDIMM and how should I populate the 48 memory slots so I have best memory performance? I have come across articles talking about reduced memory performance of X % and it being really bad if you mess up how you populate the memory slots.
  3. for something like the Xeon 6154 or 8180 should there be 6 DIMMS installed per CPU socket for best memory performance, bandwidth... etc. ? Or can there be just 2 DIMMS per CPU socket? I ask because there is mention of something about max # memory channels = 6 and this is the question I am really looking to an answer for. How bad is having 1 or 2 DIMMS per CPU vs 6 DIMMs per CPU?
  4. And what affect is there if 2, 3, 4, 6, 8 DIMMS per CPU socket are installed, resulting in whatever multiple of 8 or 16 of total system RAM between 192gb and 768gb?
ron
  • 775
  • 3
  • 9
  • 19

3 Answers3

5

Basically, the CPU has the fastest memory access with one DIMM per channel.

Adding more DIMMs increases the load on the memory bus and may decrease the RAM clock.

Dual-rank and quad-rank DIMMs cause higher load and potentially decrease the clock further than fewer ranks. Same size modules with fewer ranks are generally better.

Registered DIMMs (RDIMMs) decrease the bus load, may increase the maximum number of modules and ranks but come with a (slight) penalty in latency.

Load-reduced DIMMs decrease the bus load even further (possibly enabling more DIMMs/ranks or higher clock rate) and have the same latency penalty as RDIMMs.

Unbuffered DIMMs (UDIMMs) are the fastest but only allow very few modules and ranks. Usually they're only found in basic entry-level servers.

Low-voltage DIMMs save energy but the lower voltage swing often decreases the maximum RAM clock that's possible.

Generally, you can't mix UDIMMs, RDIMMs and LR-DIMMs in a system. You can usually mix LV and normal DIMMs but the LV DIMMs will run at the higher voltage.

The exact metrics for your system should be in the manual. There's no one rule for all.

Zac67
  • 8,639
  • 2
  • 10
  • 28
  • looks like the Intel Xeon Scalable Silver, Gold, or Platinum models or maybe it's simply socket FCLGA3647, are all 6 memory channels {as stated by the Intel online spec sheets, and i did not check every model}. I typically see 48 DIMM slots on a 4 socket server board == 12 dimm slots per CPU; are you stating it is better to have one 32GB DIMM in 6 slots per CPU vs 16 GB DIMMs in all 12 slots per CPU, to reach my total RAM value for the system? If so, can you provide any links to documentation about that? thanks. – ron Jan 25 '18 at 13:52
  • For a specific system you need to consult the manual. I can just give you the general info and use *may* and *should* where something doesn't always apply. – Zac67 Jan 25 '18 at 14:33
2

  1. I'm not sure what you mean by minimum. This would depend on your specific application requirements and the software vendor would be the one to tell you.

  2. The memory slot population will be described in the manual for the server/motherboard. It should have specific instructions on which slots to populate and in which order.

  3. Not fully utilizing the available memory channels will reduce performance but it always depends on what you're running on the server and how dependent it is on memory bandwidth specifically. This might also answer 4. As you increase the number of DIMMs per socket you'll see increased performance until you're at the number of memory channels per CPU. After that performance might degrade slightly as multiple DIMMs have to share the same memory channels.

https://www.pcworld.com/article/2982965/components/quad-channel-ram-vs-dual-channel-ram-the-shocking-truth-about-their-performance.html

I'm basing this on a few assumptions so let me know if you have any questions/corrections.

LostWander
  • 121
  • 7
  • 1. can i install just one DIMM per CPU socket = minumum 4 DIMMS @ 8gb/DIMM = 32gb minimum, or minimum 2 DIMM per CPU socket = 64gb minimum and so on. – ron Jan 23 '18 at 18:18
  • 1
    Oh, yes you should be able to install just one DIMM per socket. Each socket just needs a non-zero amount of memory to be able to work. – LostWander Jan 23 '18 at 18:21
  • "each socket just needs a non-zero amount of memory to be able to work" is not what I was asking - will 1 dimm per socket == 2 == 4 == 6 dimms per socket **specifically regarding memory performance** If I spend $20,000+ on a server and it's a dog because of unbalanced memory I will be very unhappy. – ron Jan 23 '18 at 18:25
  • For performance 1 DIMM < 2 < 4 < 6. Anything above 6 should be similar in performance to 6. That's what I meant by "you'll see increased performance until you hit the number of memory channels per CPU" – LostWander Jan 23 '18 at 18:30
  • please know (to anyone reading) this is being asked about higher end **server** boards from supermicro or dell or whoever, with socket FCLGA2011 or later. Specifically on socket FCLGA3647 for the latest **XEON** cpu's stating 6 memory channels by Intel, and and not regarding consumer grade or home pc motherboards using core i7. – ron Jan 23 '18 at 18:31
  • also the base price for this kind of server is above $20,000 with a plus or minus of $1000-$3000 depending on RAM choice which is a minor cost adjustment. Would rather not have a server configured with an unbalanced minimum amount of ram to save $2000 when it makes a $20,000+ server then useless because of poor memory performance. How quickly that performance degradation hits (if true) is basically what i am asking, from someone other than a salesperson who might be trying to sell more ram, to play it safe, where the cost increase is negligible. – ron Jan 23 '18 at 18:38
  • If you check out the link I posted that should give you an idea of just how varied the effect can be. How much you'll feel the lower performance will depend mostly on what you're running on the server. If it's software with a vendor they'll be able to tell you what they recommend, otherwise is there anything more specific you can post here? – LostWander Jan 23 '18 at 18:47
  • I appreciate your feedback and your link was worth the read, however it was "pcworld", and their test was at most half the story and flawed in my opinion; they claim no improvement in quad channel vs dual channel and I would argue they never tested to the level to begin exercising the extra memory channels, as bad as testing a quad core 3.5ghz cpu vs a 16-core 2.0ghz cpu and running test software using only 4 cores. – ron Jan 23 '18 at 19:12
  • They were testing desktop workloads which you're right wouldn't make any use of the extra bandwidth, but that's my point. No way to tell if you'll need more than 1 DIMM per socket unless we have more information on what you're running. – LostWander Jan 23 '18 at 19:19
  • The best I can tell at the moment if I populate only 1, 2, or 4 dimm slots on a Xeon board with 6 memory channels is that *it can work* but I am leaving some performance on the table that i already paid for, and that I should populate at minimum 6 dimms per cpu if i want best memory performance, 8gb/dimms = 192gb min system ram or 16gb/dimms = 384gb min system ram == what amount of RAM I should expect to buy to take full advantage of the server. I should not choose 8gb dimms x 48 = 384gb max system ram when i could need more than that. – ron Jan 23 '18 at 19:20
  • If you want the best memory performance for your money you would populate 6 memory slots per socket. Each DIMM's capacity is up to what you think you'll need for end capacity, which sounds like 16gb DIMMs. – LostWander Jan 23 '18 at 19:25
0

I had part of the same question as you did. I raised it here and documented an answer.

Does having more than one DIMMs per channel impact memory bandwidth? Yes, it does. Having one DIMM per channel seems ideal to leverage maximum memory bandwidth.

Aravindh Sathish
  • 167
  • 5