GPGPU and motherboard compatibility

Question

How can I verify the compatibility of a motherboard with GPU cards that have over 4GB memory, such as Tesla K40, K80, Titan X, etc.?

The problem is, Tesla K40 does not work properly on AMD SuperMicro servers. Searching forums shows that the motherboard must support some kind of BAR region over 4GB addressing. Which parameter defines this feature, i.e. what should we look at before purchasing the motherboard?*

Answer 1

There is only one combined requirement: your BIOS and OS must support 64-bit BARs.

Starting with native PCI Express GPUs, NVIDIA's GPUs have a 64-bit BAR capability (a Base Address Register stores the location of a PCI I/O region, such as registers or a frame buffer). This means that the GPU's PCI I/O regions (registers and frame buffer) can be placed above the 32-bit address space (the first 4GB of memory).

The decision of where the BAR is placed is made by the system BIOS at boot time. If the BIOS supports 64-bit BARs, then the NVIDIA PCI I/O regions may be placed above the 32-bit address space. If the BIOS does not support this feature, then our PCI I/O regions will be placed within the 32-bit address space (the classic way).

Unfortunately, some Linux kernels (as of 2.6.11.x) and 32-bit windows OSes do not understand or support 64-bit BARs. If the BIOS does place any NVIDIA PCI I/O regions above the 32-bit address space, the kernel will reject the BAR and the NVIDIA driver will not work.

Some useful info here:

Now, in your BIOS, check if you can turn on above 4 GB adress decoding.Once you do this, you should be fine. Configuring the BAR address below 40-bit ensures better general compatibility.