How are PCI/PCIe BARs configured to access memory on the device?

Question

In my previous question, I established that the memory address used by PCI/PCIe devices does not have to mapped to system RAM, but could also refer to "on device" memory, e.g. RAM or registers.

But how are the BARs set up in this case?

Is it simply a case (that during configuration):

• the device advertises that it wishes to expose, for example 16MB of memory,
• BAR gets assigned a base address,
• Thereafter the device maps the offsets to its internal memory?

How does this differ from when the device wishes to access 16MB of system memory (i.e. the device has no memory of its own?)

Thanks!

EDIT: How does the addressing differ between:
A: The device has no memory of its own, e.g. a Video Adapter using shared memory.
B: The device has memory, and shares it with the system (e.g. hypothetical HDD adapter with RAM buffers).

In both cases, are the BARs assigned by the OS and occupy addressable memory?

This distinction is important in a system fully-occupied with RAM (e.g. a 32-bit system with 4 GB RAM installed would be able to support a Video Adapter with 2 GB shared RAM [situation (A)]. But the same system has no address space left over for a hypothetical HDD adapter with onboard 2 GB RAM cache [situation (B)]?

Answer 1

Your interpretation of the BAR setup is correct.

Then how does it differ.

The device's on board memory/registers usually is a special RAM, it may be bi-port RAM (when each cell has two ports one is connected to the host bus, another to the device's internal bus, this allows faster and concurrent access from both sides), it may be a registers or even directly some inputs or outputs of some FPGA - Field-Programmable Gate Array that controls some logic of the device.

The main system RAM that a device may access by means of the DMA most often has a completely different designation. In most common case it is used as data source or data destination for the device. For example in case of disk drive that memory will be accessed by the device to read the data in or to write this data onto media. The location of such sources and destinations is not limited by system buffers. For example the paging (swapping in and out of the virtual memory) is performed directly to/from the physical page address of the page in question.

The other possible kind of the DMA access is when the device requests from the system some memory area for its own needs. One of the examples is an Integrated Video Adapter that usually has no its own RAM to keep the binary representation of the image to display on the screen. However this case is very much like disk write operation. The only difference is that it is continuous.

Answer 2

BARs are not used for device buffers in system RAM, and the device does not request such buffers via BARs. Rather the device's host-register interface spec will include a provision that the host must allocate a buffer of a given size - or a size within a specified range - and make it available to the device.

Rather, the host driver will allocate a buffer from system RAM and will tell the device where it is (and, if it's not of fixed size, how big it is), perhaps by simply writing the address to a device register, perhaps as part of a more complex protocol.

The device then accesses the system RAM buffer just as would access buffers associated with I/O operations, by initiating memory read and write transactions directed at the buffer.

Answer 3

PCI(e) devices can not request a dedicated system memory buffer, at least not by using standard PCI(e) configuration methods ( BARs ). The only devices that generally do this are integrated GPUs, and they have special support in the motherboard chipset that reserves the memory buffer, but these are only understood and set by the system BIOS.