PCI Express - Clarification about BAR, Memory addressing and physical memory

Question

I'm an absolute beginner starting to understand the PCI Express protocol and I need some clarifications about its mechanics.

I understood that a PCI Express endpoint device may have some memory BAR mapped in the system memory (is it always RAM the system memory we are talking about?). I need to understand what happens when I send from the CPU to the device (A) a memory read request, addressing a certain memory address (first memory BAR, offset 0).

Let's assume that my device doesn't have any in-device memory.

The root complex on behalf of CPU creates the TLP and forwards it to device (A) because the memory address of destination is assigned to (A).

A receives the TLP, unpacks it, and creates a completion TLP containing data coming from its internal application logic.

Now the TLP travels backwards to the Root Complex that unpacks it, gathers the interesting data and gives it back to the the CPU.

What is the system memory role during this communication? None?

Does the root complex communicate (in this case) with the physical memory of the system, other than sending and receiving packets to/from the endpoint?

To some people it may seem a silly question but for me this is crucial to understand the connection between the physical memory and the memory address BARs assigned to each PCI Express device.

Answer 1

When a CPU talks to a memory address, it isn't RAM that has to respond. It can be an I/O device. You can actually think of RAM as a "specialized memory-mapped I/O device" whose job is just to save and give back data, although with today's modern CPUs that have caching and such, it's not physically straightforward.

I am not too keen in the low level details of PCIe, but you seem to be wondering how the PCIe bus itself communicates with the CPU. It does so like anything else that communicates with the CPU:

• Memory mapping - i.e. a device is "mapped" where reads and writes to a range of addresses don't go to RAM, but a device or controller.
• DMA - an external device or controller reads/writes a section of RAM, without the CPU being involved at all.
• I/O ports - this is just another address space (a feature of the Intel x86 family of CPUs) that has historically been dedicated to I/O devices. You'll never find RAM here, but it works like memory mapping. The main difference between I/O ports and memory mapping is that I/O port instructions always work serially, no "out of order" or "reordering" of operations will happen here as opposed to the CPU attempting to do that with accesses to main memory.
• IRQs - an external device sends an interrupt to the CPU

The BARs and such are "mapped" into memory, and take the place of any RAM that may be "under" it. With the MMU I believe it's possible to "remap" the RAM beneath it to a different physical address.