3

I am doing some testing on my server and want to lower the width to my PCIe device (or its PCIe bridge).

In case it is relevant, I figured out how to lower my speed based on the PCI Spec with Linux's setpci and changing the Link Control 2 Register's "Target Link Speed". This allows me to drop from a 8GT/s to 5GT/s or even 2.5GT/s.

Here is an example of that working:

Server:~ # setpci -s 83:04.0 98.w=1:ff
Server:~ # setpci -s 83:04.0 78.w=20:20
Server:~ # ./check_speed_width
  84:00.0 has reported reduced PCIe speed: 2.5GT/s instead of 8GT/s
  bridge device 83:04.0 has reported reduced PCIe speed: 2.5GT/s instead of 8GT/s
Server:~ # setpci -s 83:04.0 98.w=2:ff
Server:~ # setpci -s 83:04.0 78.w=20:20
Server:~ # ./check_speed_width
  84:00.0 has reported reduced PCIe speed: 5GT/s instead of 8GT/s
  bridge device 83:04.0 has reported reduced PCIe speed: 5GT/s instead of 8GT/s

Some of my ideas so far to lower width are:

  • Maybe I just missed a writable register? Target Link Speed is a register with read and write capabilities. When it comes to the Width registers, I only see read-only registers listed in the PCI spec.

  • Maybe there is a way to disable a some of the lanes separate from the PCIe spec?

  • I know that stuff can train/initialize to lower widths if it is having problems. So maybe I can inject errors onto a particular lane and cause it to drop the lane?

TaylorSanchez
  • 171
  • 1
  • 6
  • 1
    I found that the "initiate link width change is implementation specific". So this may not be something I can do in a general sense. Based on the pcisig [Understanding PCIe ® 2.0 Bandwidth Management](http://kavi.pcisig.com/developers/main/training_materials/get_document?doc_id=090831b9a2b1210b2822c06e469992d9d028f13d) – TaylorSanchez Mar 16 '16 at 23:36
  • Some server manufacturers have started adding PCIe bifurcation for uses like multiple NVMe drives. So it may be possible to do this in the future via BIOS settings. – TaylorSanchez Jan 10 '18 at 16:54
  • Hey @TaylorSanchez. Apologies for offtopic necro-posting, but can I have your `check_speed_width` script and whatever is the algorithm for computing the PCI register values for link speed reduction? – intelfx Jun 08 '21 at 07:22
  • @intelfx I don't have access to it anymore. If I remember correctly used mainly python to parse `lspci -vvv` or `lspci -xxx` output. The xxx output would require reading up on the pci spec to understand the registers – TaylorSanchez Jun 24 '21 at 18:33
  • @intelfx I got an old coworker to explain the logic. Essentially you need to compare the the link capability with the link status `LnkCap:` and `LnkSta`. A big catch is, the parent or child devices may have a lower LnkCap causing the entire bus to be slow. You can only be as fast as the slowest LnkCap in the branch. The `lspci -tv` shows you the bus tree. I think using the `x` output was much more consistent to parse than the `vvv`. Writing the script took a good amount of reading lscpi and the pci spec documentation. – TaylorSanchez Jun 25 '21 at 17:11

0 Answers0