Force renegotiation of PCI Express link speed? x2 card reverts to x1 width

Question

In an effort to circumvent compatibility and cost barriers to using SSD drives with newer HP ProLiant Gen8 servers, I'm working to validate PCIe-based SSDs on the platform. I've been experimenting with an interesting product from Other World Computing called the Accelsior E2.

This is a basic design; a PCIe card with a Marvell 6Gbps SATA RAID controller and two SSD "blades" connected to the card. These can be passed-through to the OS for software RAID (ZFS, for instance) or leveraged as a hardware RAID0 stripe or RAID1 mirrored pair. Nifty. It's really just compacting a controller and disks into a really small form-factor.

The problem:

Look at that PCIe connector. That's a PCie x2 interface. Physical PCIe slot/lane sizes are typically x1, x4, x8 and x16, with electrical connections usually being x1, x4, x8 and x16. That's fine. I've used x1 cards in servers before.

I started testing the performance of this card on a booted system and discovered that read/write speeds where throttled to ~410 MB/s, regardless of server/slot/BIOS configuration. The servers in use were HP ProLiant G6, G7 and Gen8 (Nehalem, Westmere and Sandy Bridge) systems with x4 and x8 PCIe slots. Looking at the card's BIOS showed that the device negotiated: PCIe 2.0 5.0Gbps x1 - So it's only using one PCIe lane instead of two, thus only half the advertised bandwidth is available.

Is there any way to force a PCIe device to run at a different speed?

My research shows that PCIe x2 is a bit of an oddball lane width... The PCI Express standard apparently does not require compatibility with x2 lane widths, so my guess is that the controllers on my servers are falling back to x1... Do I have any recourse?

Abbreviated lspci -vvv output. Note the difference between the LnkSta and LnkCap lines.

05:00.0 SATA controller: Marvell Technology Group Ltd. Device 9230 (rev 10) (prog-if 01 [AHCI 1.0])
        Subsystem: Marvell Technology Group Ltd. Device 9230
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
                DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s L1, Latency L0 <512ns, L1 <64us
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
        Kernel driver in use: ahci
        Kernel modules: ahci

I'm assuming you are putting the card into a x4 slot, right? In any case, I'm not sure how the HP will handle the x2 card. In an x4 slot, it should be able to use both lanes but, as you mentioned, there is no guarantee that it has to use it. All that being said, I'm not sure how much of a performance bump you might see as the 6gbs SATA is still generally viewed as being limited to a max of 4.8 gb/s with the encoding overhead with a single PCIe lane having a throughput max of 5gbs. — Rex, Nov 20 '13 at 14:51
@Rex *"x4 and x8 PCIe slots"* were tried. The card has two discrete 6Gbps SATA SSDs onboard, so there is a speed bump available once/if PCIe x2 is negotiated. — ewwhite, Nov 20 '13 at 14:55
Ah.. missed the x4/x8 slots in the question, but you do say "PCIe card with *a* Marvell 6Gbps SATA RAID controller" implying a single controller :) If it has two discrete controllers, is each SSD connected to it's own controller? Then how can it do hardware RAID across the two different controllers? — Rex, Nov 20 '13 at 15:00
@Rex There are two SSDs connected to one controller. A 6Gbps RAID controller has 6Gbps bandwidth per SAS/SATA lane. — ewwhite, Nov 20 '13 at 15:07
The PCIe spec does not require that slots support anything except 1x. When a card is inserted the host queries it for lane support, and whatever the highest power of two that both support is used (so 1x, 2x, 4x, 8x, 16x, or 32x). Also, each lane on an endpoint (host or card) may support initial negotiation, but typically only the first lane does (when additional lanes support negotiation the slot will be capable of bifurcation, plugging more than one card in through the use of a splitter). As noted below, the servers in question do not support 2x lane mode. — Chris S, Dec 04 '13 at 19:16
@ChrisS It turns out that the DL380p Gen8 *does* negotiate 2x properly. — ewwhite, Dec 04 '13 at 19:22

score 6 · Answer 1 · answered Nov 20 '13 at 15:53

6

That's the official answer from OWS, now another answer if it is possible to force hp controller to up it to x2 width, that will require some more research. -)

enter image description here

answered Nov 20 '13 at 15:53

Danila Ladner

5,241
21
30

But the card is in a G6 ProLiant at the moment - `Manufacturer: HP Product Name: ProLiant DL180 G6` – ewwhite Nov 20 '13 at 16:02
Right, likely all Proliant DL pci controllers will have issue with the card. – Danila Ladner Nov 20 '13 at 16:13
Look at you, going to the *source*!! – ewwhite Nov 20 '13 at 16:38
Also on Gen7, Gen8 for first slot on pcie riser will always get 1x: 1 PCIe 2.0 X1 X4 Half Length/Full Height from here: http://h18000.www1.hp.com/products/quickspecs/14339_na/14339_na.pdf I assume you tried different slots. – Danila Ladner Nov 20 '13 at 19:10

ewwhite · Accepted Answer · 2014-12-15T23:16:15.357

3

I tried this again on a slightly different HP platform, the 2U HP ProLiant DL380p Gen8 versus the 1U DL360p Gen8. I received the proper results using this combination of hardware.

Current Gen8 host firmware now allows the PCIe links to negotiate at the proper speeds, so these devices ARE compatible with ProLiant DL380p Gen8 servers.

Speed 5GT/s, Width x2, ASPM L0s L1, Latency L0 <512ns, L1 <64us
LnkSta: Speed 5GT/s, Width x2, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

edited Dec 15 '14 at 23:16

answered Dec 04 '13 at 18:57

ewwhite

194,921
91
434
799

Could you tell me what firmware is that? I guess firmwares from that version onwards will support it properly... I'm waiting for a DL380e gen8 so I dont know what fw it has yet but want to be prepared – Dredok Apr 29 '20 at 16:38
1

No idea. That was almost 7 years ago. But just make sure your DL380e is on current firmware. – ewwhite Apr 29 '20 at 20:02

MikeyB · Answer 3 · 2013-11-20T15:29:58.020

I have on hand an IBM server with a Broadcom 4×1GbE card in it… negotiated down from 2.0x4 to 2.0x2:

0b:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) 
Subsystem: IBM Device 03a9 
Capabilities: [ac] Express (v2) Endpoint, MSI 00
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Latency L0 <2us, L1 <2us 
LnkSta: Speed 5GT/s, Width x2, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

It had a x4 connector which I'm guessing is only fully used when put into a PCIe v1 slot, so it seems that it's negotiating down to x2 when used in a PCIe v2 slot (since 5GT/s bidirectional is enough for 4×1GbE).

Could the same thing be happening with your card?

(If not, my answer is: use IBM instead of HP :P)

Force renegotiation of PCI Express link speed? x2 card reverts to x1 width

3 Answers3

Linked