More than 5Gbps bandwidth reported from a USB3.0 / USB3.1 Gen 1 Root Hub, how is it possible?

Old title: Is this USB3.0 hub a USB3.1 (Gen 2) hub in disguise? More than 5Gbps bandwidth reported

I'm running a couple of USB3 machine vision cameras. Each camera is 2448x2048 at 75 fps, 8-bit data so round about 3Gbp/s. One one PC I have, I can't run both cameras at full speed, on another PC I can. I was (previously) under the impression that a single USB3.0/3.1 Gen 1 hub was limited to 5Gbp/s.

This is an important issue to raise, because in the past we've had problems running multiple high bandwidth cameras on what we've referred to as 'a single controller'. Modern cameras can easily saturate a single USB3.1 connection. In the past the solution has been to add a PCI expansion card. It seems like in some cases this isn't necessary, and I'm interested in understanding why.

In previous edits I referred to the Root Hub (which the cameras are plugged into) as just a 'Hub' which caused some confusion - my ignorance here, I wasn't aware there was a difference. This seems to be critical.
Much of the prevailing info online, suggests that even a root hub should be limited to 5Gbps, for example:

A single USB host controller almost always serves more than one USB port, but the total bandwidth available to those ports will be shared. Think of a single USB port with a four, or even eight-port hub attached. The total available bandwidth for any host controller or root hub can be something close to the following (your bandwidth can vary):
• USB 3.0 — Total theoretical bandwidth per host controller = 5Gbps (625MB/s)
• USB 3.0 — Total practical bandwidth per host controller = 3.2Gbps (~400MB/s)
• USB 2.0 — Total theoretical bandwidth per host controller = 480Mbps (60MB/s)
• USB 2.0 — Total practical bandwidth per host controller = 308Mbps (~36MB/s)
From https://www.oculus.com/blog/oculus-roomscale-balancing-bandwidth-on-usb/

Motherboard vendors advertise everything as just USB3.0 and USB3.1. In reality this is often USB3.1 Gen 1 (5Gbps) and USB3.1 Gen 2 (10Gbps). This caused further confusion.
In addition, Windows displays USB3.1 Gen 1 root hubs as USB3.0

Setup:

Both cameras are rated for USB3.0
Both cameras are plugged into the motherboard directly, in USB3.1 Gen 1 sockets according to motherboard docs.
Both cameras are plugged into the same USB Root Hub as reported by the device manager, for example:

Observations:

On one machine, with an Asus H110M-R motherboard, I get the expected behaviour that one camera runs at full speed (75fps) and the other is reduced to around 7fps. Both cameras are in the same USB3.0 root hub.
On a different machine, with an Asus Impact VIII motherboard, both cameras will run at full speed (no fps slowdown). In fact we can plug in more cameras and we can push around 10Gbp/s (possibly a bit more). The fact that we get 10Gbps suggests this is actually a USB3.1 Gen 2 controller, but Windows reports that we're using the USB3.1 Gen 1 controller (reported as USB3.0). It could also be a coincidentally close bandwidth limit (I would be amazed if we actually got the theoretical 10Gbps throughput).
Manual verification too - I can wave my hand in front of the cameras and both are clearly running at > 50fps (at least).
Since the viewing software automatically adjusts the frame rate (down) on the first PC, I don't see why it would be incorrect on the other one. Similarly, bandwidth monitoring software (Advanced USB Port Monitor) also seems to be correct, and the bandwidth plot changes if we manually slow down one of the cameras.

More info on the motherboards:

The Impact VIII is advertised as having two controllers: a USB3.1 Gen 1 and a USB3.1 Gen 2. The ports are physically separated on the backplate. There are four rear ports are labelled as USB3.0 (and two on the front), there's also a red USB-C and USB-A labelled USB3.1.
The H110M-R has two USB3.1 Gen 1 ports and 6 USB2 ports.
The Impact VIII uses an Intel Sunrise Point-H USB 3.0 xHCI Controller (PCI\VEN_8086&DEV_a12f) and an Intel Alpine Ridge DSL6540 controller for USB3.1 (PCI\VEN_8086&DEV_15b6).
The H110M-R also reports PCI\VEN_8086&DEV_a12f as its USB3.0 controller.
So we're using the same type of root hub, apparently?
The Impact VIII uses a Z170 chipset, while the H110M-R uses the H110 chipset.

The PCH contains an eXtensible Host Controller Interface (xHCI) host controller which supports up to 14–USB 2.0 ports and up to 10 –USB 3.0 ports with board routing, ACPI table and BIOS considerations. This controller allows data transfers of up to 5Gb/s. The controller supports SuperSpeed (SS), High-Speed (HS), Full-Speed (FS) and Low- Speed (LS) traffic on the bus. The xHCI controller supports USB Debug port on all USB 3.0-capable ports. The xHCI also suppo rts USB Attached SCIS Protocol (UASP).

From the 100-series manual https://www.intel.com/content/www/us/en/chipsets/100-series-chipset-datasheet-vol-1.html

Assuming that these devices are really working at full speed, some thoughts/possibilities are:

~~Does the Impact VIII use a single USB (Gen 2) controller that's running all the ports (but enumerates as two)? The spec seems to suggest not, and Device Manager reports there being two controllers.~~
Can a single USB3.0 controller support more than 5Gbps? I assumed not.
Can a USB3.1 hub advertise itself as a USB3.0 hub?

However, the main question is - Why is one system limited to 5Gbps and one not, when both seem to use a single USB3.1 Gen 1 controller (of the same type) which the cameras are plugged into?

Unless Windows is misreporting, I'm not using the Gen 2 controller. I'm going to investigate to see how much bandwidth I can actually pull on the Impact VIII.

Josh

Posted 2017-09-11T15:18:51.200

Reputation: 123

Did you measure the bandwidth or just calculate it? Are you sure your cameras aren't doing compression? What is the make and model of the cameras? – Spiff – 2017-09-11T15:41:37.513

@Spiff I've measured using a monitoring tool, which correlates to the calculation. I've physically tested them by waving a hand in front - they're obviously not running at a reduced frame rate as on the other PC. I thought about compression, but I've also tried uncovering/covering the cameras and the data rate is flat. The slowdown is really obvious on the H110M0-R. – Josh – 2017-09-11T15:50:33.570

Answers

See Ali C's answer below.

You have the cameras plugged into two ports of a Root Hub. I'd guess that these ports are in fact connected to the Z170 chipset.

You have to consider two things:

A "Root Hub" is not a "Hub" - it's rather a thing that provides n USB ports.
All of the ports on a "Root Hub" aren't channelled through a single USB link as they would be for a standard "Hub", thus if the Root Hub is connected to the system with sufficient bandwidth, then the total bandwidth across the ports may exceed the USB spec.

The Maximus VIII Impact uses the Z170 chipset - block diagram below. As you can see, it supports "Up to 10x USB 3.0 ports".

The H110M-R uses the H110 chipset - which supports "Up to 4 USB 3.0 ports", so I'd expect similar capabilities - if you are connected to the motherboard's USB 3.0 ports directly.

But from your wording I suspect that the cameras are connected via a physical / external hub on this PC... in which case all of the camera data needs to be transferred over the single USB 3.0 link between the hub and PC. If this is not the case, then it could be a driver or architectural improvement between the chipsets...

If the Root Hub has the bandwidth between it and the processor, then the total bandwidth on the ports will be able to exceed the USB 3.0 bandwidth.

However, if you have a Root Hub connected via a single PCIe lane, then the bandwidth would be constrained by the PCIe link (still only 3.9GB/s for a single lane gen 5 link), and thus the two ports would only have ~3.9Gb/s to play with.

A similar effect may be in play due to the H110's DMI2 link (5 GT/s) to the CPU vs. the Z170's DMI3 link (8 GT/s).

Attie

Posted 2017-09-11T15:18:51.200

Reputation: 14 841

By hub I meant what device manager reports (edited my question for clarity). The cameras are connected straight into the motherboard in both cases. – Josh – 2017-09-11T15:54:12.983

This seems like a pretty compelling answer. Somewhat related followup from that, am I right in thinking that a USB3.1 Gen 2 hub (not a root device) would still be limited to 5Gbps if you connected a number of USB3.0 device to it? – Josh – 2017-09-11T16:32:24.730

Right, so you could run 2x 3.0/3.1 Gen 1 devices at full speed on a 3.1 Gen 2 hub. Although it looks like aside from chipset support, there are very few Gen 2 hubs available to purchase. – Josh – 2017-09-11T17:22:53.843

I'd be careful with this presumption. It would be nice to think that you could fit two full USB 3.0 bandwidth devices over a single USB 3.1 link... but I have a feeling (don't quote me) that this won't work out. – Attie – 2017-09-11T17:26:23.097

@Ramhound - that's correct, this answer suggests that the reason I'm seeing higher than expected bandwidth is could be because I'm connected to a USB3.1 Gen 1 root hub. Which led me to ask would you expect similar behaviour (capped at 10Gbps) using a 'normal' Gen 2 hub. – Josh – 2017-09-11T17:31:23.913

@Attie I'm sure due to protocol overhead you couldn't max out, but sounds like you can do better than one, which might be useful. In this case the cameras 'only' use around 3Gbps, so a Gen 2 hub would probably be enough to saturate two USB3.0 connections. – Josh – 2017-09-11T17:31:26.347

Wrong answer, speculations. Bandwidth is limited by common PIPE bus shared across all port's PHYs, and is aligned with controller bus-mastering engine. – Ale..chenski – 2017-09-13T07:50:19.990

Thanks for the info and your answer @Ali ... I've up voted your answer, and will point to it from mine. – Attie – 2017-09-13T07:55:28.717

Root hubs are no different from ordinary hubs. The total bandwidth of a root hub is determined by architecture of host controller and data interface between xHCI controller and internal fabric, and between xHCI and USB PHYs (bunch of them, which determines the number of possible ports). If it is the original USB 3.0 core controller with Gen1 PHYs, the throughput can't exceed 5 Gbps.

The only reasonable explanation for this observation is that the Z170 chip has essentially the USB 3.1 Gen2 core, with corresponding PIPE 10Gbps buses inside, but the individual PHYs are capable only of Gen1 speed. There could be many reasons why the PHYs can be restricted to Gen1 speed - unresolved bugs in new encoding protocol, physical layer doesn't fully meet USB-IF certification criteria (could be jitter, could be eye, whatever), or dozens of other deficiencies like glitches in power management or link training instability. Besides, it would be really annoying for public to have a system with 10 Super-Speed ports crippled at 1/10th of expected bandwidth, people might have perception dissonance.

Ale..chenski

Posted 2017-09-11T15:18:51.200

Reputation: 9 749

Thanks Ali. Do you have a standardized diagram of the architecture of a root hub? – Attie – 2017-09-13T08:13:31.833

Presumably this is still affected by the number of PCIe lanes given to the controller, as Attie suggested? For instance, the 'true' Gen 2 controller on the Impact VIII is given 4x PCIe3 lanes for an advertised 32Gbps bandwidth (even though it only has two physical ports). https://techreport.com/r.x/2016_1_24_Asus_ROG_Maximus_VIII_Impact_MiniITX_motherboard_reviewed/05-diagram_pci_routing.gif

– Josh – 2017-09-13T12:11:46.543

@Attie, the diagram is not in public domain. You need to register with Synopsis to get more info on DesignWare USB 3.1 IP. More, the controller core is integrated by Intel into their proprietary IOSF (fabric interconnect) architecture. To get a general idea, examine this picture, https://csdl-images.computer.org/mags/mi/2015/02/figures/mmi20150200263.gif

– Ale..chenski – 2017-09-13T17:27:08.257

@Josh, PCIe lanes are not an appropriate metrics, the xHCI controller is integrated directly to IOSF, and PCIe lanes use separate ports. PCIe lane allocation is used only with external xHCI controllers. – Ale..chenski – 2017-09-13T17:30:02.860

@Attie, some details on universal interconnect between various PHYs (PIPE) can be found here, https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/phy-interface-pci-express-sata-usb30-architectures-3.1.pdf

– Ale..chenski – 2017-09-13T17:40:06.127

According to the Hardware IDs (PCI\VEN_8086&DEV_15b6), this is the Alpine Ridge (DSL6540) controller which is indeed USB 3.1-Gen2/Thunderbolt3.

So this USB3.0 comes from the generic USB3.x driver from Windows 10, so this should be only a display issue in device manager.

magicandre1981

Posted 2017-09-11T15:18:51.200

Reputation: 86 560

Windows reports two hubs, both Intel, one USB3.0 and one USB3.1. – Josh – 2017-09-11T15:56:36.433

post a picture of all USB 3.x devices in device manager. also use http://www.uwe-sieber.de/usbtreeview_e.html and post a picture.

– magicandre1981 – 2017-09-11T15:58:18.223

http://imgur.com/a/LBUPs Ports 21/22. Device manager has the same hub names. – Josh – 2017-09-11T16:22:03.220

select the USb3.1 controller and post a picture of the details – magicandre1981 – 2017-09-11T16:35:23.910

http://imgur.com/a/g9rvn – Josh – 2017-09-11T16:38:41.540

ok, according to the Hardware IDs, this is the Alpine Ridge (DSL6540) controller which is indeed USB 3.1-Gen2/Thunderbolt3. so this USB3.0 comes from the generic USB3.x driver from Windows 10, so this should be only a display issue in device manager

– magicandre1981 – 2017-09-11T16:54:53.797

But that's not what the cameras are plugged into :)

This is the other one: https://vendev.org/pci/ven_8086&dev_a12f/

– Josh – 2017-09-11T17:20:35.903

"Why is one system limited to 5Gbps and one not, when both seem to use a single USB3.1 Gen 1 controller of the same type?" 1 controller is gen2 with 10gbit and one is gen1 with 5gbit/s – magicandre1981 – 2017-09-12T14:06:40.623

I get that, but the cameras are only plugged into a Gen 1 controller on each system. The Gen 2 controller separate, and unused (as the device tree shows). Unless I'm missing something here? – Josh – 2017-09-12T16:50:55.190