How Infiniband manages to get a speed of 100Gbit/s. The PCIE x16 has maximum speed of 32Gbit/s. For example, NVidia DGX-1 station has 4 such ports. How is this possible?
Asked
Active
Viewed 244 times
-2
-
https://en.wikipedia.org/wiki/InfiniBand https://en.wikipedia.org/wiki/NVLink – ceejayoz Feb 07 '19 at 02:13
-
1Don't forget RDMA. – Michael Hampton Feb 07 '19 at 03:02
-
@MichaelHampton So, basically the main advantage is the ability to expose your RAM to other computers? So, instead of connecting to PCIE in DGX-1 it connects directly to RAM and uses RAM bandwidth to do its job, right? – Konstantin Solomatov Feb 07 '19 at 03:13
-
2That's just about the only way to move that large an amount of data from the network to the computer that quickly. – Michael Hampton Feb 07 '19 at 03:22
1 Answers
3
I don't know much about Infiniband, but the PCIE 3.0 specification is says that a single lane is capable of sending up to 8GT/s. Therefore a 16x PCI 3.0 connection should be capable of 128GT/s. GT/s is 'gigatransfers per second'.
PCIE 3.0 uses a 128b/130b encoding scheme. Which means that it requires 130 bytes to transfer 128 bytes. So to translate that from "transfers per second" to "bits per second" you have to multiply the max GT/s of 128 by 128/130, which gives you just about 126Gbps of bandwidth.
So I guess the 126Gbps PCIE 3.0 16x is enough for 100Gbps infiniband speed.
I am not sure, but I think the '32' number you are using comes from the fact that PCIe is bi-directional. So theoretically: if you are transferring data in both directions it should be capable of almost 128Gbs in both directions, for a total of 256Gbps, which is then 32GB/s when converted from bits to bytes. Sounds a bit bogus to me, but I suppose it's relevant when it comes to video cards.
Nate Moseman
- 106
- 2