Questions tagged [infiniband]

InfiniBand is a switched fabric computer network communications link used in high-performance computing and enterprise data centers. This tag should be used for questions about IB related hardware and software.

InfiniBand(IB) is a switched fabric computer network communications link used in high-performance computing and enterprise data centers.

Its features include high throughput, low latency, quality of service and failover, and it is designed to be scalable. The InfiniBand architecture specification defines a connection between processor nodes and high performance I/O nodes such as storage devices. InfiniBand host bus adapters and network switches are manufactured by Mellanox and Intel.

In addition to hardware manufactured by Mellanox and Intel there are a variety of software stacks available for the configuration and management of IB fabrics. Most major Linux distributions have some built-in support for Infiniband technologies, and the Open Fabrics Alliance(OFA) maintains the Open Fabrics Enterprise Distribution(OFED) which provides many software and firmware packages for the management, maintenance and setup of IB networks. In addition to the 'official' OFED maintained by the OFA, Mellanox and Intel both produce software stacks for their products that typically include their own customized versions of OFED.

108 questions
0
votes
1 answer

Debian-based Infiniband point-to-point Voltaire dual port cards

I have the following configuration: I------------------I I------------------I I I(P0)---------(P0)I I I Voltaire HCA 0 I I Voltaire HCA 1 I I I(P1)---------(P1)I …
0
votes
0 answers

Getting eth_ipoib in Ubuntu

I've just discovered an ethernet driver called eth_ipoib which I'd like to experiment with. However, there appears to be no kernel module for it in the mainline. http://lwn.net/Articles/507258/ How do I get this going under Ubuntu server? I haven't…
hookenz
  • 14,132
  • 22
  • 86
  • 142
0
votes
0 answers

Possible to connect multiple InfiniBand switches?

Is it possible to connect multiple InfiniBand switches? If so, how can this be done? I've attempted to connect the switches directly via a cable, but that is not working, so I assume that I will need to configure the switches via the console, if…
cmorris14
  • 181
  • 2
  • 3
  • 11
0
votes
2 answers

Is ibdump in Mellanox OFED 2 supported in Ubuntu 14.04?

I have Ubuntu 14.04. I have Mellanox Technologies MT27500 Family [ConnectX-3] HCAs. I was able to pass all the tests like ibping and ib_rdma_lat. Now I'm running MPI tests to verify that InfiniBand is working. How can I actually see that there is…
Grigory
  • 167
  • 1
  • 2
  • 9
0
votes
1 answer

How much throughput increase to expect from link aggregation (LACP/802.3ad)?

I identified that data transfer within my cluster will be bottlenecked by the network interconnect (I'm saturating the dual SAS connection to my storage with sequential workloads), and I'm undecided between a 10 GbE or 40/56 Gb Infiniband to…
0
votes
1 answer

Cannot use SDP in Debian Squeezy

I have severe problems running SDP on Debian Squeeze. I'm using two machines with Mellanox adapters each. My /etc/modules looks like this: mlx4_ib # Mellanox ConnectX cards #ib_mthca # some mellanox cards #iw_cxgb3 # Chelsio T3 cards #iw_nes #…
Ulf Klose
  • 387
  • 1
  • 5
  • 18
0
votes
4 answers

allow infiniband for non root users

I got Infiniband running on RHEL 6.3 [root@master ~]# ibv_devinfo hca_id: mthca0 transport: InfiniBand (0) fw_ver: 4.7.927 node_guid: 0017:08ff:ffd0:6f1c sys_image_guid: 0017:08ff:ffd0:6f1f vendor_id: …
user1219721
  • 467
  • 1
  • 6
  • 15
0
votes
3 answers

Areca 1880ix RAID hangs

Areca RAID controller ARC-1880ix-12 (firmware 1.50) hangs when on high load. My setup is: Chenbro 3U chassis Intel S5500BC mainboard Xeon 5603 CPU 16GB of RAM 12 Seagate SAS drives ST32000645SS (2 of them as hot spare, 10 as RAID10) Mellanox…
Dave
  • 51
  • 5
0
votes
1 answer

What is Infiniband?

What is Infiniband? What I have found is its a fabric switch network medium used for high performance computing. But more specificly what is it? Is it networking medium in the same way as cat5e is to ethernet? Or is it more like Fiber Channel where…
ianc1215
  • 1,965
  • 7
  • 34
  • 55
0
votes
0 answers

`failed to found root port` when using `ibdiagnet`

I am trying to run ibdiagnet tool to test the InfiniBand connections as instructed by NVIDIA Sharp. Yet I got the following error: $ sudo ibdiagnet Running: ibdiagnet ---------- Load Plugins from: /usr/share/ibdiagnet2.1.1/plugins/ (You can specify…
Qin Heyang
  • 101
  • 1
0
votes
0 answers

Infiniband adapter down

edit: On CentOS 8.5, tried with Mellanox driver 4.9-4.1.7.0 (legacy) and 5.5-1.0.3.2: I am not able to get my Infiniband adapter working. The output of ibstat states that it is down: CA 'mlx5_0' CA type: MT4123 Number of ports:…
Holger
  • 31
  • 4
0
votes
1 answer

Latency of memory accesses via interconnectors

I'm trying to compare latencies of different node interconnects for a cluster. The goal is to minimize the memory access latency. I have obtained some benchmarks regarding one of the hardware implementations of NUMA architecture with many CPUs. This…
Piotr M
  • 23
  • 3
-1
votes
1 answer

2 servers with infiniband cards and one IB switch i want to use the Ip Over IB

I have 2 super Micro servers with XCP installed i have 2 Infiniband cards on them, i have IB switch also, Now i want to connect Both servers with IP O IB, How can i do that, if i am using Infiniband i need to disable the Ethernet Card. How can i…
-2
votes
1 answer

How Infiniband manages to have 100Gbit/s

How Infiniband manages to get a speed of 100Gbit/s. The PCIE x16 has maximum speed of 32Gbit/s. For example, NVidia DGX-1 station has 4 such ports. How is this possible?
-2
votes
3 answers

Are there HP Infinihost III ex drivers for windows available?

I picked up some infiniband cards off ebay for development/testing purposes. I was hoping that there would be some drivers available under windows 7, but it doesn't seem to be recognised by the OFED software which would seem to contain windows…
hookenz
  • 14,132
  • 22
  • 86
  • 142