Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [infiniband]

InfiniBand is a switched fabric computer network communications link used in high-performance computing and enterprise data centers. This tag should be used for questions about IB related hardware and software.

InfiniBand(IB) is a switched fabric computer network communications link used in high-performance computing and enterprise data centers.

Its features include high throughput, low latency, quality of service and failover, and it is designed to be scalable. The InfiniBand architecture specification defines a connection between processor nodes and high performance I/O nodes such as storage devices. InfiniBand host bus adapters and network switches are manufactured by Mellanox and Intel.

In addition to hardware manufactured by Mellanox and Intel there are a variety of software stacks available for the configuration and management of IB fabrics. Most major Linux distributions have some built-in support for Infiniband technologies, and the Open Fabrics Alliance(OFA) maintains the Open Fabrics Enterprise Distribution(OFED) which provides many software and firmware packages for the management, maintenance and setup of IB networks. In addition to the 'official' OFED maintained by the OFA, Mellanox and Intel both produce software stacks for their products that typically include their own customized versions of OFED.

108 questions
1
vote
1 answer

Infiniband configured and connected but indicator lights unlit on HBAs and no functionality in OS

I have installed two dual-port FDR Infiniband VPI HBAs, one in each of two servers running CentOS 6.9, server1>lspci 03:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3] server2>lspci 81:00.0 Network controller: Mellanox…
rg6
  • 185
  • 1
  • 11
1
vote
1 answer

Get firmware version for InfiniBand adapter from QLogic

How can I find the correct firmware version number for a QLogic InfiniBand adapter on a Debian server? ibstat displays an empty version number like so: Firmware version: Hardware version: 2 lspci -vvv is also silent on the issue like…
rookie099
  • 345
  • 2
  • 14
1
vote
2 answers

In Infiniband can I use the same port for IPoIB traffic as well as RDMA traffic?

Good day, and my disclaimer that I am no IB expert and have not worked on IB directly yet. My question seems a trivial one, yet I could not find anything in google that answers it. Question: In Infiniband can I use the same port for IPoIB traffic…
BeGa
  • 21
  • 6
1
vote
1 answer

ifconfig apparently showing wrong RX/TX values for InfiniBand HCA

Recently, I executed a watch -n 1 ipconfig on one of our Linux cluster computing nodes while it was running a 48-process MPI run, disributed over several nodes. Oddly, while Ethernet packets seem to be counted correctly (a few kb/s due to the SSH…
andreee
  • 133
  • 1
  • 6
1
vote
0 answers

Error while creating an SRP target in container using targetcli

I'm trying to setup a container (CentOS 7) that hosts an SRP target using LIO. As far as I can tell the container has the correct access to the bits on the host (CoreOS 4.7.3) because I'm able to start targetcli without error and have been…
geoffjay
  • 131
  • 5
1
vote
0 answers

Are all Infiniband CX4 cables work the same (support both SDR and DDR rate)?

Are all Infiniband CX4 cables work the same (support both 10Gb SDR and 20Gb DDR rate)? Or actually there are some kind of standards only some CX4 cables can do 20Gb DDR? Same question goes for 40Gb QDR and 56Gb FDR QSFP cables.
user6105869
  • 11
  • 2
1
vote
1 answer

ibping error: “can't serve class 50 on this port”

I try to run an ibping server on a machine with an infiniband card: ibping -S I get this error message: ibping: iberror: failed: can't serve class 50 on this port Any idea to solve the problem? For information, the infiniband cable is apparently…
Tom Cornebize
  • 121
  • 4
1
vote
1 answer

Set a upper bandwidth limit for Infiniband HCAs

I have to deal with PCIe congestion and I need to find a way to reduce PCIe traffic. I get several IB cards in one node (among other PCIe devices). HCAs are FDR Infiniband. Putting those HCAs in FDR-10 or QDR reduces too much the total bandwidth.…
jyvet
  • 111
  • 5
1
vote
1 answer

Infiniband switch, uplink

Regarding switches like Mellanox Voltaire VLT-30011 (QDR IB, managed). I see a gigabit ethernet port on the switch which in the manual (http://www.mellanox.com/related-docs/prod_ib_switch_systems/4036_2036_Installation_Manual.pdf) is labeled "for…
Jay
  • 199
  • 1
  • 9
1
vote
1 answer

In Infiniband networks, what is an MGID and how does it relate to multicast group?

I have this error in my Open Subnet manager log. Sep 22 01:49:53 router3-1 docker[12431]: Sep 22 01:49:53 089580 [68860700] 0x01 -> mcmr_rcv_join_mgrp: ERR 1B11: Port 0x10c37b1f7ebb0001 (MT25408 ConnectX Mellanox Technologies) failed to join…
hookenz
  • 14,132
  • 22
  • 86
  • 142
1
vote
0 answers

Centos. infiniband multicast join failed for status -22

I have a cluster with Linux Centos 6.5 as OS. All the machines are connected by infiniband, the network is working fine but in the /var/log/messages of all the machines I can see the following message ib0: multicast join failed for…
jmlero
  • 11
  • 2
1
vote
1 answer

Infiniband cable

What is the difference between an active and a passive cable? What is the difference between QSFP and QSFP+? Given the choice between copper and fiber, for a very short distance like 3 feet max, is one better than the other or…
Ivan
  • 289
  • 4
  • 13
1
vote
0 answers

Setting up Infiniband network - wrong route

So I'm trying to set up an Infiniband network alongside my Ethernet network. I have 10 compute nodes and one conductor node. All 11 machines are running CentOS and are plugged in to an Infiniband switch and an Ethernet switch. Ethernet:…
ostrich
  • 71
  • 1
  • 8
1
vote
1 answer

InfiniBand RDMA latency test on Xen's dom0 crashes

The short story: while setting up InfiniBand connection between two servers, I cannot complete the RDMA latency test. It crashes even with breaking the ssh connection. The long story. The first server is the Xen 4.4 with Ubuntu 14.04 as dom0…
Grigory
  • 167
  • 1
  • 2
  • 9
1
vote
0 answers

HP ConnectX cards identifying as ConnectX EN

I bought a lot of four HP branded ConnectX card on Ebay. The cards are : HP InfiniBand 4X DDR ConnectX PCI-E Gen2 Dual-Port HCA 483514-B2 My problem is that two out of four cards identify themselves as ConnectX EN cards. This one seems ok: 04:00.0…
user228840
  • 11
  • 3
1 2 3
4
5 6 7 8