Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [infiniband]

InfiniBand is a switched fabric computer network communications link used in high-performance computing and enterprise data centers. This tag should be used for questions about IB related hardware and software.

InfiniBand(IB) is a switched fabric computer network communications link used in high-performance computing and enterprise data centers.

Its features include high throughput, low latency, quality of service and failover, and it is designed to be scalable. The InfiniBand architecture specification defines a connection between processor nodes and high performance I/O nodes such as storage devices. InfiniBand host bus adapters and network switches are manufactured by Mellanox and Intel.

In addition to hardware manufactured by Mellanox and Intel there are a variety of software stacks available for the configuration and management of IB fabrics. Most major Linux distributions have some built-in support for Infiniband technologies, and the Open Fabrics Alliance(OFA) maintains the Open Fabrics Enterprise Distribution(OFED) which provides many software and firmware packages for the management, maintenance and setup of IB networks. In addition to the 'official' OFED maintained by the OFA, Mellanox and Intel both produce software stacks for their products that typically include their own customized versions of OFED.

108 questions
4
votes
2 answers

Virtual infiniband on kvm/qemu or another open source platform?

Is there any way to set up a virtual infiniband network within kvm/qemu? If not, is there a viable open source alternative which would allow virtual infiniband networking? (Note that I'm not looking to give VMs access to a physical IB network, I'm…
kdt
  • 1,360
  • 3
  • 22
  • 34
4
votes
5 answers

Infiniband on Linux?

I will build computing cluster (about 64 nodes) and I'm looking for some informations. I want to connect them with Infiniband and install Linux. I tried to google it a little bit, but I couldn't find any informations how to set up a simple working…
klew
  • 713
  • 2
  • 11
  • 16
3
votes
0 answers

Kernel modules won't load for NFS/RoCE Ubuntu 16.04 with latest drivers/software

I'm having trouble with NFS over RoCE on Ubuntu 16.04 using the latest OFED package provided be Mellanox (MLNX_OFED_LINUX-3.3-1.0.4.0-ubuntu16.04-x86_64.tgz). My cards are Mellanox 10Gbe and are RoCE v1 enabled. Works with Inbox drivers/software but…
Ryan Babchishin
  • 6,160
  • 2
  • 16
  • 36
3
votes
2 answers

Mellanox and Infiniband latency

I have two hosts with Voltaire HCA500Ex-D (MT25408 ConnectX Mellanox) 10Gbit cards connected to a Cisco SFS7000d IB switch via CX4 3m cables. I'm really concerned about latency which is higher than on 1Gbit ethernet connection using same…
kobazik
  • 31
  • 1
3
votes
1 answer

iSCSI TGTD over InfiniBand Performance

I have just configured a server-to-server 4x SDR InfiniBand link to increase bandwidth for the backend storage of my virtual machines, specifically a database server. I am using iSCSI with TGTD on CentOS 6 and I am able to achieve about 6.45Gbps…
jaseeey
  • 1,462
  • 16
  • 20
3
votes
1 answer

Is it possible to update Supermicro ConnectX-3 Firmware with newer firmware from Mellanox?

I have a Supermicro server with a X9DRT-HIBFF motherboard installed. It has on-board Infiniband based on the ConnectX-3 chipset from Mellanox. Supermicro supply firmware version 2.10.0700 and unfortunately it doesn't support SR-IOV, it's too old. …
hookenz
  • 14,132
  • 22
  • 86
  • 142
3
votes
2 answers

Mellanox dual-port HCA, can ping if ib0 pair connected but not if only ib1 pair connected

I have installed 2 mellanox FDR dual-port ConnectX-3 HCA cards (CX354A), each to a separate machines. The machines are connected directly to each other (switchless configuration). Both ports on the cards are connected such that port1 is to port1 and…
FC Yit
  • 31
  • 1
  • 3
3
votes
1 answer

back to back infiniband

I've been given two hpc cluster nodes, each with an infiniband hca. Is it possible to connect them back-to-back, without using a switch ?
petrus
  • 5,287
  • 25
  • 42
2
votes
0 answers

Can I use Ethernet transceivers on InfiniBand devices?

Specifically, can I use 40GBASE-whatever transceivers as 40G QDR transceiver on common Mellanox devices? If not, what's the cause of such incompatibility?
Proton
  • 121
  • 3
2
votes
2 answers

Dual-port RDMA Throughput Issue

I have two nodes connected with dual-port Mellanox Connext-X3 VPI HCAs via an IB switch. The nodes are two socket machines with Hasswell CPUs and 2 16GB DIMMs per each socket (totaling 64GB). Everything seems to work perfectly, except for the…
Mohammad Hedayati
  • 629
  • 1
  • 6
  • 12
2
votes
1 answer

Replaced storage controller, now can't find ESXi ZFS mount point

We have an older file server (running Gentoo of all things) that recently suffered a storage controller failure. The parts were replaced and everything is back online except one item, probably the most important: the ESXi mount point. The former…
Christian Holton
  • 21
  • 3
2
votes
1 answer

How many infiniband adapters should be used in multi socket servers?

Should dual socket motherboards have an infinity band adapter for each CPU? That is, should there be two infiniband band adapters, one in each CPU's PCIe slot. Would this eliminate the signal going through QPI or is the time for the signal to travel…
Darthtrader
  • 311
  • 1
  • 5
  • 12
2
votes
0 answers

Connecting 3 Server with Infiniband without a switch

Since the question infiniband network between 3 servers was closed as not a question, but I have basically the same problem I will try to formulate it as a real question. I got myself a dirt cheap Dell C6100 with 4 Nodes I want to do some number…
the_architect
  • 121
  • 2
2
votes
4 answers

set the Ca name on unmanaged mellanox switch

How can I set the Hostname/Description of a Mellanox/Infiniband unmanaged switch? I would like a way to abstractly distinguish quickly which switches are which when doing 'ibswitches' or 'ibnetdiscover'. For HCAs that are in Servers, the…
John
  • 23
  • 5
2
votes
0 answers

CentOS 7 - Systemctl dependency problems with network mount and OpenSM

I have the follwoing problem. I want to mount a glusterfs share over Infiniband on an CentOS 7 client. The glusterfs share '/data' is severed by a host called storage1. For the client I I setup and minimal CentOS system. I installed rdma and opensm…
M.K. aka Grisu
  • 141
  • 1
  • 8
1
2
3 4 5 6 7 8