Questions tagged [infiniband]

InfiniBand is a switched fabric computer network communications link used in high-performance computing and enterprise data centers. This tag should be used for questions about IB related hardware and software.

InfiniBand(IB) is a switched fabric computer network communications link used in high-performance computing and enterprise data centers.

Its features include high throughput, low latency, quality of service and failover, and it is designed to be scalable. The InfiniBand architecture specification defines a connection between processor nodes and high performance I/O nodes such as storage devices. InfiniBand host bus adapters and network switches are manufactured by Mellanox and Intel.

In addition to hardware manufactured by Mellanox and Intel there are a variety of software stacks available for the configuration and management of IB fabrics. Most major Linux distributions have some built-in support for Infiniband technologies, and the Open Fabrics Alliance(OFA) maintains the Open Fabrics Enterprise Distribution(OFED) which provides many software and firmware packages for the management, maintenance and setup of IB networks. In addition to the 'official' OFED maintained by the OFA, Mellanox and Intel both produce software stacks for their products that typically include their own customized versions of OFED.

108 questions
14
votes
3 answers

How can I determine which switch the Infiniband subnet manager is running on?

I recently inherited an Infiniband network containing multiple switches, and I know that one of these switches is running the subnet manager. The rest supposedly have that feature turned off, or were never enabled. The trouble is, I have no idea…
ajdecon
  • 1,291
  • 4
  • 14
  • 21
10
votes
2 answers

what does `pipe ` mean in `ping` output?

While running a test to diagnose some alleged network problem, I got this output from ping: --- r01c01b11n02 ping statistics --- 30000 packets transmitted, 29995 received, 0% packet loss, time 6309ms rtt min/avg/max/mdev = 0.091/0.343/118.051/3.605…
Riccardo Murri
  • 243
  • 2
  • 8
8
votes
1 answer

File-sharing over Infiniband on FreeBSD

I have a pair of servers, which I'd like to connect to each other directly -- rather than through the switch shared by the rest of the network. I expect to benefit from lower latency and higher bandwidth afforded by the direct connection (including…
Mikhail T.
  • 2,272
  • 1
  • 22
  • 49
8
votes
1 answer

IPoIB (IP over InfiniBand) vs. RDMA performance

I have partly inherited a Linux HA cluster at the center of which currently serves a connection with DRBD 8 over IPoIB (IP over InfiniBand) between two Debian hosts. It ain't broken, so I won't fix it. I have also noticed that DRBD 9 supports RDMA,…
rookie09
  • 573
  • 1
  • 5
  • 14
8
votes
1 answer

Connect two infiniband cards to each other without a switch

Is it possible to connect two Mellanox ConnectX-2 cards on two seperate computers to each other without a switch in between? I am just trying to learn to use verbs programming without having to spend $ on an expensive switch. Do I need a special…
Ivan
  • 289
  • 4
  • 13
6
votes
4 answers

switchless Infiniband between two servers on RHEL 6.3

I have 2 servers running RHEL 6.3 which have 2 port Infiniband cards >lspci | grep -i infini 07:00.0 InfiniBand: QLogic Corp. IBA7322 QDR InfiniBand HCA (rev 02) I'm interested in connecting them directly to each other bypassing an Infiniband…
exfizik
  • 161
  • 1
  • 6
5
votes
2 answers

Compatibility and difference between QSFP+ and QSFP 40Gb/s infiniband connections

I am new to infiniband networks, but i need to extend an existing infiniband cluster with some nodes. During my exploratory web-research I came across some to me ambiguous descriptions of cables, connectors and switches. According to Wikipedia QSFP…
Hjan
  • 153
  • 1
  • 5
5
votes
1 answer

Passing through RDMA network devices to docker containers

I'm wanting to passthrough infiniband to a docker container so that I can run some high performance apps over ipoib and use rdma. Currently, I'm doing this with xen virtual machines. Now I'm looking into using CoreOS and docker as a much lighter…
hookenz
  • 14,132
  • 22
  • 86
  • 142
5
votes
2 answers

Infiniband port status UP but can't open UMAD port ((null):0)

My system has 2 infiniband devices, one of which has both the ports up. $> ibstatus Infiniband device 'mlx4_0' port 1 status: default gid: fe80:0000:0000:0000:0002:c903:000f:0a9f base lid: 0x22 sm lid: …
Sidjana
  • 51
  • 1
  • 1
  • 3
4
votes
2 answers

Set up Windows 10 SRP initiator

How do can I setup an SRP Windows 10 initiator? I've seen several mentions of an initiator being included in WinOF but I haven't found any instructions on how to do it. The initiator will be Windows 10 Pro with a Connect-X2 DDR interface. I have…
chew socks
  • 203
  • 1
  • 8
4
votes
2 answers

Infiniband Verifying that RDMA is working

I have two identical computers with Mellanox cards connected to each other through a cable. No switch. Using opensm. I have run several tests, including ping_pong tests, ibping, etc. They all seem to work. However, when I run this test, it comes…
Ivan
  • 289
  • 4
  • 13
4
votes
1 answer

How do I enable SR-IOV for Mellanox ConnectX-3 Ubuntu 12.04.3?

I have a Supermicro 2027TR-HTFRF server that I'm using as a xen server. What I'm wanting to do is get SR-IOV operating for the Connectx-3 card so that I can expose Infiniband to virtual machines. Following this recipe I managed to get the I350…
hookenz
  • 14,132
  • 22
  • 86
  • 142
4
votes
9 answers

InfiniBand Storage

I'm contemplating the next restructuring of my medium-size storage. It's currently about 30TB, shared via AoE. My main options are: Keep as is. It can still grow for a while. Go iSCSI. currently it's a little slower, but there are more…
Javier
  • 9,078
  • 2
  • 23
  • 24
4
votes
2 answers

View Infiniband routing table generated by OpenSM?

As I understand it, the subnet manager of an Infiniband network calculates the best routes between each pair of nodes on the network and provides these routes to the nodes when they want to communicate. Is there any way to get the subnet manager…
ajdecon
  • 1,291
  • 4
  • 14
  • 21
4
votes
4 answers

Set up simple Infiniband Block Storage (SRP or iSER)

I'm trying to figure out how to set up a simple storage system which exports block storage over Infiniband, using either SRP or iSER. I'm very early in the process, and at the moment I'm basically just looking for a tutorial on the level of, "You…
ajdecon
  • 1,291
  • 4
  • 14
  • 21
1
2 3 4 5 6 7 8