4

I will build computing cluster (about 64 nodes) and I'm looking for some informations. I want to connect them with Infiniband and install Linux. I tried to google it a little bit, but I couldn't find any informations how to set up a simple working configuration.

I want to use it to access NAS and for interprocess communication (MPI, etc.)

Is it easy to set up Infiniband on Linux? How to do it? Or maybe there is better solution?

HopelessN00b
  • 53,385
  • 32
  • 133
  • 208
klew
  • 713
  • 2
  • 11
  • 16

5 Answers5

4

The Infiniband itself is not a big deal to set up. You have to load the driver for your device and if you don't want to use anything else then IP the ipoib kernel module.

1 IP OVER INFINIBAND 2
3 The ib_ipoib driver is an implementation of the IP over InfiniBand 4 protocol as specified by RFC 4391 and 4392, issued by the IETF ipoib 5 working group. It is a "native" implementation in the sense of 6 setting the interface type to ARPHRD_INFINIBAND and the hardware 7 address length to 20 (earlier proprietary implementations 8 masqueraded to the kernel as ethernet interfaces).

After this you are able to use your device as a normal ethernet device.

Istvan
  • 2,562
  • 3
  • 20
  • 28
2

Do you absolutely need Inifiband? Or do you just need high bandwidth links between things? You might be better off bonding several 1GigE links or going with 10GigE.

kbyrd
  • 3,604
  • 2
  • 23
  • 34
  • NAS and IPC have different performance characteristics. I would seriously consider FC for NAS, or perhaps FCoE or even iSCSI if you have gigE. Bonding is good if you have sufficient ports available, but do your calculations first, some multi-port NICs can't sustain the aggregate transfer rates required. Personally, I would use separate switches and networks for FCoE (layer 1) and TCP/IP (layer 3) traffic. – CyberFonic May 29 '09 at 23:52
1

Easy? cheap? not really, but definitely possible. Since you're not going for the budget solution (that's six figures for the metal only), you'd probably want to use support from a paid, enterprise, OS.

Dani
  • 1,216
  • 1
  • 13
  • 20
0

In our university Scientific Linux,which is prepared by globally known research labs, is used on the cluster. With a wef extra packages Infiniband and IP over Infiniband is possible.

hayalci
  • 3,611
  • 3
  • 25
  • 37
0

Sun HPC Linux Edition is a Linux clustered HPC server with Infiniband. I don't know the details of setting it up, but some good performance numbers are coming.

Charlie Martin
  • 163
  • 6