-6

Let's say I have 3 different servers, each one with an infiniband card. Each card has 2 different ports. (I don't know about the model yet)

Is it possible to create 3 different networks and to allow the 3 servers to communicate with each other without any problems? (and any spof).

I guess I just have to setup the /etc/hosts correctly.

I really don't know about infiniband, so please help me :)

Thanks in advance.

EDIT: Point is to NOT USE a switch!

grumpf
  • 39
  • 1
  • 2
  • 1
    InfiniBand fabric does not use IP, therefore /etc/hosts thing does not apply. Yet you can run IP on top of InfiniBand. – Dmitri Chubarov Nov 08 '12 at 20:07
  • good place to ask these questions https://www.openfabrics.org/index.php?option=com_kunena&view=entrypage&defaultmenu=52&Itemid=50 – tony roth Nov 08 '12 at 20:22
  • 1
    Dmitri, IB supports IP encapsulation - in fact it's used quite extensively in at least two very large CGI renderfarms I'm aware of. Though I'll admit it's really not the lowest cost IP solution :) – Chopper3 Nov 08 '12 at 20:33
  • 1
    Why do you want to do this? – ewwhite Nov 08 '12 at 21:05
  • 1
    Buy two cards per system? Build a triangle? – Zoredache Nov 08 '12 at 22:52
  • The beauty of IB is that the switching gear is very cheap compared to ethernet. An 8 port qdr (40g) switch is only $1700(us) from mellanox. Like Steve said below 2 nodes no problem 3 nodes I'd think there would be a subnet manager issue. – tony roth Nov 09 '12 at 14:40
  • 1
    Pity the question was closed. The answer is actually yes. You could have done what you ask. Whether it's a good idea is another question. But, I can see why you'd do this if you were trying to save money on an IB switch. – hookenz Mar 26 '15 at 01:33
  • OP: I really hope you didn't give up, because the people below have no idea what they are talking about. You can connect each node to 2 other nodes and they will form a total of 3 networks (InfiniBand subnets). – Navin Mar 25 '16 at 09:48

3 Answers3

5

You're just talking about doing a point-to-point connection from each host to each of the other two right?

I've not tried it, but from what I have heard you can do point-to-point connections using IB without using a switch (just as you can with ethernet).

Problems might come in with the subnet manager(s) - I'm not sure if each node's subnet manager can run actively for one of the connected networks and backup for the other. You may need to run two subnet managers per box and bind each to a specific port. Not sure how to do that either but definitely worth researching that side of it more.

And obviously having a couple of switches would make things easier (and more scalable).

EEAA
  • 108,414
  • 18
  • 172
  • 242
Steve
  • 51
  • 2
  • Yes, you need 1 subnet manager per network. It is also perfectly safe to run 2 subnet managers per node, but you don't need to. – Navin Mar 25 '16 at 09:41
4

Yes, just buy one or two (for resilience) IB switches. That said you don't mention which protocols you want to use, though it seems like you might want to run IP over IB, which works - and editing hosts (or having a proper DNS environment) will help to make that work but IB is really designed for IB-specific protocols, IP over IB works great, it's not just a mainstream usage case ok.

Chopper3
  • 100,240
  • 9
  • 106
  • 238
2

Your question basically boils down to this:

I have 3 servers with Inifiniband. Can I create a network between them?

Which is analogous to

I have 3 servers with Inifiniband Ethernet. Can I create a network between them?

So the answer is yes. But then when you throw this into the mix:

Point is to NOT USE a switch!

The answer is no. You need a switch to get 2 or more of anything to work together. Even if you daisy chain them off each other (eww) then you still have a switch in the middle of the daisy chain. I'm not saying you can daisy chain them though, so don't take that as an assumption that you can.

Mark Henderson
  • 68,316
  • 31
  • 175
  • 255
  • Your old answer is actually wrong. A proper installation would use a switch. I guess he was trying to save several thousand on an IB switch. – hookenz Mar 26 '15 at 01:34
  • Not sure what was wrong about my answer? I told him that he has to use a switch in the last paragraph. – Mark Henderson Mar 26 '15 at 01:40
  • 1
    No you could connect them in a ring configuration since each HBA has 2 ports. Becomes like a daisy chain. Since each other server is directly connected to another they would all be able to talk even if one link went down. He'd have to run a subnet manager on each link though. – hookenz Mar 26 '15 at 01:44
  • Which I also mentioned - chaining them off each other. Never having used infiniband I wasn't sure if that was actually possible though, but then the nodes in the middle of the ring become a "switch" anyway. Maybe not a device sitting in a rack, but they still switch packets between nodes on the network. – Mark Henderson Mar 26 '15 at 01:46
  • 2
    @MarkHenderson If you've never used InfiniBand, why are you making up stuff? You do *not* need any switches. 3 nodes with 3 subnet managers can use 3 wires to connect directly to each other. Did you even read the question? – Navin Mar 25 '16 at 09:46