0

I have a server which is intended to be a vmware host. This server has two Infiniband Mellanox ConnectX-2 dual-port cards. What I want to do is to make the following setup:

The first card should work as a native Ethernet 10G network adapter, talking to Ethernet 10G switch on the other end.

The second card should work as a native Infiniband 40G adapter with IPoIB enabled, talking to Infiniband switch on the other end.

Now, I am installing VMware ESXi 6, which has embedded IB drivers, but they support Ethernet mode only, so I have four Ethernet ports. I've downloaded and installed Mellanox drivers 1.9.10.0 (removing embedded drivers), but they are also Ethernet-only drivers.

Then I've downloaded Mellanox OFED driver 1.8.2.4 and installed them, removing 1.9.10.0. These drivers are Infiniband-only - so now I have four IB ports with no means of switching them to Ethernet. The really interesting thing about this driver is that it's "mlx4_core" module has a parameter called "port_type_array", which is supposed to switch port types between IB and Eth. The problem is, that the package doesn't include "mlx4_en" driver, which is responsible for Ethernet, so when I change it, the ports, that should be Ethernet, simply disappear from the system.

So I have two questions:

1) Is it possible to install mlx4_en driver separately and to link it with the older mlx4_core from OFED package?

2) Is it possible to somehow force VMware to use different drivers for each of my cards?

Tigran Baluyan
  • 61
  • 5
  • Have you tried using SR-IOV? That way, you can pass the VF device directly through to the guests that need it. I'm not really sure the switch or bridge devices used in esxi support RDMA which is a major performance booster for app's that support it. – hookenz May 25 '15 at 08:25
  • I've thought about it, but haven't tried yet. It would be a back-up solution for me, but I would like to have some "native" vmware support... – Tigran Baluyan May 25 '15 at 09:14

1 Answers1

1

ESXi has no real direct construct for dealing with IB other than when it's emulating something else such as Ethernet or FC/SCSI, it just has these Ethernet/DCB switches. What I would do would be to leave the Ethernet drivers in place then user Direct I/O to pass control if the 3rd and 4th controllers to your VMs, that way the OSs inside the VMs can do what they like, though this does rather limit the number of VMs that dan use them. I'm happy to be wrong in this so I'd also advise you to speak with Mellanox, I've been dealing with them on a physical implementation recently and they were very helpful.

Chopper3
  • 100,240
  • 9
  • 106
  • 238