How to get virtualized SR-IOV Infiniband interface UP?

Question

I've spent several days on this now and I've managed to get SR-IOV working with the Mellanox Infiniband card using the latest firmware.

The Virtual Functions appear in Dom0 as

06:00.1 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function] 06:00.2 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function] 06:00.3 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function] 06:00.4 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]

I've then detached 06:00.1 from Dom0 and assigned it to xen-pciback.

I've passed this into a Xen test domain.

lspci inside the test DomU shows:

00:01.1 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]

I have the following modules loaded in DomU

mlx4_ib
rdma_ucm
ib_umad
ib_uverbs
ib_ipoib

dmesg output for mlx4 drivers shows:

[   11.956787] mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011)
[   11.956789] mlx4_core: Initializing 0000:00:01.1
[   11.956859] mlx4_core 0000:00:01.1: enabling device (0000 -> 0002)
[   11.957242] mlx4_core 0000:00:01.1: Xen PCI mapped GSI0 to IRQ30
[   11.957581] mlx4_core 0000:00:01.1: Detected virtual function - running in slave mode
[   11.957606] mlx4_core 0000:00:01.1: Sending reset
[   11.957699] mlx4_core 0000:00:01.1: Sending vhcr0
[   11.976090] mlx4_core 0000:00:01.1: HCA minimum page size:512
[   11.976672] mlx4_core 0000:00:01.1: Timestamping is not supported in slave mode.
[   12.068079] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (April 4, 2008)
[   12.184072] mlx4_core 0000:00:01.1: mlx4_ib: multi-function enabled
[   12.184075] mlx4_core 0000:00:01.1: mlx4_ib: operating in qp1 tunnel mode

I've even got the ib0 device appearing.

ib0       Link encap:UNSPEC  HWaddr 80-00-05-49-FE-80-00-00-00-00-00-00-00-00-00-00  
          inet addr:10.10.10.10  Bcast:10.10.10.255  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:2044  Metric:1
          RX packets:117303 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:256 
          RX bytes:6576132 (6.5 MB)  TX bytes:0 (0.0 B)

I can even locally ping 10.10.10.10.

However, those pings aren't sent out on the infiniband fabric.

It appears to be because the link is down. ibstat shows:

CA 'mlx4_0'
    CA type: MT4100
    Number of ports: 1
    Firmware version: 2.30.3000
    Hardware version: 0
    Node GUID: 0x001405005ef41f25
    System image GUID: 0x002590ffff175727
    Port 1:
        State: Down
        Physical state: LinkUp
        Rate: 10
        Base lid: 9
        LMC: 0
        SM lid: 1
        Capability mask: 0x02514868
        Port GUID: 0x0000000000000000

How do I get it UP? the domU link is UP but not the VF one?

And the answer is actually found here: According to this link: http://www.spinics.net/lists/linux-rdma/msg13307.html

What do I need for the slave VF's port to become active? I'm running opensm 3.3.13 on a different box, is that new enough? (does SR-IOV require any SM support?)

Yes, as Hal noted, at minimum you need opensm 3.3.14 (http://marc.info/?l=linux-rdma&m=133819320432335&w=2) as it is the 1st version to support alias-guid et al stuff needed for SRIOV, 3.3.15 is also out now, so you want the 2nd version that supports this... basically you need IB link for the PPF and the slave to get an alias guid registered for it @ the SM. We (IL team) were off Tues/Wed as of a holiday, will try to get you further details tonight and if not, by tomorrow, sure.

I've now upgraded OpenSM and will report back soon.

EDIT: OK, it's now working. However I'm getting a log blowout for opensm. The OpenSM process is writing hundreds of entries per second of the form:

Sep 30 20:36:26 707784 [7DC1700] 0x01 -> validate_requested_mgid: ERR 1B01: Wrong MGID Prefix 0x8000 must be 0xFF
Sep 30 20:36:26 707810 [7DC1700] 0x01 -> mcmr_rcv_create_new_mgrp: ERR 1B22: Invalid requested MGID
Sep 30 20:36:26 708096 [8DC3700] 0x01 -> validate_requested_mgid: ERR 1B01: Wrong MGID Prefix 0x8000 must be 0xFF
Sep 30 20:36:26 708119 [8DC3700] 0x01 -> mcmr_rcv_create_new_mgrp: ERR 1B22: Invalid requested MGID
Sep 30 20:36:26 708391 [FF5B0700] 0x01 -> validate_requested_mgid: ERR 1B01: Wrong MGID Prefix 0x8000 must be 0xFF
Sep 30 20:36:26 708421 [FF5B0700] 0x01 -> mcmr_rcv_create_new_mgrp: ERR 1B22: Invalid requested MGID
Sep 30 20:36:26 708696 [3DB9700] 0x01 -> validate_requested_mgid: ERR 1B01: Wrong MGID Prefix 0x8000 must be 0xFF
Sep 30 20:36:26 708719 [3DB9700] 0x01 -> mcmr_rcv_create_new_mgrp: ERR 1B22: Invalid requested MGID

~~And the above error messages went away when I rebooted and gave Dom0 more memory. I currently have 2GB allocated to it with autoballooning off.~~ Unfortunately, they are back with no obvious reason. So I've asked a new question that relates to that here

I'm not really sure why it works in dom0 but in my case I have to have OpenSM running on the Dom0 which has the VF's. I presume this is because the OpenSM instance running on Dom0 knows about the VF's and can advertise them while a subnet manager on another node doesn't? that's my guess. I hope the other xen node will pick up it's VF's as well. That may end up becoming another question. For now it's working with a single Xen node.

In Dom0. sminfo: sm lid 1 sm guid 0x2590ffff1758d1, activity count 40515 priority 0 state 3 SMINFO_MASTER — hookenz, Oct 01 '13 at 03:30
Ok, rebooting and giving Dom0 more memory (2GB) seems to have made those errors go away. I'm not sure if it was more memory or rebooting that seems to have solved it. — hookenz, Oct 01 '13 at 04:13
Thanks a lot! this saved me! I had a SM running on the switch, but this was not sufficient. After I started the SM on one of the nodes (after doing the whole sysfs magic) I now have IB in the VMs! — Jounathaen, Dec 10 '19 at 21:43
@jounathaen - glad someone found this useful. I thought at the time I was breaking new ground. Going where none had gone before. — hookenz, Dec 11 '19 at 03:12

score 1 · Answer 1 · answered Oct 01 '13 at 03:06

1

OpenSM must be installed and started on hypervisor host to bring the state up. Then start start OpenSM with option: PORTS="ALL".

answered Oct 01 '13 at 03:06

Danila Ladner

5,241
21
30

1

OpenSM is already running on another host on the fabric. – hookenz Oct 01 '13 at 03:29
More info added to the bottom of you original question – hookenz Oct 01 '13 at 03:31
In switchless configuration OpenSM should run on both hosts. – Danila Ladner Oct 01 '13 at 03:39
There are two switches. Although I don't think they are managed – hookenz Oct 01 '13 at 03:51
Ok with the last error, can you please show me what MTU size for the interface? – Danila Ladner Oct 01 '13 at 03:54
2044 bytes. You can see that in the above ifconfig output – hookenz Oct 01 '13 at 03:58
hmm, it seems that it doesn't work unless opensm is running on the hypervisor. but then I get a lot of errors. – hookenz Oct 01 '13 at 04:00
2

Why Port GUID: 0x0000000000000000??? Just saw it. It is not supposed to be 0. – Danila Ladner Oct 01 '13 at 13:33
It's not zero's when opensm is loaded on dom0. – hookenz Oct 01 '13 at 21:39
I am having same issue all my `VF` are `Down` I am running `SM` on my HW switch – Satish Feb 02 '22 at 21:14

How to get virtualized SR-IOV Infiniband interface UP?

1 Answers1