Can not setup Mellanox connectx-2 in 40g speed

Question

I have Mellanox connectX-2 network card (MT26428) and I installed MLNX_OFED_LINUX-3.4-1.0.0.0-ubuntu16.04-x86_64 driver from Mellanox repository but I'm wondering this equipment setup 20G at maximum although I expected it to setup 40G instead.

here is the result of ibstatus:

Infiniband device 'mlx4_0' port 1 status:
  default gid:   0000:0000:0000:0000:0000:0000:0000:0000
  base lid:  0x6
  sm lid:        0x2f
  state:         4: ACTIVE
  phys state:    5: LinkUp
  rate:      20 Gb/sec (4X DDR)
  link_layer:    InfiniBand

Infiniband device 'mlx4_0' port 2 status:
  default gid:   0000:0000:0000:0000:0000:0000:0000:0000
  base lid:  0x0
  sm lid:        0x0
  state:         4: ACTIVE
  phys state:    5: LinkUp
  rate:      10 Gb/sec (1X QDR)
  link_layer:    Ethernet

Infiniband device 'mlx4_1' port 1 status:
  default gid:   0000:0000:0000:0000:0000:0000:0000:0000
  base lid:  0x0
  sm lid:        0x0
  state:         1: DOWN
  phys state:    2: Polling
  rate:      10 Gb/sec (4X)
  link_layer:    InfiniBand

Infiniband device 'mlx4_1' port 2 status:
  default gid:   0000:0000:0000:0000:0000:0000:0000:0000
  base lid:  0x0
  sm lid:        0x0
  state:         4: ACTIVE
  phys state:    5: LinkUp
  rate:      10 Gb/sec (1X QDR)
  link_layer:    Ethernet

note that the value of default gid is reasonable and I change it for sake of security issues.

note that these connectX-2 cards are connected by a switch (model) with 12m QSFP copper cable. I also test it with 10m and 1m QSFP copper cable in back to back manner without the involving switch and get the same result, unfortunately.

Did you change the GUIDs only in the output shown here, or did you actually change them on your adapter configuration? The GUIDs must be unique for proper operation. — Paul, Jan 09 '19 at 21:26
Yes, I changed the GIDs for maybe security reasons! :D they are absolutely different from some sort of zeros. and yes they were unique. — MDK, Jan 11 '19 at 06:27

score 0 · Answer 1 · answered Jan 09 '19 at 21:32

This is probably a cable incompatibility issue. Without posting the model of the cables, it's impossible to know if the cables you are using are QDR capable. Note that it's easy to mix up DDR, QDR, and FDR cables because they all can look the same. Model verification is the only way to know of a cable's capabilities.

You can find detailed cable information by running ibdiagnet from Mellanox OFED. Note that the cable identification plugin is not available in the distribution provided utilities.

ibdiagnet --get_cable_info

There will be a file output, ibdiagnet2.cables. Look at the program output for the exact path. For each system/port/LID you'll find details for the cables that are plugged in.

Can not setup Mellanox connectx-2 in 40g speed

1 Answers1