Installing Mellanox ConnectX-2 10gbe PCIe Card on Centos 6

Question

I just installed a Mellanox ConnectX-2 10gbe PCIe x8 Card into my server running CentOS 6.7. On boot, dmesg shows the mlx4_core driver being loaded automatically however I see no eth1 device corresponding to the card. ifconfig -a only shows eth0, which is my onboard/motherboard Gigabit network interface.

So, I deleted /etc/udev/rules.d/70-persistent-net.rules and rebooted. It re-created 70-persistent-net.rules but only with my onboard/motherboard gigabit NIC, no entry for the Mellanox 10gbe card.

Output of /etc/udev/rules.d/70-persistent-net.rules

# PCI device 0x10ec:0x8168 (r8169)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:30:67:aa:bb:cc", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

Good news is, lspci shows the Mellanox card. Output of lspci | grep -i ether

02:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 03)

Relevant output from dmesg:

mlx4_core: Mellanox ConnectX core driver v2.2-1 (Feb, 2014)
mlx4_core: Initializing 0000:02:00.0
mlx4_core 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
mlx4_core 0000:02:00.0: setting latency timer to 64
  alloc irq_desc for 26 on node -1
  alloc kstat_irqs on node -1
mlx4_core 0000:02:00.0: irq 26 for MSI/MSI-X
  alloc irq_desc for 27 on node -1
  alloc kstat_irqs on node -1
mlx4_core 0000:02:00.0: irq 27 for MSI/MSI-X
  alloc irq_desc for 28 on node -1
  alloc kstat_irqs on node -1
mlx4_core 0000:02:00.0: irq 28 for MSI/MSI-X
  alloc irq_desc for 29 on node -1
  alloc kstat_irqs on node -1
mlx4_core 0000:02:00.0: irq 29 for MSI/MSI-X
  alloc irq_desc for 30 on node -1
  alloc kstat_irqs on node -1
mlx4_core 0000:02:00.0: irq 30 for MSI/MSI-X
  alloc irq_desc for 31 on node -1
  alloc kstat_irqs on node -1
mlx4_core 0000:02:00.0: irq 31 for MSI/MSI-X
  alloc irq_desc for 32 on node -1
  alloc kstat_irqs on node -1
mlx4_core 0000:02:00.0: irq 32 for MSI/MSI-X
  alloc irq_desc for 33 on node -1
  alloc kstat_irqs on node -1
mlx4_core 0000:02:00.0: irq 33 for MSI/MSI-X
  alloc irq_desc for 34 on node -1
  alloc kstat_irqs on node -1
mlx4_core 0000:02:00.0: irq 34 for MSI/MSI-X
  alloc irq_desc for 35 on node -1
  alloc kstat_irqs on node -1
mlx4_core 0000:02:00.0: irq 35 for MSI/MSI-X
  alloc irq_desc for 36 on node -1
  alloc kstat_irqs on node -1
mlx4_core 0000:02:00.0: irq 36 for MSI/MSI-X
  alloc irq_desc for 37 on node -1
  alloc kstat_irqs on node -1
mlx4_core 0000:02:00.0: irq 37 for MSI/MSI-X

Any ideas why my Mellanox ConnectX-2 is not showing up as an eth1 device?

ewwhite · Answer 1 · 2016-06-02T17:40:52.327

This card should work natively without any changes. Just remove your 70-persistent-net.rules/reboot and/or check dmesg output.

You may have an issue with a UUID or MAC in the /etc/sysconfig/network-scripts/ifcfg-ethX files.

If neither of those are the case, download the driver from Mellanox or HP.

# modinfo  mlx4_en
filename:       /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-en/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko
version:        2.3-2.0.1 (Dec  7 2014)
license:        Dual BSD/GPL
description:    Mellanox ConnectX HCA Ethernet driver
author:         Liran Liss, Yevgeny Petrilin
srcversion:     86DD1A5FE5CF17268AFBB0D
depends:        mlx4_core,compat,ptp
vermagic:       2.6.32-504.el6.x86_64 SMP mod_unload modversions
parm:           inline_thold:threshold for using inline data (uint)
parm:           udp_rss:Enable RSS for incoming UDP traffic (uint)
parm:           num_lro:Dummy module parameter to prevent loading issues (uint)
parm:           pfctx:Priority based Flow Control policy on TX[7:0]. Per priority bit mask (uint)
parm:           pfcrx:Priority based Flow Control policy on RX[7:0]. Per priority bit mask (uint)

I already tried deleting `/etc/udev/rules.d/70-persistent-net.rules` and rebooting, as mentioned above. It did not automatically add the Mellanox card to the `/etc/udev/rules.d/70-persistent-net.rules` file. It only adds my onboard/motherboard Gigabit network interface. — Crash Override, Jun 02 '16 at 17:49

Crash Override · Accepted Answer · 2016-06-08T03:13:49.007

I figured out the solution. All I had to do was enter two commands to activate the network card in CentOS 6:

sudo modprobe mlx4_core
sudo modprobe mlx4_en

Now I have eth1 in ifconfig -a and can configure it (IP address, netmask, etc.)

To make it permanent across reboots, I added a new file /etc/sysconfig/modules/mellanox.modules and set its permissions (chmod) to 755, owner/group as root:

#!/bin/sh
/sbin/modprobe mlx4_core
/sbin/modprobe mlx4_en

This automatically loads the Mellanox kernal module built-in to CentOS on bootup, just before activating the eth0/eth1 configs specified in /etc/sysconfig/network-scripts

Now it works just like any other network interface.

Installing Mellanox ConnectX-2 10gbe PCIe Card on Centos 6

2 Answers2