I think I have a Infiniband setup issue. I'm not very experienced with setting up Infiniband or networks. If I try to force MPICH to use IB, I get errors:
[Bryan@node1 shared]$ ./mpich-3.3.1/bin/mpiexec -hosts=node1,node2 -iface=ib0 -n 4 ./test
[mpiexec@node1] HYDU_sock_get_iface_ip (../../../../mpich-3.3.1/src/pm/hydra/utils/sock/sock.c:496): unable to find interface ib0
[mpiexec@node1] HYDU_sock_create_and_listen_portstr (../../../../mpich-3.3.1/src/pm/hydra/utils/sock/sock.c:550): unable to get network interface IP
[mpiexec@node1] HYD_pmci_launch_procs (../../../../mpich-3.3.1/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:114): unable to create PMI port
[mpiexec@node1] main (../../../../mpich-3.3.1/src/pm/hydra/ui/mpich/mpiexec.c:332): process manager returned error launching processes
Similar problem with OpenMPI. I've been trying several days to solve this. Probably something simple that I missed.
More background:
1. I installed a Mellanox ConnectX-3 MCX354A-QCBT into two machines; node1 and node2. Connected port 1 to port 1 directly to each other. No switch. No port 2 connection.
2. Each machine is running Centos 7.
3. Set up passwordless ssh.
4. Installed the Mellanox drivers.
5. Installed MPICH in a shared folder.
6. I start the Mellanox drivers on both machines with sudo mst start.
7. I start opensm on node1. It enters master state.
checking the connection, looks good:
[Bryan@node1 shared]$ sudo ibnodes
Ca : 0x0002c90300ee7620 ports 2 "node2 HCA-1"
Ca : 0x0002c90300ee69e0 ports 2 "node1 HCA-1"
I used ibping as well to test the connection. Everything seems to be fine.
I run a test MPI program without forcing IB, works fine:
[Bryan@node1 shared]$ ./mpich-3.3.1/bin/mpicc -std=c11 MPI_Test.c -o test
[Bryan@node1 shared]$ ./mpich-3.3.1/bin/mpiexec -hosts=node1,node2 -n 4 ./test
Rank 0: Hostname node1
Rank 2: Hostname node1
Rank 1: Hostname node2
Rank 3: Hostname node2
nmcli shows disconnected. I thought that MPI didn't need an IP address for IB, but I tried setting up a connection anyway:
[Bryan@node1 shared]$ sudo nmcli con add con-name ib0 ifname ib0 type infiniband ip4 10.0.0.1
[sudo] password for Bryan:
Connection 'ib0' (09669a93-98b3-4fcb-9fed-ea65fff65e24) successfully added.
[Bryan@node1 shared]$ nmcli device status
DEVICE TYPE STATE CONNECTION
enp4s0 ethernet connected enp4s0
ib0 infiniband connected ib0
ib1 infiniband unavailable --
[Bryan@node1 shared]$ nmcli device show ib0
GENERAL.DEVICE: ib0
GENERAL.TYPE: infiniband
GENERAL.HWADDR: A0:00:02:20:FE:80:00:00:00:00:00:00:00:02:C9:03:00:EE:69:E1
GENERAL.MTU: 2044
GENERAL.STATE: 100 (connected)
GENERAL.CONNECTION: ib0
GENERAL.CON-PATH: /org/freedesktop/NetworkManager/ActiveConnection/54
IP4.ADDRESS[1]: 10.0.0.1/32
IP4.GATEWAY: --
IP4.ROUTE[1]: dst = 10.0.0.1/32, nh = 0.0.0.0, mt = 150
IP6.ADDRESS[1]: fe80::7a74:c87f:2d49:2cfc/64
IP6.GATEWAY: --
IP6.ROUTE[1]: dst = fe80::/64, nh = ::, mt = 150
IP6.ROUTE[2]: dst = ff00::/8, nh = ::, mt = 256, table=255
I did the same with node2 except using 10.0.0.2 as the IP.
If I try to run again forcing IB after setting up the IP's, it will just hang until I ctrl+c. What have I missed?