here is more complete info for persons looking to solve this
Issue faced on RH 6.3 Linux 2.6.32-279.9.1.el6.x86_64 #1 SMP Fri Aug 31 09:04:24 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
**#ibstat**
CA 'mlx4_0'
CA type: MT4099
Number of ports: 2
Firmware version: 2.10.700
Hardware version: 0
Node GUID: 0x0002c90300129780
System image GUID: 0x0002c901013029781
Port 1:
State: Active
Physical state: LinkUp
Rate: 56
Base lid: 1
LMC: 0
SM lid: 1
Capability mask: 0x0251486a
Port GUID: 0x0002c901013029781
Link layer: InfiniBand
1.Create the missing file as root:
**vi /etc/udev/rules.d/90-rdma.rules**
------------ cut here ------------
KERNEL=="umad*", SYMLINK+="infiniband/%k"
KERNEL=="issm*", SYMLINK+="infiniband/%k"
KERNEL=="ucm*", SYMLINK+="infiniband/%k", MODE="0666"
KERNEL=="uverbs*", SYMLINK+="infiniband/%k", MODE="0666"
KERNEL=="uat", SYMLINK+="infiniband/%k", MODE="0666"
KERNEL=="ucma", SYMLINK+="infiniband/%k", MODE="0666"
KERNEL=="rdma_cm", SYMLINK+="infiniband/%k", MODE="0666"
------------ cut here ------------
on the management node (ie. head node, service node etc)
2.Copy this file via ssh or any preferred method to any compute node in the cluster.
**#ssh compute000 cp /home/90-rdma.rules /etc/udev/rules.d/90-rdma.rules**
**#ssh compute001 cp /home/90-rdma.rules /etc/udev/rules.d/90-rdma.rules**
**#ssh compute002 cp /home/90-rdma.rules /etc/udev/rules.d/90-rdma.rules**
**#ssh compute003 cp /home/90-rdma.rules /etc/udev/rules.d/90-rdma.rules**
**#ssh compute004 cp /home/90-rdma.rules /etc/udev/rules.d/90-rdma.rules**
etc
3.Verify that the file is created in folder of every compute node in /etc/udev/rules.d
**#ssh compute000 ls /etc/udev/rules.d | grep rdm
*#90-rdma.rules ***
4.Restart all the compute nodes and management nodes.
NOTE:
a. After the change the user will still get this result when running the command
ibv_devices
[root@master ~]# ibv_devices
device node GUID
------ ----------------
mlx4_0 0002c901013029781
but don't worry just run your preferred mpi application and will be fine.
b. The issue is regardless the use of any HCA vendor, is directly connected to the OS.
c. This seems to be caused by a change made in upstream to the rdma package (no more udev rules), the infiniband devices get created by the kernel with the wrong permissions. This problem has been reported as by users of CentOS 6.3 and Scientific Linux 6.3
Hope will help others