I have the following configuration:
I------------------I I------------------I
I I(P0)---------(P0)I I
I Voltaire HCA 0 I I Voltaire HCA 1 I
I I(P1)---------(P1)I I
I------------------I I------------------I
The two HCAs are directly connected and I have managed to get opensm working to some extent. The odd thing is that I can ping to and from every interface but one. I can ping to it but not from it. Although I have significant experience in IP networks I'm somewhat at a loss when it comes to infiniband. The sole function of this is to have two NFS shares (allowing 10G bandwidth to each).