Yes, you can directly connect them. But you must ensure that on at least one of the machines you run the opensm subnet manager.
From the software side, if you were using Ubuntu 14.04, install these packages:
sudo apt-get install opensm infiniband-diags librdmacm-dev libibverbs-dev libmlx4-dev
Then add these to /etc/modules
mlx4_ib
ip_ipoib
rdma_ucm
Add an IP address to the card. edit /etc/network/interfaces
auto ib0
iface ib0 inet static
address 10.0.0.1
netmask 255.255.255.0
Reboot.
Opensm will automatically start. Repeat the above for the other machine but use a different IP: 10.0.0.2.
All going well you should be able to ping 10.0.0.2 from 10.0.0.1 and vice versa.
Run some tests:
sudo ibnodes
Oh, there is a bug in scripts for ibnodes and several others. You can fix them by editing them. Mostly they are shell scripts that point to /usr/local/sbin when they are actually calling apps in /usr/sbin.
If you find one, go right ahead and edit it and change IBPATH accordingly.
I have never gotten around to filing a bug report against Ubuntu for that. Hopefully someone else will do it. It's been a problem for ages.
Output will look something like:
Ca : 0x001a4bffff0c9374 ports 2 "HP Lion Cub DDR 128MB"
Ca : 0x001a4bffff0c446c ports 2 "HP Lion Cub DDR 128MB"
Ca : 0x001a4bffff0c4438 ports 2 "HP Lion Cub DDR 128MB"
Switch : 0x000b8cffff006aa8 ports 24 "MT47396 Infiniscale-III Mellanox Technologies" base port 0 lid 2 lmc 0