0

I attempted to get some help with this question a few days back: Peer to peer connection over SFP+ Passive 10Gbe cable

It was however closed as a duplicate. Now I have setup the physical connections between the two servers and want to see if I can get some help debugging a bit.

On the server I brought up the two devices with the following settings:

eth2      Link encap:Ethernet  HWaddr 00:25:90:55:71:AE  
          inet addr:10.0.0.1  Bcast:10.255.255.255  Mask:255.0.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:357 errors:0 dropped:0 overruns:0 frame:0
          TX packets:338 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:61212 (59.7 KiB)  TX bytes:27137 (26.5 KiB)

eth3      Link encap:Ethernet  HWaddr 00:25:90:55:71:AF  
          inet addr:10.0.0.2  Bcast:10.255.255.255  Mask:255.0.0.0
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:251 errors:0 dropped:0 overruns:0 frame:0
          TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:37566 (36.6 KiB)  TX bytes:4672 (4.5 KiB)

Then on the NAS I brought up the two interfaces with the following settings:

eth2      Link encap:Ethernet  HWaddr 00:1B:21:D4:8A:E0  
          inet addr:10.0.0.3  Bcast:10.255.255.255  Mask:255.0.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:261 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:10000 
          RX bytes:0 (0.0 B)  TX bytes:24296 (23.7 KiB)

eth3      Link encap:Ethernet  HWaddr 00:1B:21:D4:8A:E1  
          inet addr:10.0.0.4  Bcast:10.255.255.255  Mask:255.0.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:197 errors:0 dropped:0 overruns:0 frame:0
          TX packets:200 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:10000 
          RX bytes:18242 (17.8 KiB)  TX bytes:32648 (31.8 KiB)

I am unable to ping from either end of this deal and hoping for some advice on how to further debug this since I was told in the above post that it would work.

Thanks in advance for any and all help and advice.

Community
  • 1
Adam Parrish
  • 143
  • 7
  • How are the 4 connections configured? Both cover the same subnet range (`10.0.0.0/8` - that's a lot of address space to use for this...), so if they're just straight connections between `eth2 <-> eth2` and `eth3 <-> eth3` without anything connecting between them, then you can expect communication to fail as the nodes try to talk to the opposite eth3 address node over eth2 or vice-versa. If they're just peer to peer links, assign them two different IP address ranges; /30 will be plenty large. – Shane Madden Dec 20 '11 at 16:56
  • Shane I actually changed the configuration to be 1.1.1.1 <--> 1.1.1.2 and on the other port 1.1.2.1 <--> 1.1.2.2 with the subnet 255.255.255.252 just as you stated. This worked perfectly and likely I was just tired when I attempted it the night before with the huge subnet. @ShaneMadden thanks. – Adam Parrish Dec 21 '11 at 13:38

1 Answers1

1

It appears that @ShaneMadden above had good advice.

I tried configuring the Server with eth0 as 1.1.1.1 subnet 255.255.255.252 then on eth0 on the NAS I used 1.1.1.2 with same subnet.

I also setup eth1 as 1.1.2.1 and eth1 on NAS as 1.1.2.2 same subnet.

I was able to see that both NFS mounts from the NAS were using their respective 10gbe connections via some NFS performance testing that I found here: http://nfs.sourceforge.net/nfs-howto/ar01s05.html

Namely these two commands:

Writes:

time dd if=/dev/zero of=/mnt/home/testfile bs=16k count=16384

Reads: 

time dd if=/mnt/home/testfile of=/dev/null bs=16k
Adam Parrish
  • 143
  • 7