Not sure I really get 10 Gbit between our servers on the 10 Gbit network

Question

We have a 3 Unix (1 Ubuntu 2 Redhat) servers connected via their 10 Gbit network cards to a 10 Gbit switch (our IT person made sure we got the correct cables). We experience some odd behaviour when sending files between the servers (either via scp 10Gbit-ip or a folder mounted via fstab using 10 Gbit ip). The speed is usually far from 10 Gbit but slightly over 1 Gbit. And on one of the server the progess bar is very quick but then gets stuck on 100 % for a long long time. Also true for scp on that server.

To make sure we got Full-Duplex and 10000 Mbit/s I used two different functions and they give conflicting information. Does someone have any ideas of what is going on? mii-tools and ethtool don't give the same information:

em1 = 1 Gbit network (via 10 Gbit port) em2 = 10 Gbit network (via 10 Gbit port)

ifconfig:

[admin@XXXX ~]$ ifconfig 
em1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet XXXX  netmask 255.255.255.0  broadcast XXXXX
        inet6 fe80::eef4:bbff:fed5:c230  prefixlen 64  scopeid 0x20<link>
        ether ec:f4:bb:d5:c2:30  txqueuelen 1000  (Ethernet)
        RX packets 3938043336  bytes 17103106879433 (15.5 TiB)
        RX errors 0  dropped 28  overruns 0  frame 0
        TX packets 8178327495  bytes 8360136815032 (7.6 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

em2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet XXXXX netmask 255.255.255.0  broadcast XXXX
        inet6 fe80::eef4:bbff:fed5:c232  prefixlen 64  scopeid 0x20<link>
        ether ec:f4:bb:d5:c2:32  txqueuelen 1000  (Ethernet)
        RX packets 3314001  bytes 28089131102 (26.1 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7632971  bytes 7936298958 (7.3 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions

mii-tool:

[admin@XXX ~]$ sudo mii-tool em1
[sudo] password for admin: 
SIOCGMIIREG on em1 failed: Invalid argument
SIOCGMIIREG on em1 failed: Invalid argument
SIOCGMIIREG on em1 failed: Invalid argument
em1: 1000 Mbit, half duplex, link ok

[admin@XXX ~]$ sudo mii-tool em2
SIOCGMIIREG on em2 failed: Invalid argument
SIOCGMIIREG on em2 failed: Invalid argument
SIOCGMIIREG on em2 failed: Invalid argument
em2: 1000 Mbit, half duplex, link ok

ethool:

[admin@XXXX ~]$ sudo ethtool em1
Settings for em1:
    Supported ports: [ TP ]
    Supported link modes:   100baseT/Full 
                            1000baseT/Full 
                            10000baseT/Full 
    Supported pause frame use: No
    Supports auto-negotiation: Yes
    Advertised link modes:  100baseT/Full 
                            1000baseT/Full 
                            10000baseT/Full 
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Speed: 1000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: on
    MDI-X: Unknown
    Supports Wake-on: umbg
    Wake-on: g
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes

[admin@XXXX ~]$ sudo ethtool em2

Settings for em2:
    Supported ports: [ TP ]
    Supported link modes:   100baseT/Full 
                            1000baseT/Full 
                            10000baseT/Full 
    Supported pause frame use: No
    Supports auto-negotiation: Yes
    Advertised link modes:  100baseT/Full 
                            1000baseT/Full 
                            10000baseT/Full 
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Speed: 10000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: on
    MDI-X: Unknown
    Supports Wake-on: umbg
    Wake-on: g
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes

As you can see everything looks good with ethtool (full dubplex and full speed) but not with mii-tool (1000 Mb/s and half duplex). What could be the reason and potential test / fix?

`mii-tool` is outdated, `ethtool` is the way to go. As for (10Gbps) NICs' speed in general: there are some/many ways to enhance the speed. Jumbo frames, flow control, offloading, kernel tuning, etc. The used protocol/application affects the speed as well, of course. — Lenniey, Jan 16 '18 at 13:03
Also don't test from disk-based-filesystem-to-disk-based-filesystem, test from a ramdisk of some form, something that's consistently faster than 10Gbps anyway - otherwise you're not testing your network on its own. — Chopper3, Jan 16 '18 at 13:08
@Lenniey Thanks! I'm trying to test the speed with iperf3 now but I'm having trouble for the servers to find each other. They can ping each other but gives: iperf3: error - unable to connect to server: No route to host when trying iperf3. I have asked our IT guy to check the firewall. Any pointers? — Robin, Jan 16 '18 at 13:09
@Chopper3 Thanks! That is a really good tip. Trying to get iperf3 set-up so I can test the speed from there... Currently probable firewall issues. — Robin, Jan 16 '18 at 13:10
Default iperf server settings use port 5001, so this port obviously shouldn't be blocked by your firewall. — Lenniey, Jan 16 '18 at 13:44
be careful to use different networks for the 1G and 10G connections, or create proper routes. Else the default routes may direct your traffic to the 1G link... For instance let's say Machine1 has 1G IP 192.168.1.1, 10G IP 192.168.1.11 and Machine2 has 1G IP 192.168.1.2 and 10G IP 192.168.1.12. When you connect to 192.168.1.12 from Machine1, it will go through the 1G link, not the 10G link! — wazoox, Jan 18 '18 at 18:04
@Chopper3 I managed to install iperf3 and open the ports. All looks good! Thanks! So I guess a slow transfer could depend on the temporary workload of the NAS (7200 rmp HDD and 10 Gb SSD cache for files < 0.2 MB) that we have hooked up to the 10 Gbit hub. However, the virtual machine running on one of the ubuntu servers (with 10 Gbit) does not reach 10 Gbit in the iperf3 test. The host does. The viirtual machine reaches 1 Gbit on the 1 Gbit network and perhaps 2 Gbit on the 10 Gbit network. — Robin, Jan 22 '18 at 12:54

Not sure I really get 10 Gbit between our servers on the 10 Gbit network

0 Answers0