Why am I only achieving 2.5Gbps over a 10Gbe direct connection between 2 machines?

Question

I have 2 machines directly connected to each other with a 7 foot Cat6a ethernet cable (included in the box of the NIC cards). The PCIe x4 NIC I bought and in both machines is this: https://www.amazon.com/gp/product/B07CW2C2J1

I'm trying to debug why I'm getting almost exactly 2500Mbps transfer between these 2 machines. Any tips or obvious errors I'm overlooking to acheive closer to 10Gbps?

Here is what I've tested:

Configuration (Machine A)

Machine A ifconfig:

enp7s0    Link encap:Ethernet  HWaddr 24:5e:be:2c:c1:53  
      inet addr:2.0.0.20  Bcast:2.0.0.255  Mask:255.255.255.0
      inet6 addr: fe80::265e:beff:fe2c:c153/64 Scope:Link
      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
      RX packets:17225416 errors:0 dropped:0 overruns:0 frame:0
      TX packets:7021731 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1000 
      RX bytes:25712055299 (25.7 GB)  TX bytes:9701557546 (9.7 GB)

Machine A ip link:

3: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 24:5e:be:2c:c1:53 brd ff:ff:ff:ff:ff:ff

Machine A ethtool enp7so:

Settings for enp7s0:
    Supported ports: [ TP ]
    Supported link modes:   100baseT/Full 
                            1000baseT/Full 
                            10000baseT/Full 
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Advertised link modes:  100baseT/Full 
                            1000baseT/Full 
                            10000baseT/Full 
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Speed: 10000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: on
    MDI-X: Unknown
    Supports Wake-on: g
    Wake-on: g
    Link detected: yes

Configuration (Machine B)

Machine B ifconfig:

enp101s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
    inet 2.0.0.10  netmask 255.255.255.0  broadcast 2.0.0.255
    inet6 fe80::265e:beff:fe2c:c0dc  prefixlen 64  scopeid 0x20<link>
    ether 24:5e:be:2c:c0:dc  txqueuelen 1000  (Ethernet)
    RX packets 2332894765  bytes 3532248694886 (3.5 TB)
    RX errors 0  dropped 0  overruns 0  frame 0
    TX packets 107128853  bytes 32005739542 (32.0 GB)
    TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Machine B ip link:

3: enp101s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 24:5e:be:2c:c0:dc brd ff:ff:ff:ff:ff:ff

Machine B ethtool enp101s0:

Settings for enp101s0:
Supported ports: [ TP ]
Supported link modes:   100baseT/Full 
                        1000baseT/Full 
                        10000baseT/Full 
                        2500baseT/Full 
                        5000baseT/Full 
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes:  100baseT/Full 
                        1000baseT/Full 
                        10000baseT/Full 
                        2500baseT/Full 
                        5000baseT/Full 
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
MDI-X: Unknown
Link detected: yes

Debug Steps So Far

I did a netcat on /dev/zero on one machine to /dev/null on the other (B -> A):

3.15GiB 0:00:09 [ 353MiB/s]

I also ran ifperf with 2 window sizes (the default 64k and the below 256k) and saw identical results:

iperf -s -w 256k
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  416 KByte (WARNING: requested  250 KByte)
------------------------------------------------------------
[  4] local 2.0.0.10 port 5001 connected with 2.0.0.20 port 55364
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  2.85 GBytes  2.45 Gbits/sec

Just to test to make sure and remove the network variable in the transfer:

cat /dev/zero | pv > /dev/null
21.0GiB 0:00:04 [5.18GiB/s]

Are these in a QNAP NAS or are they in regular desktop PCs? If it is in a qnap, some of those devices probably can't saturate a 10G link. I'd check your CPU utilisation during these test transfers. — Mark Henderson, Jan 31 '19 at 21:16
"_Cat6e ethernet cable_" There is no such thing. The current cable categories are 3, 5e, 6, 6a, and 8. — Ron Maupin, Jan 31 '19 at 21:17
@RonMaupin that was a typo :) I meant 6a. Updating... Thanks for the catch — Mike, Jan 31 '19 at 21:20
@MarkHenderson they are regular desktop PCs, just using QNAP cards NICs — Mike, Jan 31 '19 at 21:22

Dmitriy Kupch · Answer 1 · 2019-01-31T21:16:49.467

1

I remember I faced the same issue once. It was all related to LRO and GRO. You may want to disable large-receive-offload and generic-receive-offload on both ends and see if it makes the changes.

Run the following to temporarily disable those:

large-receive-offload

ethtool -K enp101s0 lro off

ethtool -K enp7s0 lro off

generic-receive-offload

ethtool -K enp101s0 gro off

ethtool -K enp7s0 gro off

By running the following you will actually examine the changes:

ethtool -k enp7s0

please note that -k -K, uppercase makes changes and lowercase simply outputs the values

I always disable GRO and LRO on 10 NICs.

You can read more here: https://lwn.net/Articles/358910/

But try to disable it and see if it changes the speed, if it will, then I will give you the steps to make a permanent change in your /etc/network/interfaces.

edited Jan 31 '19 at 21:16

answered Jan 31 '19 at 21:10

Dmitriy Kupch

451
2
6

Thanks for the response! I will test and report back to you in ~ 4 hours. Rebooted Machine B remotely and forgot I don't have sshd launch on boot yet. – Mike Jan 31 '19 at 21:23
@Mike were you able to verify the solution? – Dmitriy Kupch Feb 01 '19 at 15:51
Ah! Sorry for the late response. It did help a little (avg 2.9Gbps). I've been doing some more research and it appears that there are a TON of variables that need to be tweak to optimize for 10gbps on this card + ubuntu. – Mike Feb 02 '19 at 15:36

score 0 · Answer 2 · answered Jun 21 '22 at 11:50

0

There may also a other factor that causes the slow transfer. I stumbled on to a case studi, where ssh was pointed out as a cause for the slow transmission. see https://www.intel.com/content/dam/support/us/en/documents/network/sb/fedexcasestudyfinal.pdf

answered Jun 21 '22 at 11:50

Rob

1

As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Jun 23 '22 at 07:43

score 0 · Answer 3 · answered Jun 21 '22 at 19:06

Running iperf, a 256k window simply isn't sufficient to saturate a 10G link.

Doing the math: 256 kB * 8 bit/byte / 2.56 Gbit/s = .8 ms

So, if the RTT is .8 ms - quite reasonable for directly connected 10GBASE-T - there can't be any more throughput than those 2.56 Gbit/s, regardless the bandwidth. Try quadrupling the window.

Also, make sure that the PCIe slots you put those NICs in have sufficient bandwidth. The mechanical size is sometimes misleading, you need to check the electrical size. The NICs support PCIe 3.0 x4, so you need at least PCIe 2.0 x4 or PCIe 3.0 x2 for uncompromised speed.

Offloading features should actually speed up the connection (by lowering CPU overhead), but they might not work correctly, depending on the hardware implementation and the driver.

Why am I only achieving 2.5Gbps over a 10Gbe direct connection between 2 machines?

Configuration (Machine A)

Configuration (Machine B)

Debug Steps So Far

3 Answers3

Linked