Very low throughput on 10GbE network

Question

I have two Linux machines, each equipped with a Solarflare SFN5122F 10GbE NIC. The two NICs are connected together with an SFP+ Direct Attach cable.

I am using netperf to measure TCP throughput between the two machines. On one box, I run:

netserver

and on the other:

netperf -t TCP_STREAM -H 192.168.x.x -- -m 32768

I get:

MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.x.x (192.168.x.x) port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  32768    10.02    1321.34

The measured throughput is 1.3Gb/s. This is 7.5x below the theoretical maximum, and only 30% faster than 1GbE.

What steps can I take to troubleshoot this?

@gekkz: Changing MTU to 9000 on both ends made no perceptible difference. — NPE, Apr 10 '12 at 20:12
Then is the CPU of the sending system the bottleneck? Random data generation at that speed can be expensive. — Shane Madden, Apr 10 '12 at 20:16
What is the hardware of the boxes involved? You cannot achieve 10GbE line rate with anything less than modern Westmere/Nehalem/Sandy Bridge boxes. — pfo, Apr 10 '12 at 20:55
I've had the exact same problem for over a year. I think the problem is with 10GbE. I'm thinking that even 8Gbps Fibre Channel is much better at throughput than 10GbE. — cohortq, Mar 05 '13 at 18:22

pQd · Answer 1 · 2012-04-11T10:06:19.437

few things:

did you try adjusting mtu to make use of jumbo-frames?
are you absolutely sure that the link between the two servers does not have any packet losses?
does ethtool show you any errors on the interfaces on both ends?
what does top/atop say during the prolonged test - do you see any of the cores fully occupied by iowait?

you'll [most probably] not achieve full 10gbit on a single tcp session but you can do some additional tuning to get closer to it by tweaking tcp handling and communication with the network card itself - take a look here or here .

Very low throughput on 10GbE network

1 Answers1