10g network cards running at less than 1 gigabit

Question

I have installed 10 gigabit fiber-optic cards in two of my servers, but when I test their throughput they run at only around 600Mbps or so:

$ scp random.1G rcm2:/tmp
random.1G 100% 1024MB  73.1MB/s   00:14

My routes look correct. The network drivers and hardware seem to be setup correctly.

Can someone please tell me where I should be looking next?

Here is more information:

Routing

172.16.157.34

$ ip route get 172.16.157.35
172.16.157.35 dev eth1  src 172.16.157.34

172.16.157.35

$ ip route get 172.16.157.34
172.16.157.34 dev eth2  src 172.16.157.35

Hardware/ifconfig

172.16.157.34

$ ifconfig
...
eth2      Link encap:Ethernet  HWaddr 90:e2:ba:38:aa:ee  
          inet addr:172.16.157.34  Bcast:172.16.157.63  Mask:255.255.255.224
...
$ lshw -class network
   ...
   description: Ethernet interface
   product: 82599EB 10-Gigabit SFI/SFP+ Network Connection
   ...
   serial: 90:e2:ba:38:aa:ee
   ...

172.16.157.35

$ ifconfig
...
eth1      Link encap:Ethernet  HWaddr 90:E2:BA:38:AA:82  
          inet addr:172.16.157.35  Bcast:172.16.157.63  Mask:255.255.255.224
...
$ lshw -class network
   ...
   description: Ethernet interface
   product: 82599EB 10-Gigabit SFI/SFP+ Network Connection
   ...
   serial: 90:e2:ba:38:aa:82
   ...

Driver setup

172.16.157.34

$ sudo ethtool eth2 
Settings for eth2:
    Supported ports: [ FIBRE ]
    Supported link modes:   1000baseT/Full 
                            10000baseT/Full 
    Supported pause frame use: No
    Supports auto-negotiation: Yes
    Advertised link modes:  1000baseT/Full 
                            10000baseT/Full 
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Speed: 10000Mb/s
    Duplex: Full
    Port: FIBRE
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: on
    Supports Wake-on: d
    Wake-on: d
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes

172.16.157.35

$ sudo ethtool eth1
[sudo] password for jsp: 
Settings for eth1:
    Supported ports: [ FIBRE ]
    Supported link modes:   1000baseT/Full 
                            10000baseT/Full 
    Supported pause frame use: No
    Supports auto-negotiation: Yes
    Advertised link modes:  1000baseT/Full 
                            10000baseT/Full 
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Speed: 10000Mb/s
    Duplex: Full
    Port: FIBRE
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: on
    Supports Wake-on: d
    Wake-on: d
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes

What kind of drives/RAID controller are you running. 600Mbps is about all an 11G Dell PERC700 controller can push. Do you see the same issue when you drop the 1G file onto a ramdisk and copy it to a ramdisk? — Zypher, Jul 25 '13 at 19:40
Use iperf for testing link speed. It is a far better tool for this purpose. — Zoredache, Jul 25 '13 at 19:57

longneck · Accepted Answer · 2013-07-25T23:02:16.803

2

scp has a large overhead for encrypting the data while in flight. Something thinner like nc will give you better throughput and a more realistic test of the physical capabilities of your server and network.

Also, make sure your bottleneck doesn't exist somewhere else like your disks. Use iperf or something similar to test non-diskbound throughput.

edited Jul 25 '13 at 23:02

answered Jul 25 '13 at 19:25

longneck

22,793
4
50
84

Thanks, I will look into using `nc`. However, I doubt that scp's overhead is causing a 10X slowdown. In fact, the throughput of my 10G network is the same as the throughput of my 1G network, so I doubt that my choice of benchmark is the cause of the problem. – jsp Jul 25 '13 at 19:37
1

Your premise is invalid. 73.1 MB/s is 500 mbps. So you are not reaching the limit of your 1G network, either. – longneck Jul 25 '13 at 19:38
I'm not sure I follow. My 1G network runs at ~500bmps. My 10G network also runs at ~500mbps. This implies to me that my 10G network is not really running at 10G. – jsp Jul 25 '13 at 19:47
I do not know scp or nc, but the rated speed in xbps can only be attained if the packets being sent are of max size. The rated pps can only be attained if the packets aare min size. FWIW. – dbasnett Jul 25 '13 at 19:51
1

@jsp Your 1G network should be capable of more like 900 mbps of payload throughput. the fact that you only get 500 mbps means your bottleneck is somewhere else, i.e. the encryption overhead of `scp`. – longneck Jul 25 '13 at 19:53
3

`t ~500bmps. My 10G network also runs at ~500mbps.`. Not really, It is more likely that it implies that your **DRIVES** may be operating at 500mbps. Use a tool that doesn't depend on disk I/O for testing your network capacity. – Zoredache Jul 25 '13 at 19:59
@Zoredache good point. – longneck Jul 25 '13 at 20:10
@Zoredache i get get your point, i'll try iperf – jsp Jul 25 '13 at 20:25
@longneck duh, OK i get it, thanks for your help. i tried using iperf -- it was easier for me than using nc -- and I am indeed running at 10G. thanks again. – jsp Jul 25 '13 at 20:42
If you want to use drive independent network traffic speed, you should get your data from /dev/zero and write them to /dev/null, something like this: `dd if=/dev/zero bs=4096 count=1048576 | ssh user@host 'cat > /dev/null'`. Because /dev/zero compresses well over ssh, but ssh uses encryption, it kind of balances itself out. Alternatively, you could use netcat and /dev/urandom (`nc -l -p 1111 dd of=/dev/null` to listen on the receiving host, `dd if=/dev/urandom bs=4096 count=1048576 | nc host 1111` on the originating host) – Ursula Jul 26 '13 at 22:16

10g network cards running at less than 1 gigabit

Routing

Hardware/ifconfig

Driver setup

1 Answers1