Why the bandwidth is not ~10G?

Question

I originally posted on "Network Engineering", but I was told to ask here.

I have 2 servers connected with 10G cable between them directly. I need to know that the connection is good and it has indeed 10G throughput. If not how I can check what is the issue? I used IPERF to check the bandwidth.

UPDATE I configured jumbo packets, extreme interrupt rate, max values for receive/transmit buffers. For the IPERF I used just "iperf3.exe -s" for server and "iperf3.exe -c IP" for client.

According to the test the bandwidth is 1.5-1.7 Gbits/sec. Does it mean the connection is 10G? What could be the problem?

Server A is Windows 8.1 Pro 64bit

Server B is Windows 7 Enterprise 64bit

This is the output when ServerA (192.168.10.100) is server and ServerB (192.168.10.200) is client:

-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.10.200, port 63772
[  5] local 192.168.10.100 port 5201 connected to 192.168.10.200 port 63773
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   170 MBytes  1.42 Gbits/sec
[  5]   1.00-2.00   sec   217 MBytes  1.82 Gbits/sec
[  5]   2.00-3.00   sec   217 MBytes  1.82 Gbits/sec
[  5]   3.00-4.00   sec   214 MBytes  1.80 Gbits/sec
[  5]   4.00-5.00   sec   217 MBytes  1.82 Gbits/sec
[  5]   5.00-6.00   sec   207 MBytes  1.74 Gbits/sec
[  5]   6.00-7.00   sec   199 MBytes  1.67 Gbits/sec
[  5]   7.00-8.00   sec   206 MBytes  1.73 Gbits/sec
[  5]   8.00-9.00   sec   192 MBytes  1.61 Gbits/sec
[  5]   9.00-10.00  sec   209 MBytes  1.75 Gbits/sec
[  5]  10.00-10.16  sec  30.3 MBytes  1.63 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.16  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.16  sec  2.03 GBytes  1.72 Gbits/sec                  receiver

This is the output when B (192.168.10.200) is server and A (192.168.10.100) is client:

-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.10.100, port 55565
[  5] local 192.168.10.200 port 5201 connected to 192.168.10.100 port 55566
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   170 MBytes  1.43 Gbits/sec
[  5]   1.00-2.00   sec   175 MBytes  1.47 Gbits/sec
[  5]   2.00-3.00   sec   175 MBytes  1.47 Gbits/sec
[  5]   3.00-4.00   sec   179 MBytes  1.50 Gbits/sec
[  5]   4.00-5.00   sec   181 MBytes  1.52 Gbits/sec
[  5]   5.00-6.00   sec   200 MBytes  1.68 Gbits/sec
[  5]   6.00-7.00   sec   176 MBytes  1.47 Gbits/sec
[  5]   7.00-8.00   sec   177 MBytes  1.48 Gbits/sec
[  5]   8.00-9.00   sec   186 MBytes  1.56 Gbits/sec
[  5]   9.00-10.00  sec   179 MBytes  1.50 Gbits/sec
[  5]  10.00-10.04  sec  6.42 MBytes  1.46 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.04  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.04  sec  1.76 GBytes  1.51 Gbits/sec                  receiver

UPDATE2 - UDP

When 192.168.10.100 is server

iperf3.exe -c 192.168.10.100 -u -V
iperf 3.1.3
CYGWIN_NT-6.1 ServerB 2.5.1(0.297/5/3) 2016-04-21 22:14 x86_64
Time: Tue, 09 Aug 2016 19:58:03 GMT
Connecting to host 192.168.10.100, port 5201
      Cookie: ServerB.1470772683.561800.7c902a3e559
[  4] local 192.168.10.200 port 63675 connected to 192.168.10.100 port 5201
Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 1
0 second test
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec   128 KBytes  1.05 Mbits/sec  16
[  4]   1.00-2.00   sec   128 KBytes  1.05 Mbits/sec  16
[  4]   2.00-3.00   sec   128 KBytes  1.05 Mbits/sec  16
[  4]   3.00-4.00   sec   128 KBytes  1.05 Mbits/sec  16
[  4]   4.00-5.00   sec   128 KBytes  1.05 Mbits/sec  16
[  4]   5.00-6.00   sec   128 KBytes  1.05 Mbits/sec  16
[  4]   6.00-7.00   sec   128 KBytes  1.05 Mbits/sec  16
[  4]   7.00-8.00   sec   128 KBytes  1.05 Mbits/sec  16
[  4]   8.00-9.00   sec   128 KBytes  1.05 Mbits/sec  16
[  4]   9.00-10.00  sec   128 KBytes  1.05 Mbits/sec  16
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datag
rams
[  4]   0.00-10.00  sec  1.25 MBytes  1.05 Mbits/sec  0.437 ms  0/160 (0%)
[  4] Sent 160 datagrams
CPU Utilization: local/sender 1.7% (0.9%u/0.8%s), remote/receiver 0.1% (0.0%u/0.
1%s)

When 192.168.10.200 is server

iperf3.exe -c 192.168.10.200 -u -V
iperf 3.1.3
CYGWIN_NT-6.3 ServerA 2.5.1(0.297/5/3) 2016-04-21 22:14 x86_64
Time: Tue, 09 Aug 2016 20:37:11 GMT
Connecting to host 192.168.10.200, port 5201
      Cookie: ServerA.1470775031.986727.6f8e913650af9b
[  4] local 192.168.10.100 port 51834 connected to 192.168.10.200 port 5201
Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 1
0 second test
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec   152 KBytes  1.24 Mbits/sec  19
[  4]   1.00-2.00   sec   144 KBytes  1.18 Mbits/sec  18
[  4]   2.00-3.00   sec   160 KBytes  1.31 Mbits/sec  20
[  4]   3.00-4.00   sec   160 KBytes  1.31 Mbits/sec  20
[  4]   4.00-5.00   sec   152 KBytes  1.25 Mbits/sec  19
[  4]   5.00-6.00   sec   152 KBytes  1.24 Mbits/sec  19
[  4]   6.00-7.00   sec   152 KBytes  1.25 Mbits/sec  19
[  4]   7.00-8.00   sec   152 KBytes  1.25 Mbits/sec  19
[  4]   8.00-9.00   sec   152 KBytes  1.24 Mbits/sec  19
[  4]   9.00-10.00  sec   144 KBytes  1.18 Mbits/sec  18
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datag
rams
[  4]   0.00-10.00  sec  1.48 MBytes  1.25 Mbits/sec  0.202 ms  0/189 (0%)
[  4] Sent 189 datagrams
CPU Utilization: local/sender 0.1% (0.1%u/0.0%s), remote/receiver 0.0% (0.0%u/0.
0%s)

UPDATE3 - NTTTCP W/O -NDL

ServerA

ntttcp.exe -s -m 8,*,192.168.10.200 -l 128k -a 2 -t 15
Copyright Version 5.31
Network activity progressing...


Thread  Time(s) Throughput(KB/s) Avg B / Compl
======  ======= ================ =============
     0   15.000        65544.533    131072.000
     1   15.000        65518.933    131072.000
     2   15.000        93585.067    131072.000
     3   15.002        63906.146    131072.000
     4   15.000        65536.000    131072.000
     5   15.000        62310.400    131072.000
     6   15.001        81837.744    131072.000
     7   15.001        73262.316    131072.000


#####  Totals:  #####


   Bytes(MEG)    realtime(s) Avg Frame Size Throughput(MB/s)
================ =========== ============== ================
     8371.875000      15.001       8854.867          558.088


Throughput(Buffers/s) Cycles/Byte       Buffers
===================== =========== =============
             4464.702      17.198     66975.000


DPCs(count/s) Pkts(num/DPC)   Intr(count/s) Pkts(num/intr)
============= ============= =============== ==============
    12120.992         0.823       30450.170          0.328


Packets Sent Packets Received Retransmits Errors Avg. CPU %
============ ================ =========== ====== ==========
      991381           149616           9      0     36.026

ServerB

ntttcp.exe -r -m 8,*,192.168.10.200 -rb 2M -a 16 -t 15
Copyright Version 5.31
Network activity progressing...


Thread  Time(s) Throughput(KB/s) Avg B / Compl
======  ======= ================ =============
     0   15.000        65546.450     60268.990
     1   14.999        65523.318     60229.282
     2   14.999        93582.889     60659.943
     3   14.999        63900.843     61621.870
     4   15.000        65536.817     60278.174
     5   14.999        62316.071     61141.628
     6   14.999        81840.156     60394.032
     7   14.999        73264.934     60858.538


#####  Totals:  #####


   Bytes(MEG)    realtime(s) Avg Frame Size Throughput(MB/s)
================ =========== ============== ================
     8371.320061      14.999       8942.835          558.125


Throughput(Buffers/s) Cycles/Byte       Buffers
===================== =========== =============
             8930.003       5.314    133941.121


DPCs(count/s) Pkts(num/DPC)   Intr(count/s) Pkts(num/intr)
============= ============= =============== ==============
     7965.264         8.216       14260.151          4.589


Packets Sent Packets Received Retransmits Errors Avg. CPU %
============ ================ =========== ====== ==========
      146841           981564           1      0      9.699

UPDATE4 - NTTTCP WITH -NDL

ServerA

ntttcp.exe -s -m 8,*,192.168.10.200 -l 128k -a 2 -t 15 -ndl
Copyright Version 5.31
Network activity progressing...


Thread  Time(s) Throughput(KB/s) Avg B / Compl
======  ======= ================ =============
     0   14.999        78059.604    131072.000
     1   15.000        76902.400    131072.000
     2   14.999        59882.392    131072.000
     3   14.999        62485.232    131072.000
     4   14.999        72734.449    131072.000
     5   15.001        77520.165    131072.000
     6   15.000        62702.933    131072.000
     7   15.000        62771.200    131072.000


#####  Totals:  #####


   Bytes(MEG)    realtime(s) Avg Frame Size Throughput(MB/s)
================ =========== ============== ================
     8101.250000      15.000       8912.094          540.083


Throughput(Buffers/s) Cycles/Byte       Buffers
===================== =========== =============
             4320.667      17.355     64810.000


DPCs(count/s) Pkts(num/DPC)   Intr(count/s) Pkts(num/intr)
============= ============= =============== ==============
     9966.333         0.974       29515.333          0.329


Packets Sent Packets Received Retransmits Errors Avg. CPU %
============ ================ =========== ====== ==========
      953174           145579           1      0     35.182

ServerB

ntttcp.exe -r -m 8,*,192.168.10.200 -rb 2M -a 16 -t 15 -ndl
Copyright Version 5.31
Network activity progressing...


Thread  Time(s) Throughput(KB/s) Avg B / Compl
======  ======= ================ =============
     0   15.000        78054.000     60255.789
     1   15.001        76903.606     60839.375
     2   15.000        59880.183     61786.888
     3   14.999        62483.832     60812.856
     4   15.000        72730.333     61516.405
     5   14.999        77520.335     60398.365
     6   14.998        62701.660     60628.917
     7   14.998        62771.436     60585.766


#####  Totals:  #####


   Bytes(MEG)    realtime(s) Avg Frame Size Throughput(MB/s)
================ =========== ============== ================
     8100.944093      14.999       8943.556          540.099


Throughput(Buffers/s) Cycles/Byte       Buffers
===================== =========== =============
             8641.583       4.888    129615.105


DPCs(count/s) Pkts(num/DPC)   Intr(count/s) Pkts(num/intr)
============= ============= =============== ==============
     7912.594         8.003       14043.670          4.509


Packets Sent Packets Received Retransmits Errors Avg. CPU %
============ ================ =========== ====== ==========
      143914           949785           0      0      8.633

Ok, `ntttcp` shows some saner values, with throughout at about 500 MB/s, still only half of theoretical performance. Does your network adapter support IP/TCP checks offload? If so, try enabling it and retry with the benchmark. Give a look at the configured TCP window size, as described (for example) [here](https://www.duckware.com/blog/how-windows-is-killing-internet-download-speeds/index.html) — shodanshok, Aug 10 '16 at 16:38
@shodanshok, I set `Window Auto-Tuning` to `normal` and now `ntttcp` shows `1550MB/s` bandwidth, which is definitely 10G. Why IPERF shows 1.65Gb/s though? When I use `-u` flag for UDP, IPERF shows `1.05Mb/s`? Why it's slower for UDP? — theateist, Aug 11 '16 at 21:05
1550 MB/s are **beyond** 10 Gb/s capability. Are you sure about the result? Anyway, I think iperf has problem selecting an appropriate TCP window value, but I can't Imagine why UDP results are so low... — shodanshok, Aug 11 '16 at 21:52
@shodanshok, look [here](http://superuser.com/questions/580311/why-is-my-udp-so-slow) for why UDB is low. — theateist, Aug 11 '16 at 22:20
@theateits Oh sure, my bad, I was under impression that you already specified the required bandwidth on the UDP test. Anyway, I'll write an answer based on the test done. If you want, please mark it as accepted for future reference. — shodanshok, Aug 12 '16 at 08:28
@shodanshok, I'm using WireShark to check the tcp window size. I can see that every few packets the windows size drops from 62720 to 17000. How I can check what can be the problem? — theateist, Aug 12 '16 at 17:01
Withthe performance you recorded in the latest runs, I think there are no real problema to address. For better performance, you should probably use a server-oriented operating system. — shodanshok, Aug 12 '16 at 19:05
@shodanshok, verifying that I have 10G connection was one step to understanding the issue I have. I'll create other post then. Thanks anyway. — theateist, Aug 12 '16 at 22:12

score 2 · Accepted Answer · answered Aug 12 '16 at 08:36

It turn out that the problem was related to how Windows 7/8 clients manage the TCP Window scaling.

By default, Windows client editions prefer relatively small TCP window, prioritizing latency vs throughput. When using very fast link, as in this case, setting auto-tuning to normal can bring significant bandwidth gains.

To set auto-tuning to saner value, you can use the command netsh interface tcp set heuristics disabled. Anyway, give a look here for more information.

Why the bandwidth is not ~10G?

1 Answers1