After extensive troubleshooting why the TCP speed over an MPLS circuit is slow in one direction, I figured out that by adjusting the TCP window size the situation improved a lot. Maybe you can tell me why.
Details:
- The ping between both hosts is 7ms
- Both machines run Debian Wheezy with kernel 3.2.0-4
- Both machines have default TCP settings
- Both machines have an 10 Gbps uplink
Results with the default TCP Settings:
Location A -> Location B: iperf -c $Location B --> ~2 Gbps
Location B -> Location A: iperf -c $Location A --> 65 - 250 Mbps
After applying the following changes to /etc/sysctl.conf I experienced much better results
net.ipv4.tcp_window_scaling = 1
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 16384 16777216
Results after adjusted TCP Settings:
Location A -> Location B: iperf -c $Location B --> ~2 Gbps
Location B -> Location A: iperf -c $Location A --> 150 Mbps - 2 Gbps (mostly ~1.8 Gbps)
My question is what might negatively affect the TCP window scaling? Here is the output of a tcptrace I have recorded. Maybe it helps to identify the problem.
Thank you for your help!
Regards, Leon Kramer
#
# Speed: ~2 Gbps
#
1 arg remaining, starting with 'tcpdump.lol'
Ostermann's tcptrace -- version 6.6.7 -- Thu Nov 4, 2004
260307 packets seen, 260307 TCP packets traced
elapsed wallclock time: 0:00:17.740455, 14673 pkts/sec analyzed
trace file elapsed time: 0:00:03.798052
TCP connection info:
2 TCP connections traced:
================================
TCP connection 2:
host c: 10.1.2.5:59242
host d: 10.1.2.2:5201
complete conn: RESET (SYNs: 2) (FINs: 0)
first packet: Thu Nov 13 00:05:18.256992 2014
last packet: Thu Nov 13 00:05:21.876883 2014
elapsed time: 0:00:03.619891
total packets: 260276
filename: tcpdump.lol
c->d: d->c:
total packets: 156463 total packets: 103813
resets sent: 0 resets sent: 154
ack pkts sent: 156462 ack pkts sent: 103660
pure acks sent: 1 pure acks sent: 103658
sack pkts sent: 0 sack pkts sent: 8716
dsack pkts sent: 0 dsack pkts sent: 6
max sack blks/ack: 0 max sack blks/ack: 1
unique bytes sent: 674405781 unique bytes sent: 0
actual data pkts: 156461 actual data pkts: 0
actual data bytes: 674415917 actual data bytes: 0
rexmt data pkts: 6 rexmt data pkts: 0
rexmt data bytes: 10136 rexmt data bytes: 0
zwnd probe pkts: 0 zwnd probe pkts: 0
zwnd probe bytes: 0 zwnd probe bytes: 0
outoforder pkts: 3784 outoforder pkts: 0
pushed data pkts: 28 pushed data pkts: 0
SYN/FIN pkts sent: 1/0 SYN/FIN pkts sent: 1/0
req 1323 ws/ts: Y/Y req 1323 ws/ts: Y/Y
adv wind scale: 7 adv wind scale: 7
req sack: Y req sack: Y
sacks sent: 0 sacks sent: 8716
urgent data pkts: 0 pkts urgent data pkts: 0 pkts
urgent data bytes: 0 bytes urgent data bytes: 0 bytes
mss requested: 1460 bytes mss requested: 1460 bytes
max segm size: 14480 bytes max segm size: 0 bytes
min segm size: 37 bytes min segm size: 0 bytes
avg segm size: 4310 bytes avg segm size: 0 bytes
max win adv: 14720 bytes max win adv: 3145728 bytes
min win adv: 14720 bytes min win adv: 14592 bytes
zero win adv: 0 times zero win adv: 0 times
avg win adv: 14720 bytes avg win adv: 3123096 bytes
initial window: 37 bytes initial window: 0 bytes
initial window: 1 pkts initial window: 0 pkts
ttl stream length: NA ttl stream length: NA
missed data: NA missed data: NA
truncated data: 0 bytes truncated data: 0 bytes
truncated packets: 0 pkts truncated packets: 0 pkts
data xmit time: 3.612 secs data xmit time: 0.000 secs
idletime max: 43.3 ms idletime max: 43.3 ms
throughput: 186305549 Bps throughput: 0 Bps
====================================================================================
#
# Speed: ~65 Mbps
#
1 arg remaining, starting with 'tcpdump.lol'
Ostermann's tcptrace -- version 6.6.7 -- Thu Nov 4, 2004
652370 packets seen, 652370 TCP packets traced
elapsed wallclock time: 0:00:07.985123, 81698 pkts/sec analyzed
trace file elapsed time: 0:02:13.121598
TCP connection info:
2 TCP connections traced:
================================
TCP connection 2:
host c: 10.1.2.2:45259
host d: 10.1.2.5:5201
complete conn: RESET (SYNs: 2) (FINs: 1)
first packet: Wed Nov 12 22:42:06.606682 2014
last packet: Wed Nov 12 22:44:19.553859 2014
elapsed time: 0:02:12.947177
total packets: 652339
filename: tcpdump.lol
c->d: d->c:
total packets: 446774 total packets: 205565
resets sent: 0 resets sent: 22
ack pkts sent: 446773 ack pkts sent: 205543
pure acks sent: 1 pure acks sent: 205541
sack pkts sent: 0 sack pkts sent: 9322
dsack pkts sent: 0 dsack pkts sent: 2
max sack blks/ack: 0 max sack blks/ack: 3
unique bytes sent: 1058366405 unique bytes sent: 0
actual data pkts: 446772 actual data pkts: 0
actual data bytes: 1058379437 actual data bytes: 0
rexmt data pkts: 5 rexmt data pkts: 0
rexmt data bytes: 13032 rexmt data bytes: 0
zwnd probe pkts: 0 zwnd probe pkts: 0
zwnd probe bytes: 0 zwnd probe bytes: 0
outoforder pkts: 1577 outoforder pkts: 0
pushed data pkts: 5733 pushed data pkts: 0
SYN/FIN pkts sent: 1/0 SYN/FIN pkts sent: 1/1
req 1323 ws/ts: Y/Y req 1323 ws/ts: Y/Y
adv wind scale: 7 adv wind scale: 7
req sack: Y req sack: Y
sacks sent: 0 sacks sent: 9322
urgent data pkts: 0 pkts urgent data pkts: 0 pkts
urgent data bytes: 0 bytes urgent data bytes: 0 bytes
mss requested: 1460 bytes mss requested: 1460 bytes
max segm size: 18824 bytes max segm size: 0 bytes
min segm size: 37 bytes min segm size: 0 bytes
avg segm size: 2368 bytes avg segm size: 0 bytes
max win adv: 14720 bytes max win adv: 561920 bytes
min win adv: 14720 bytes min win adv: 14592 bytes
zero win adv: 0 times zero win adv: 0 times
avg win adv: 14720 bytes avg win adv: 533768 bytes
initial window: 37 bytes initial window: 0 bytes
initial window: 1 pkts initial window: 0 pkts
ttl stream length: NA ttl stream length: 0 bytes
missed data: NA missed data: 0 bytes
truncated data: 0 bytes truncated data: 0 bytes
truncated packets: 0 pkts truncated packets: 0 pkts
data xmit time: 132.940 secs data xmit time: 0.000 secs
idletime max: 37.8 ms idletime max: 37.8 ms
throughput: 7960804 Bps throughput: 0 Bps
