3

How can I speed up my Windows 7 tcp file transfers over a 100 mbps link between Chicago and London?

Right now, I am transferring two 4GB files in two sftp sessions. Each file transfer is running at a rate of 500 KB/s (4 Mbits/sec). (B == bytes, b == bits, naturally). This is quite slow, as iperf tests show a rate of 16.8 Mbits/sec for a single TCP stream. That's 2 MB/sec.

Starting the second sftp session does not affect the speed of the first one iota. This makes sense to me as I noticed that my iperf tests were limited under TCP- each connection was 16 Mbps which leaves room on the wire for more connections. I was able to make iperf saturate my line by running many parallel connections, and it was a linear increase in bandwidth. That is, with 5 parallel connections I achieved bandwidth of about 5x16 == 80 Mbps.

I note that local file transfers using sftp are around 30 MB/s... quite good. So I don't think SSH channel encryption is the bottleneck.

I have reviewed What is the best way to transfer a single large file over a high-speed, high-latency WAN link? as well as other serverfault questions and there are a lot of interesting tools but what I am looking for is a way for me to adjust Windows' TCP stack to make sure I get my full 2 MB at least... and maybe more, since the iPerf tests were across untuned machines.

I notice, too, that when I drag-and-drop a file on a London machine from my Chicago fileserver, it seems to transfer at quite a nice clip (as shown by looking at the details during the transfer) but after a few minutes it drops off the cliff into the 10's of Kbps. It was very odd.

My latency is steady at around 88 ms, so I think the network infrastructure is good. We have a dedicated line; this is not a VPN over the Internet or anything like that.

========

I have a Linux VM that is "close" to my Windows desktop, network-wise. That is, they are on different subnets but both machines go into switches that both go into another switch that goes to London. Right now I am running an sftp from Linux to London and it is running at 8 MB/s while my desktop continues at 500 KB/s.

======== Ok, I put my Linux laptop on exactly the same switch as my Desktop. Where the desktop is performing an sftp at 500 KB/s at this moment, the Linux laptop is sending a 4Gig file at 7.6 MB/s. To the same server in London, of course.

======== Another test: I was using SecureFX by VanDyke Corp. I also have Cygwin on my PC, and I just went and used the CLI sftp command there. I just increased my throughput fivefold... I'm now getting 2.5 MB/s instead of 500k! :-\ This is on par with my iPerf results of 16 Mbps. Which I would have been content with... although now I notice that my laptop is three times as fast as that.

======== Another test: On my London machine, I started a Windows Explorer file copy from my fileserver in Chicago to the locally-mounted Downloads folder (that is, on an internal hard drive). The throughput was a rock-steady 5.49 MB/s for 12 minutes or so. That's close enough to fine for me! And again, copying via SecureCRT was miserably slow- a steady 500 KB/s.

The only thing I can conclude is that although there may be TCP tweaks that will help, the app you use can make a big difference! Furthermore, I don't know why my filecopy acted so strangely the other day. If it happens again I will attempt to debug.

Thanks for the suggestions and replies.

======================

Ok, I have followed Todd Wilcox' links and figured a few things out:

  • First, according to Microsoft, from https://technet.microsoft.com/en-us/library/cc957546.aspx?f=255&MSPPError=-2147217396, "TCP uses a receive window... For Ethernet networks, the default value of this entry is 0x4470 (17,520, or 12 segments of 1,460 bytes each)" Well I have a WAN with 80ms latency so this is much too small.
  • I have set both GlobalMaxTcpWindowSize and TcpWindowSize in the registry (the latter on my adapter) to 1 MiB, then went to the command line and used netsh to change the Receive Window Auto-Tuning Level from normal to disabled. My network transfer speeds did not change, so I rebooted my machine and again performed an sftp from my desktop to the London server.
  • Throughput plummeted to 700 KB/s. Needless to say, I set everything back and rebooted again. Now my throughput is back to 2.1 MB/s.
  • During the sftp, Wireshark shows a strange sawtooth pattern to the TCP window scaling, going from about 64000 to 62000 bytes then back up again. The cycle takes about 35 seconds.
  • Let's say that my average is 63000 bytes window size. Therefore, my Bps throughput should be (from http://bradhedlund.com/2008/12/19/how-to-calculate-tcp-throughput-for-long-distance-links/), Window-Size (bytes) / Latency (sec) == 700 KB/s. Strange that I'm getting 2.1 MB/s... that's higher than the theoretical! I'm sure there are no Wan accelerators in between or anything more complicated than a Cisco 4948. Maybe Wireshark's window scaling graph is not the same as the window size, I don't know, but Googling told me that that was where I'd find the TCP window size.
  • Things aren't adding up. Although I did find that my twiddling did screw things up, I don't know how to definitively tell what my TCP window sizes are. Furthermore, the oft-sited Brad Hedlund page gives some calculations that are not reflected in my experience.
  • Finally, and this is the greatest lesson I have learned: I think I'll leave well enough alone. If we find we need better throughput, maybe I'll recommend a WAN accelerator. But since we're going halfway across the world we oughtn't expect phenomenal bandwidth from a single TCP connection.
Community
  • 1
Mike S
  • 1,103
  • 5
  • 19
  • 40
  • 1
    "I note that local file transfers using sftp are around 30 MB/s" and "I am looking for is a way for me to adjust Windows' TCP" seem to be at odds. Weren't you using the same Windows TCP stack with the same configuration for the local file transfers that run at 30 MB/s? Seems like the problem lies outside of Windows. Possibly on the dedicated line. – Todd Wilcox May 19 '16 at 16:04
  • Riding off what Todd mentioned, the first thing that came to mind is that this sounds like an issue that is occurring outside of your Windows environment. Is this dedicated line used for anything besides pure data transfer from the remote server? Is it used for any sort of VOIP or video communication? If so I could see QoS kicking in and throttling your transfer rates. What sort of networking equipment lies between your workstation and the server? – mAnwithapc May 19 '16 at 16:23
  • @ToddWilcox absolutely the problem lies outside of Windows, but I think it's a physics problem. That is, we have 88 ms latency and thousands of miles to traverse. I want to modify TCP so it can handle that situation better (possibly at the expense of my speeds within Chicago but I'll deal with that if necessary. I only need to improve the situation for a subset of machines). – Mike S May 19 '16 at 20:28
  • @mAnwithapc - We don't have VOIP or video. At the time of the transfer it's conceivable that someone else is doing something, but remember that I can run iPerf and get a full 100 Mbps, as expected. But each individual TCP socket communication is not speedy enough, I'm guessing because of TCP ACKs and whatnot. I don't know enough TCP tuning to twiddle... only enough to be dangerous. I'm hoping someone can help me twiddle. The networking equipment is: Cisco switch, Cisco switch, Cisco, MetroNID, Cisco gear in my service provider, a really long cable, an ASA, two more switches. Standard stuff. – Mike S May 19 '16 at 20:35
  • BTW, the traffic is policed coming from our MetroNID into the service provider, however we have hard coded our switch port to 100 Meg Full. We were experiencing many iPerf drops prior to the policing but they disappeared once we set our port speed. The network traffic back from London to Chicago is shaped, but I'm not transferring from there to here so that shouldn't come into play in any event. – Mike S May 19 '16 at 20:49
  • I used to deal with a Metro Area Ethernet link that had similar problems. I shrunk the MTU on one of my servers down to about 1300 and got a serious jump in performance.. As far as I can figure somewhere in the middle the packets were getting fragmented. – Tim Brigham May 19 '16 at 20:58
  • Good suggestion, I will try it tomorrow or next week. ...Have to reboot to enable that, it seems. :-( – Mike S May 19 '16 at 21:24
  • 1
    @mikes I'd suggest trying to ping a destination server using the '-l' flag.. It's a good way to test without needing a server reboot. As long as you aren't getting a 'packet needs to be fragmented' the size is OK. – Tim Brigham May 20 '16 at 13:18
  • Mike, did you get everything sorted out and that's why you accepted my answer? I'm curious to know what steps ended up making it all work. – Todd Wilcox May 20 '16 at 16:09
  • @ToddWilcox - Read the last set of bullet points. I haven't sorted everything out- the calculations don't correspond to my observations- but I accepted your answer because it got me to a place where I think I have an understanding of the status quo, and from here I can at least tell my boss that if we want to swap a Terabyte from Chicago to London, he can expect it to take 138.88 hours (1 TB divided by 2 MBps). Or pay for a WAN accelerator, or more equipment on the London side (like a tape drive for example). – Mike S May 20 '16 at 16:46
  • 1
    TCP is a pretty poor protocol for high-latency links. You can try experimenting with UDP-based file transfer tools such as Tsunami UDP: http://tsunami-udp.sourceforge.net/ – Andrew Henle May 20 '16 at 17:40
  • Thanks. There are actually a number of solutions to this issue. One of them is here: http://multipath-tcp.org/. Another UDP based transfer project is here: http://udt.sourceforge.net/ . If worse comes to worse, one can always resort to this: https://en.wikipedia.org/wiki/Sneakernet – Mike S May 20 '16 at 20:15

1 Answers1

3

Formula to calculate optimal TCP window size (source):

Bandwidth-in-bits-per-second * Round-trip-latency-in-seconds = TCP window size in bits / 8 = TCP window size in bytes

In your case: 100 000 000 * .088 = 8 800 000 bits or 1 100 000 bytes

This is configurable in the Windows registry in the TcpWindowSize key in a valid range of 0–0x3FFFFFFF (1 073 741 823 decimal), so that figure is in the valid range.

The default value is the smallest of the following (Note: "Values greater than 64 KB can be achieved only when connecting to other systems that support RFC 1323 Window Scaling"):

  • 0xFFFF
  • GlobalMaxTcpWindowSize (another registry parameter)
  • The larger of four times the MSS (Maximum Segment Size)
  • 16384 rounded up to an even multiple of the MSS

The stack also tunes itself based on the media speed:

  • Below 1 Mbps: 8 KB
  • 1 Mbps – 100 Mbps: 17 KB
  • Greater than 100 Mbps: 64 KB

Source (This link is now mostly dead - slightly alive, but only a miracle could bring it back)

Also see: http://bradhedlund.com/2008/12/19/how-to-calculate-tcp-throughput-for-long-distance-links/

And: https://technet.microsoft.com/en-us/library/cc938219.aspx

Community
  • 1
Todd Wilcox
  • 2,831
  • 2
  • 19
  • 31
  • 1
    thanks. Regarding a different file transfer mechanism, I now have a benchmark and that is a Linux VM that is networkly-near my Windows box (though not on the same VLAN, admittedly). See my updated question- its performance is good, to the same host as Windows. So I can eliminate the protocol, hardware firewalls (there are none), and anti-virus settings on the remote end (it's Linux, and the same host for both transfers). I have disabled my local anti-virus and there is no change in performance. Windows firewall was and remains turned off. – Mike S May 19 '16 at 20:56
  • Regarding your links- I have seen those solutions. While they will make things faster, which I like, they still don't get my TCP connection on Windows from 500 KB/s to something approaching Linux'. – Mike S May 19 '16 at 20:59
  • @MikeS I've just learned a lot I didn't know about this and gutted my answer accordingly. If you've already adjusted the TCP window size, you might update your question with that, along with anything else you've already adjusted in the TCP stack on the Windows machine. I'm also curious how fast you've been able to get in Windows, since knowing that might give clues as to where other problems still exist. – Todd Wilcox May 19 '16 at 21:11
  • 1
    Just updated my question. A lot of my trouble is in the app! Now I've gotten my transfer speed on par with iPerf's, but I just had to go and test it against Linux which was much faster still. So I have a new benchmark to try to reach. :-( – Mike S May 19 '16 at 21:22
  • @MikeS I'm sure there's a reason why "Just use Linux" isn't the answer? – Todd Wilcox May 19 '16 at 21:29
  • 1
    I don't have enough points to add as a comment to the great answer above, I just wanted to share the links are out of date now. A newer technet link for setting the window size is here: https://technet.microsoft.com/en-us/library/cc938219.aspx – KrisTC Jun 15 '17 at 11:25