How to prevent TCP ZeroWindow on writing a large file to a Windows share?

Question

I've been given access to a share on a Windows Server 2003 SP1 system (10.a.bbb.ccc) which is a file and printer server, and regularly large files get copied to that share. However, occasionally such a copy fails. When reproducing this issue using Robocopy (on 10.xxx.yy.zzz), I get something like

 70.4%
2013/07/31 11:20:21 ERROR 64 (0x00000040) Copying File <<file name removed>>
The specified network name is no longer available.

Waiting 30 seconds... Retrying...
         New File          105.2 m   <<file name removed>>
  0.0%

dumpcap + Wireshark show that when this happens, in the middle of copying suddenly the server does not accept any data anymore on TCP port 445, by setting the window size to zero:

No.     Time           Source                Destination           Protocol Length Info
   7303 5.841186000    10.a.bbb.ccc          10.xxx.yy.zzz         TCP      60     [TCP ZeroWindow] microsoft-ds > 57918 [ACK] Seq=10864 Ack=6973070 Win=0 Len=0
   7304 6.149715000    10.xxx.yy.zzz         10.a.bbb.ccc          TCP      55     [TCP ZeroWindowProbe] [TCP segment of a reassembled PDU]
   7305 6.150137000    10.a.bbb.ccc          10.xxx.yy.zzz         TCP      60     [TCP ZeroWindowProbeAck] [TCP ZeroWindow] microsoft-ds > 57918 [ACK] Seq=10864 Ack=6973070 Win=0 Len=0
   7306 6.749711000    10.xxx.yy.zzz         10.a.bbb.ccc          TCP      55     [TCP ZeroWindowProbe] [TCP segment of a reassembled PDU]
   7307 6.750087000    10.a.bbb.ccc          10.xxx.yy.zzz         TCP      60     [TCP ZeroWindowProbeAck] [TCP ZeroWindow] microsoft-ds > 57918 [ACK] Seq=10864 Ack=6973070 Win=0 Len=0
   7308 7.946779000    10.xxx.yy.zzz         10.a.bbb.ccc          TCP      55     [TCP ZeroWindowProbe] [TCP segment of a reassembled PDU]
   7309 7.947130000    10.a.bbb.ccc          10.xxx.yy.zzz         TCP      60     [TCP ZeroWindowProbeAck] [TCP ZeroWindow] microsoft-ds > 57918 [ACK] Seq=10864 Ack=6973070 Win=0 Len=0
   7310 10.349783000   10.xxx.yy.zzz         10.a.bbb.ccc          TCP      55     [TCP ZeroWindowProbe] [TCP segment of a reassembled PDU]
   7311 10.350201000   10.a.bbb.ccc          10.xxx.yy.zzz         TCP      60     [TCP ZeroWindowProbeAck] [TCP ZeroWindow] microsoft-ds > 57918 [ACK] Seq=10864 Ack=6973070 Win=0 Len=0
   7312 15.149910000   10.xxx.yy.zzz         10.a.bbb.ccc          TCP      55     [TCP ZeroWindowProbe] [TCP segment of a reassembled PDU]
   7313 15.150283000   10.a.bbb.ccc          10.xxx.yy.zzz         TCP      60     [TCP ZeroWindowProbeAck] [TCP ZeroWindow] microsoft-ds > 57918 [ACK] Seq=10864 Ack=6973070 Win=0 Len=0
   7314 24.747096000   10.xxx.yy.zzz         10.a.bbb.ccc          TCP      55     [TCP ZeroWindowProbe] [TCP segment of a reassembled PDU]
   7315 24.756210000   10.a.bbb.ccc          10.xxx.yy.zzz         TCP      60     [TCP ZeroWindowProbeAck] [TCP ZeroWindow] microsoft-ds > 57918 [ACK] Seq=10864 Ack=6973070 Win=0 Len=0
   7316 43.958531000   10.xxx.yy.zzz         10.a.bbb.ccc          TCP      55     [TCP ZeroWindowProbe] [TCP segment of a reassembled PDU]
   7317 43.958863000   10.a.bbb.ccc          10.xxx.yy.zzz         TCP      60     [TCP ZeroWindowProbeAck] [TCP ZeroWindow] microsoft-ds > 57918 [ACK] Seq=10864 Ack=6973070 Win=0 Len=0
   7318 75.216401000   10.xxx.yy.zzz         10.a.bbb.ccc          TCP      54     57918 > microsoft-ds [RST, ACK] Seq=6973070 Ack=10864 Win=0 Len=0
   7319 75.225543000   10.xxx.yy.zzz         10.a.bbb.ccc          TCP      66     55972 > microsoft-ds [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
   7320 75.225933000   10.a.bbb.ccc          10.xxx.yy.zzz         TCP      66     microsoft-ds > 55972 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460 WS=1 SACK_PERM=1

So after 70 seconds the client (here: Robocopy) calls it quits.

My question: Is this a known issue with Windows shares? What can be investigated/debugged/traced on the file server? Are there specific settings we need to look at or experiment with?

Thanks in advance!

Why the downvote? – MarnixKlooster ReinstateMonica Jul 31 '13 at 20:45 — MarnixKlooster ReinstateMonica, Jul 31 '13 at 20:45
Some people are just odd. Don't take it personally! – Simon Catlin Jul 31 '13 at 21:25 — Simon Catlin, Jul 31 '13 at 21:25

score 3 · Accepted Answer · answered Aug 04 '13 at 16:11

I agree with @suprjami and would like to offer this possible avenue of exploration: rather than the server's disks being too slow, consider that they may either be failing (typical hard drives freeze for about 8 seconds per bad cluster read), severely fragmented, running out of space (causing severe fragmentation and heavy disk thrashing), or you may have a heavily CPU-bound or disk-bound task on the machine starving everything else, including the network and disk subsystems. I recommend checking the server's Event Viewer for disk errors and pulling up Task Manager or Process Explorer with some columns to show page faults, CPU usage, and I/O read bytes and write bytes, and see what the numbers are doing.

Given that ZeroWindow is a symptom of the TCP receive buffer being full, I'm willing to bet the problem is something either consuming 100% CPU on the server or something causing grossly excessive network traffic into the server and locking everything up.

One other possibility is bufferbloat in an intermediate device's TCP implementation, if that device is modifying the packets it relays in any way (i.e. NAT). Do your transfers spike and drop in transfer rate, by chance?

score 0 · Answer 2 · answered Jul 31 '13 at 20:48

0

Before looking at TCP tuning parameters (you can tweak the TCP stack on Windows, but 99.9% of the time, it's auto-scaling works fine), could you give a bit more background? e.g.: what's the spec of the receiving server, are both client and server on the same layer 2 network, what speed are the NICs running at, Etc.

Also, would it be worth experimenting with ROBOCOPY's /IPG parameter, ane perhaps the /R: and /W: (retry and wait) parameters?

answered Jul 31 '13 at 20:48

Simon Catlin

5,222
3
16
20

I don't own/manage the receiving server, but I'll try to get the details and update my question. Regarding experimenting with Robocopy: I'm using that just as a tool to reproduce the problem-- the actual copies done by our production software use different mechanisms, e.g., they are done from Java using I-don't-know-which libraries. I'm seeing several different symptoms in our environment, all of which could be explained by share write actions occasionally failing. – MarnixKlooster ReinstateMonica Aug 01 '13 at 08:47
Ah. Might be difficult to resolve if you can't touch the receiving server (i.e.: the server that's waving the white flag)... – Simon Catlin Aug 04 '13 at 20:00

score 0 · Answer 3 · answered Aug 02 '13 at 22:02

0

TCP Zero Window means the receiving host is overwhemled. It has told the sending host to stop sending data, as it needs need some time to process what it's already got.

It sounds like the disks in the server are too slow, and the client eventually gives up and times the connection out.

Slow your transfer down as it leaves the client, or put faster disks in the server.

answered Aug 02 '13 at 22:02

suprjami

3,476
20
29

1

Thanks! But if slow disks are the problem, then I would not expect it not take more than 70 seconds for all buffers to be emptied? – MarnixKlooster ReinstateMonica Aug 03 '13 at 07:03

How to prevent TCP ZeroWindow on writing a large file to a Windows share?

3 Answers3