What follows is a description of the strangest network problem I have encountered.
The story
I am working with a client that reported being unable to check their email, starting this morning. Two weeks ago they completed a big upgrade that saw all of their PCs replaced and a new SBS server added to the Windows domain.
The client fetches email over POP3 using Outlook 2010 on Windows 7, but messages are not being received. The Send/Receive progress window indicates that the download is stalled at about 1% completion.
I tried connecting to the POP3 server using a telnet
client and observed a similar behaviour. After issuing the command RETR 1
I saw small chunks of data (about 1K) coming through, with increasingly long pauses in between. Between chunks of data, the pauses appear to double in length -- I observed pauses that lasted 1, 3, 7, 14, and 28 seconds. I stopped counting after that.
Executing the same test of a POP3 message download by running telnet
on a Terminal Server (Svr 2008 Std) in the same domain yielded the same results.
Then uploaded a small PHP script which would return a page of arbitrary length to a server on the internet and tried accessing it from the Terminal Server on the LAN. I tested various sizes from 1K to 1MB, and no stalls were observed. HTTP appears to be unaffected.
Lastly, I plugged my personal laptop (not a domain member) into the network and tried the POP3 test again -- the entire message downloaded right away.
Update (2011-03-10)
I used Wireshark to get a clear packet capture of the POP3 conversation today. The initial conversation (USER, PASS, LIST) works as expected with the server responding immediately. (The results of LIST fit within a single packet.) After the RETR command is issued and the message begins streaming, the delays start. My earlier estimate was slightly off, and the delays are actually of the expected duration: 1, 2, 4, 8, 16 seconds, etc. The client is sending ACKS right away, within 200 ms of each packet received.
Also, we tried connecting one of the affected workstations to the internet directly, and it was able to download messages at full speed. At this point, I strongly suspect the router (a Cisco 1711) is at fault, but I don't know enough about IOS to conduct further diagnosis.
What I know
- The POP3 server is working fine for clients outside the network.
- The cable modem is delivering a full speed connection.
- The router is probably not malfunctioning, because it worked perfectly when I connected my own machine to the network.
- The L2 switch is delivering LAN traffic at gigabit speed.
- Only the newly installed computers are exhibiting issues.
- The issue started on the new computers nearly two weeks after they were installed.
What I don't know
- What the heck causes this kind of stalling?