What might cause high latency when bandwidth is saturated?

I've been having network problems for a while and I'm trying to track down the source of the issue.

Yesterday, I was transferring a file from my laptop to my desktop with SCP, over my wifi connection. I when the download started, both computers had low ping to the local router (192.168.1.1, both sat around 10ms-50ms) and the download ran at 2-3 MB/s.

After a minute or so, I found that my desktop's latency to 192.168.1.1 shot up dramatically (> 1,000 ms) and the transfer slowed to a crawl (~200 KB/s). The laptop's latency to 192.168.1.1 stayed the same, however (10-50ms). Once the transfer finished, the desktop's latency dropped back down to its normal range.

Clearly, something goes wrong when the connection is saturated. What could it be? Does this suggest a problem with the router, or a problem with my desktop? Where would be the right place to start looking?

Patrick Collins

Posted 2015-04-16T16:21:29.920

Reputation: 179

Ummm... if a connection is saturated, then it is performing at its maximum capacity. By definition adding further traffic is going to cause packets to be queued and cause latency. – cpt_fink – 2015-04-17T04:40:33.220

@cpt_fink The transfer rate drops to 200 KB/s and stays there, which definitely isn't enough to kill the router. This is also over Wifi across my local network -- I would have hoped that I could max out at least 54 Mb/s. Also, latency stayed low on one computer but not the other --- wouldn't what you're describing cause a network-wide slowdown? – Patrick Collins – 2015-04-17T16:53:45.610

Answers

Google for "buffer bloat".

As RAM became cheap, networking gear added frame buffers so they never have to drop a frame.

Unfortunately, frame drops was how TCP noticed congestion and knew when to back off. Without dropped frames, traditional TCP implementations never notice the congestion and never back off, so they just keep sending at high rates and make it worse.

If all your networking gear just buffers more and more frames during congestion and lets the queue length grow without bound, latency goes up and up as it takes longer and longer to drain the queues.

Active queue management (AQM) techniques and mechanisms like explicit congestion notification (ECN) can alleviate this, but since the issue is not very well-publicized, it's hard to know which products avoid buffer bloat and which don't. It's not like you can look for a "No Buffer Bloat!" logo on the side of a box and know you're getting good equipment.

There are, however, Wi-Fi router after-market firmware distros that specialize in avoiding buffer bloat. At least one distro was specifically developed by the TCP researchers who first recognized the problem, and was used as their research and development platform as they looked for solutions.

Spiff

Posted 2015-04-16T16:21:29.920

Reputation: 84 656

If it's a problem with congestion, why would the latency on my laptop stay low? Are there separate buffers for each connection? I don't have direct control over the router in this case (my landlord runs the network) but I believe that it's some stock Comcast product. – Patrick Collins – 2015-04-16T21:44:04.263

High latency happens when packets make it into a queue to be transmitted across a very busy link. All packets ahead of it in queue must be sent through first. When the demand on a link is great, and queue buffers are large, this leads to high round-trip times.

Nevin Williams

Posted 2015-04-16T16:21:29.920

Reputation: 3 725

So does this imply an issue on the part of the router, or an issue with my wireless drivers? – Patrick Collins – 2015-04-16T21:37:14.673

The high round trip times are a feature, not a bug. Under saturated conditions, either a packet makes it into the queue to eventually be sent, or it gets dropped. Since a ping packet is about 27 times smaller than a fully laden data packet, it has a better chance of getting into a queue, which may not have had room for a full-frame packet.

In saturated conditions, a high round trip time is probably better than no round-trip. Ping's default use of small packets probably misleads somewhat about the amount of packetloss actually occurring. – Nevin Williams – 2015-04-17T06:44:47.120

Okay. So if the router's queue was full, why would the computer sending data (my laptop) still have low latency? If that was the cause, wouldn't it kill everything attached to the network? – Patrick Collins – 2015-04-17T16:54:58.620

"one or more queues" I suspect ICMP gets its own low-priority queue on most consumer routers today, which lets actual data proceed ahead of content-free diagnostic traffic. Else, your pings would have interfered with the transfer, and you'd have a real problem, instead of just a momentarily high ping time. – Nevin Williams – 2015-04-18T18:23:10.453

That still doesn't make sense --- if ICMP got a separate queue, then both pings would have stayed low. – Patrick Collins – 2015-04-18T19:42:50.383

ICMP would be in a queue that only gets processed after other queues are empty, or some variation on that theme. Multiple queues and favoured queue servicing for specific kinds of traffic is the basis of QoS. – Nevin Williams – 2015-04-19T01:05:30.267