1

We've been trying to debug very slow transfer rates for remote clients hitting our web server (IIS on 2k12 R2).

The RTT to the client is about 150ms, and we're struggling to get more than about 500kbps out of our 100Mbps link. The client can speed-test download over 300MBit/s to a local speed test server, so we don't believe the problem is on the client end, but in the sending end.

Wireshark shows that the sender (IIS) is never sending more than about 3 packets without waiting for an ACK. The most bytes-in-flight we ever see is about 20KB. The receive window advertised by the client is about 256kB, so the sender simply is not filling the window.

We've tried a lot of things to fix this, including disabling all off-load options in the NIC settings, turning off heuristics etc etc. It's driving us nuts, as it means our server infrastructure is basically unusable for that purpose.

Router: C2911/K9 Servers: Dell R730s with broadcomm NICs running Hyper-V 2012 R2 hypervisors

We can see upload speed to local speed test servers saturate our link, but I believe only because the latency is very low. Wireshark timestamps show the sender is waiting for ACKs that it shouldn't be waiting for.

Surely IIS doesn't do serial blocking sends with small buffers or something?

Adrien
  • 267
  • 1
  • 6
  • p.s. we even wrote a test server in boost asio which sends a 5MB file as a single send call to make sure it wasn't small blocking send problem - same speed issue. – Adrien Feb 08 '18 at 00:39
  • Looks like it could be congestion. Wireshark on each end shows outbound packets being dropped on the connection somewhere, lots of duplicate ACKs, and taking a long time to re-transmit and ramp up. – Adrien Feb 09 '18 at 04:17

1 Answers1

0

We tracked down the problem for this case. It was a wrong port setting on our router facing our demarcation switch. The router interface was set to 1Gbit, but the fibre is 100Mbit. Flow control was also disabled on the port in the switch, so the router was flooding the switch causing it to drop packets.

Forcing the router interface speed to 100Mbit solved the packet drop problem.

Adrien
  • 267
  • 1
  • 6
  • OUCH. That is one of those stupid little details that never get checked because they never happen. Until one day they DO happen and you spend ages trying to nail that. Nice it was solved. – TomTom Feb 14 '18 at 21:30