How to determine what is using up all TCP ephemeral ports?

Question

My Windows box has recently begun behaving strangely.

After being up for approximately 3-4 days, I am unable to make any outgoing TCP connections. Most applications without specific error handling will simply say "connection failed". Some investigation shows that the error is "No buffer space available."

An Event Log entry tells more of the story:

TCP/IP failed to establish an outgoing connection because the selected local endpoint was recently used to connect to the same remote endpoint. This error typically occurs when outgoing connections are opened and closed at a high rate, causing all available local ports to be used and forcing TCP/IP to reuse a local port for an outgoing connection. To minimize the risk of data corruption, the TCP/IP standard requires a minimum time period to elapse between successive connections from a given local endpoint to a given remote endpoint.

I ran netstat -qno at an admin command prompt and saw the following:

Active Connections

  Proto  Local Address          Foreign Address        State           PID
  TCP    0.0.0.0:135            0.0.0.0:0              LISTENING       972
  TCP    0.0.0.0:445            0.0.0.0:0              LISTENING       4
  TCP    0.0.0.0:3389           0.0.0.0:0              LISTENING       592
  TCP    0.0.0.0:5357           0.0.0.0:0              LISTENING       4
  TCP    0.0.0.0:6160           0.0.0.0:0              LISTENING       2808
  ... some more random TCP connections ...
  TCP    0.0.0.0:49152          0.0.0.0:0              BOUND           4
  TCP    0.0.0.0:49153          0.0.0.0:0              BOUND           4
  TCP    0.0.0.0:49154          0.0.0.0:0              BOUND           4
  TCP    0.0.0.0:49155          0.0.0.0:0              BOUND           4
  TCP    0.0.0.0:49156          0.0.0.0:0              BOUND           4
  TCP    0.0.0.0:49157          0.0.0.0:0              BOUND           4
  TCP    0.0.0.0:49158          0.0.0.0:0              BOUND           4
  TCP    0.0.0.0:49159          0.0.0.0:0              BOUND           4
  TCP    0.0.0.0:49160          0.0.0.0:0              BOUND           4
  TCP    0.0.0.0:49161          0.0.0.0:0              BOUND           4
  ... 16,383 entries, all the way up to ...
  TCP    0.0.0.0:65531          0.0.0.0:0              BOUND           4
  TCP    0.0.0.0:65532          0.0.0.0:0              BOUND           4
  TCP    0.0.0.0:65533          0.0.0.0:0              BOUND           4
  TCP    0.0.0.0:65534          0.0.0.0:0              BOUND           4
  TCP    0.0.0.0:65535          0.0.0.0:0              BOUND           4
  ... some UDP stuff, nothing out of the ordinary ...

As you can see, it is indeed the case that every single TCP outbound port is in the BOUND state, and the owning PID is 4, which represents the System process. (The same process is listening on e.g. port 445 for SMB connections.)

TCPView does not show any of these BOUND connections, only otherwise active connections that were open prior to the issue beginning.

Full malware scan with Defender reports no issues.

I have already tried stopping any unnecessary services and closing any unused applications. Even logging out and back in does not resolve the problem. The only resolution is to reboot the system, and even then, it only resolves the problem for a few more days.

I do note that when typical user applications (e.g. Chrome) use outbound TCP ports, they do show up in the BOUND state but they have the correct PID associated with them in Netstat. This seems to indicate either some serious malware or some kind of major bug in Windows.

Windows 10 1607.

What would be my next step in troubleshooting this issue?

Answer 1

I believe I have found the culprit.

I had done some experimenting with the iSCSI initiatior a while back (connecting it to a NAS4Free instance). I noticed that the iSCSI initiator was still trying to connect - every few minutes - to the iSCSI server I had running. I deleted the IP address of my (decommissioned) server and so far, it seems to have stopped the flood of bound ports.

This would make sense since the iSCSI initiator is a system-level service. Perhaps there's a bug in the service?

Answer 2

There is a windows update that caused this issue to happen with iSCSI connections.

May 9, 2017—KB4019215 https://support.microsoft.com/en-us/help/4019215/windows-8-update-kb4019215

You can remove this update or apply the latest to avoid this issue where new connections are made to deleted iSCSI targets until all the ephemeral ports are exhausted.