Ephemeral port collision

Question

There's a single client application that makes frequent TCP (HTTP) connections to the server. Connection lasts for the duration of one request and then server closes the connection so socket on the server ends up in TIME_WAIT state for few minutes. As client application makes repeated requests the number of TIME_WAIT connections on server grows and settles on ~150 sockets being in TIME_WAIT constantly.

Occasionally new connection from client chooses a recently used ephemeral source port that is in TIME_WAIT state on the server and connection fails. With 32k ephemeral port range and 150 sockets in TIME_WAIT there's >10% chance of this happening. Exact scenario is also described in RFC6056 section 2.3

I have no control on the server so reducing TIME_WAIT delay or changing behavior is not an option. It is also difficult to change client app in order to pick source port manually as I am using a 3rd party library to make HTTP requests.

Increasing ephemeral port range would only slightly decrease probability of collision. Is there a way to change ephemeral port selection strategy on a client (Linux) to avoid such collisions?

Answer 1

A socket in TIME-WAIT will gladly accept a new connection from a device using the same 5 tuple (protocol, source IP, source port, destination IP, destination port) providing that the Initial Sequence Number (ISN) of the new connection is higher than the last sequence number seen on the previous connection. As per RFC 1122:

When a connection is closed actively, it MUST linger in TIME-WAIT state for a time 2xMSL (Maximum Segment Lifetime). However, it MAY accept a new SYN from the remote TCP to reopen the connection directly from TIME-WAIT state, if it:

(1) assigns its initial sequence number for the new connection to be larger than the largest sequence number it used on the previous connection incarnation, and

(2) returns to TIME-WAIT state if the SYN turns out to be an old duplicate.

You can test this on a Linux machine by setting your ephemeral port range to a single port echo 32769 32769 > /proc/sys/net/ipv4/ip_local_port_range and then making several consecutive requests to a website with the Connection: Close HTTP header set wget --no-http-keep-alive www.example.com. Even though the 5 tuple of all your connections is the same, the server will accept the new connections whilst in TIME-WAIT because the ISN of each new connection should be higher than the sequence number last seen on the socket. In Linux, the ISN of a new connection should be constantly increasing - it is somewhat tied to the system clock.

If you are confident that TIME-WAIT is always ending up on the server side and you are unable to connect when your ephemeral ports get reused, then the ISN used by your client must not be increasing for each new connection (perhaps it is being chosen randomly or always uses the same value?). If you can ensure that each new connection from your client uses a higher ISN value than the last, you should be able to connect without issue.