tl;dr: How do i get the kernel to drop TIME_WAIT/closing sockets as forcibly and fast as possible? I don't care about data being lost since I'm not sending any.
I'm currently running a little test scenario on a very isolated machine.
I'm running two simple C programs against each other on the same host, where the purpose of the application is to connect a client socket to a server socket and drop it as fast as possible.
The application is fast enough to deadlock the socket (or rather fd) pool causing "Can't bind to address" on the client side, and the reason for it appears to be TIME_WAIT on every available socket slot. I verified this with ss -s.
To solve this, I remember setting three values under sysctl. And this used to be enough to solve the problem, but for some reason setting the following:
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 1
Doesn't work anymore. This just drops the number of connections per second down from 20k+ to roughly 4 per minute.
I might remember the wrong settings in the sysctl net configurations above, but I used to get out roughly 20k+ connections per second out of this machine.
So why is this all of a sudden deadlocking me down to 4/sec?
Same machine, a i7 running against 127.0.0.1 with Arch Linux minimal setup.
My main source of information for trying the settings were: How to reduce number of sockets in TIME_WAIT?
Edit:
It appears as if tcp_fin_timeout has no effect on the system.
Judging by this picture, there's a couple of thousand connections still in timewait even tho the server (top left) and client (top right) is no longer performing any connections.
At the end of a 10 second run, I manage to put through 18 805 connections which isn't a hole lot for this machine (~36k TIME_WAIT sockets because the server and client resides on the same machine).
Why isn't tcp_fin_timeout dropping the connections after a second?
It's definitely the OS not honoring my custom tcp_fin_timeout.
