We have the following setup: One nginx server as a loadbalancer in front of four worker application servers. The nginx server get about 100 requests per second, sometimes more that that, active connection are hovering at 110 in average.
In a somewhat regular interval (1-2 seconds), the application that accesses the nginx loadbalancer gets a bunch of failed requests where the TCP connection cannot be established (connection timeout). The problem resolves quickly and the next bunch of connections are processed fine until the next wave of connection timeouts.
I figure that the is not a huge amount of load and the nginx should have no problems handling it. My first guess was to tune the ipv4 sysctl settings based on various help pages on the internet, but no avail so far.
I already increased the port range and lowered the fin_timeout to 30. There are a lot of TIME_WAIT connections shown in netstat (~11k).
Does anyone have an idea what to do in such cases? I am aware that there might be no definitve answer to what the actual problem is, but pointers are appreciated! Thanks!
