5

We are running haproxy in production for around 10k+ concurrent users. But we are seeing lot of FIN_WAIT2, CLOSE_WAIT , LAST_ACK and TIME_WAIT in the netstat output. This output is on a 8G ubuntu-12.04 node.

   8046 CLOSE_WAIT  
      1 CLOSING  
      1 established)  
  40869 ESTABLISHED  
   1212 FIN_WAIT1  
   7575 FIN_WAIT2  
      1 Foreign  
   2252 LAST_ACK  
      7 LISTEN  
    143 SYN_RECV  
   4920 TIME_WAIT

Can someone please tell me what tweaking I need to do?
Please note that all these connections are persistent connections.

tcp_fin_timeout = 30  
tcp_keepalive_time = 1800

Right now, the application is working fine. But wondering will be there any issues as we add more users to this haproxy node.

Hennes
  • 4,772
  • 1
  • 18
  • 29
Tux
  • 51
  • 1
  • 1
  • 2

1 Answers1

8

The connections you listed in the first paragraph are either ESTABLISHED or in process of being cleaned up after they have been used. Established means what the name implies. A connection is established between one of your users and the HAProxy. Usage as intended.

The other states you mention in the first paragraph all indicate that an previously established connection has finished transferring data. The best way I can explain that is with a diagram.

TCP/IP socket close diagram

Briefly: If you have a lot of FIN_WAIT 1 and FIN_WAIT 2's then there is nothing wrong with the server. You are simply waiting for the clients to finish.

Since it is not a problem with the server, adding more users should not be a problem until you hit kernel network limits. You did not post what those are, so I can not comment on how close you are to them.

Hennes
  • 4,772
  • 1
  • 18
  • 29
  • 2
    Thanks Hennes for the detailed reply. Thanks a bunch. Since currently the haproxy is working fine, would like to know what all tweakings required if we want to serve around 100k customer from this node. Please note all these connections will be persistent. – Tux Nov 20 '12 at 05:11
  • 2
    There is not way to answer that part with the current information. We do not know which hardware you are using nor which OS (Linux? FreeBSD? Solaris, OpenBSD? HA proxy supports all of them). Nor do we know the average usage of your users. If all they do is access webpages then the connections are build, used and torn down. If they all set up a truly persistent connection then haproxy will need to keep much more status data. – Hennes Nov 20 '12 at 14:02
  • Actually you can reach the number of FDs, default in linux is 1024. I used to have this problem when using socket.io and nodejs. –  Jul 29 '14 at 21:58
  • Related: http://serverfault.com/a/478707/87017 – Pacerier Jan 23 '16 at 03:48