9

I've got Apache setup as a reverse proxy for a Java Application server (GlassFish) and I noticed there are about 100 connections in the state CLOSE_WAIT even on an idle development system:

sudo netstat -n -e -p -a -t | grep httpd | grep CLOSE_WAIT | wc -l

I'm using the following HTTP proxy settings:

ProxyPass /myapp http://localhost:8080/myapp ttl=20 max=1 smax=0
ProxyPassReverse /myapp http://localhost:8080/myapp

Why are all of these connections hanging around? I've set the "ttl=20 max=1 smax=0" so I figured all connections would be cleaned up on an idle system. Is the application server not doing its part to cleanup the connections?

Ryan
  • 420
  • 5
  • 13
  • 1
    I have a similar issue. If you use `netstat -atn`, do you see 1 byte remaining in the read or write queue? – Cameron Kerr May 13 '14 at 12:20
  • Yes, the Recv-Q shows 1 for all of the CLOSE_WAIT connections. That suggests that the Local Address (Apache) has one byte it refuses to read from the queue? The Send-Q is 0 on all so it seems the Foreign Address (GlassFish 127.0.0.1:8080) has done its part. Weird. – Ryan May 13 '14 at 14:20
  • I have the same problem with a tomcat server. Did you eventually solve the issue? – circus May 28 '14 at 19:16
  • Performance really hasn't been an issue for me so I decided to turn off connection reuse to avoid this odd behavior (disableReuse=On). It may be okay to just live with the CLOSE_WAIT connections if you really need connection pooling. – Ryan Jun 03 '14 at 17:50

3 Answers3

4

This is a known issue with mod_proxy, since 2011.

The ttl needs to be shorter than the application's keepalive, so that apache is always first to send a FIN.

Another difficulty is it's not defined at what point the connection will actually be closed.

ttl - Time to live for inactive connections and associated connection pool entries, in seconds. Once reaching this limit, a connection will not be used again; it will be closed at some later time.

OrangeDog
  • 519
  • 4
  • 20
0

I encountered similar issue and I am looking for reasoning with respect to Apache. I suspect Apache's prefork.

As for the solution, I used Nginx that solved CLOSE_WAIT issue. However the number of TIME_WAIT (20,000 approx.) shows there is something not right with the way the Java application(s) handles connections. With Nginx server and application performs much better.

I hope some one can improve this answer with technical depth.

anup
  • 657
  • 4
  • 8
  • 19
0

These CLOSE_WAIT connections are dead, it is just the server keeping them in the tcp stack in case further packets get to them. In the "good old days", Solaris servers would run out of file descriptors if this was too big, and the system would crash. We had to increase the number of total file descriptors allowed in the kernel, and reduce the cleanup interval of CLOSE_WAIT connections.

Now a days, the default number of file descriptors is usually large enough to ignore this. But the cleanup configuration can be reduced, so the number of CLOSE_WAIT connections will be reduced.

It is by the very nature of these (Glassfish, Tomcat, JBoss, ...) to use a "large" number of connections and not reuse them. You can safely ignore, in most cases.

Nic3500
  • 155
  • 8