2

I run a number of tomcat instances and occasionally some just stop reponding to requests - timeout on every connection.

I'm using AJP with mod_proxy in Apache 2.2.

I get a timeout via Apache/AJP through Tomcat's AJP connector, but also via the direct HTTP connector on 8080.

I have /server-status configured within Apache and it shows 16 requests currently being processed with W, 4 idle requests and 200+ open slots with no connection. My AJP connector is configured as:

<Connector port="8009" address="localhost"
           maxThreads="250" minSpareThreads="5" maxSpareThreads="15"
           connectionTimeout="1000"
           packetSize="16384"
           maxHttpHeaderSize="16384"
           enableLookups="false" redirectPort="8443"
           emptySessionPath="true" URIEncoding="UTF-8" protocol="AJP/1.3"/>

so it should have plenty of threads to accept new connections.

Using top I see CPU and wait both under 1% and the java process has 80% memory. There is 60M free mem and 200M free swap.

I set up a special threads.jsp page using

SystemThreadList stl = new SystemThreadList();
Thread[] allThreads = stl.getAllThreads();

which gives useful information, but in this state - it doesn't load either.

In catalina.log I see:

Mar 07, 2014 11:53:09 AM org.apache.jk.common.ChannelSocket processConnection
WARNING: processCallbacks status 2

and occassional activity from other web requests, but not mine.

Is there a way from the command line, or using a profiler to get a list of threads and stack traces to find out what is getting stuck?

Nic Cottrell
  • 1,282
  • 16
  • 31

2 Answers2

2

Set up Tomcat to enable RMI with these options:

-Dcom.sun.management.jmxremote.port=12346 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false 
-Djava.rmi.server.hostname=myhost

(This is an example, not a statement of best practice for security...)

Use jconsole which is in the JDK and connect to your server.

ETL
  • 6,443
  • 1
  • 26
  • 47
1

If Tomcat is also unresponsive through the direct HTTP connector, then the issue is not with AJP but something is amiss inside Tomcat.

You can get a full thread dump to be written to catalina.out by sending a 

kill -quit <tomcat PID>
flup
  • 111
  • 4