We have an application running on WebSphere (unfortunately it is 6.1 which is no longer supported, it has not yet been migrated in production to a later version) which becomes entirely unresponsive because of hung threads.
As far as I can tell we entirely exhaust one of the thread pools.
I have activated hung thread detection and I get a core/thread dump when hung threads are detected.
The server can run for several days without problems but has crashed twice this week.
When load the core/thread dump in "IBM Thread and Monitor Dump Analyzer for Java", it tells me that there are a certain number of hung threads (this time it was 2, last time 11) and multiple (usually around 40) threads "waiting on condition" and some running threads. I believe one of the thread pool has around that size (50).
Now what I see in there are threads waiting for locks, having locks or in wait.
Most of them show a stack track which always ends like this:
at java/lang/Object.wait(Native Method) at java/lang/Object.wait(Object.java:231)
Now, how can I track this down to either a server configuration problem, application issue, WebSphere problem or something else? How is this supposed to help me track down the problem when almost everything in there refers to IBM code?
I cannot ask IBM's help as 6.1 is now an unsupported version of WebSphere and while work has been done to make it work under WebSphere 7 we are not yet ready to switch to it in Production yet.
These are the stack trace of the threads which are blocked:
HAManager.thread.pool : 0
Blocked
Waiting for Monitor Lock on java/lang/Object@12648F78/12648F84
at java/lang/Object.wait(Native Method)
at java/lang/Object.wait(Object.java:231)
at com/ibm/ws/util/BoundedBuffer.waitGet_(BoundedBuffer.java:203)
at com/ibm/ws/util/BoundedBuffer.take(BoundedBuffer.java:564)
at com/ibm/ws/util/ThreadPool.getTask(ThreadPool.java:869)
at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1617)
WebContainer : 5
Blocked
Waiting for Monitor Lock on com/ibm/ejs/ras/TraceLogger@11265878/11265884
at com/ibm/ejs/ras/TraceLogger.doLog(TraceLogger.java:299)
at com/ibm/ejs/ras/TraceLogger.processEvent(TraceLogger.java:286)
at com/ibm/ws/logging/WsHandlerWrapper.publish(WsHandlerWrapper.java:43)
at java/util/logging/Logger.log(Logger.java:526)
at com/ibm/ejs/ras/Tr.logToJSR47Logger(Tr.java:1625)
at com/ibm/ejs/ras/Tr.fireEvent(Tr.java:1587)
at com/ibm/ejs/ras/Tr.fireTraceEvent(Tr.java:1509)
at com/ibm/ejs/ras/Tr.debug(Tr.java:610)
at com/ibm/ws/rsadapter/spi/WSRdbManagedConnectionImpl.inGlobalTransaction(WSRdbManagedConnectionImpl.java:1045)
at com/ibm/ws/rsadapter/jdbc/WSJdbcConnection.commit(WSJdbcConnection.java:961)
...
The threads "waiting on condition" are a mix of:
at java/lang/Thread.sleep(Native Method)
at java/lang/Thread.sleep(Thread.java:941)
....
and lots of
at java/lang/Object.wait(Native Method)
at java/lang/Object.wait(Object.java:231)
...
I have a pretty peculiar one, why is it waiting on this?:
at java/util/Calendar.setFieldsComputed(Calendar.java:1402)
at java/util/Calendar.complete(Calendar.java:1349)
at java/util/Calendar.get(Calendar.java:1120)
at java/text/SimpleDateFormat.subFormat(SimpleDateFormat.java:930)
at java/text/SimpleDateFormat.format(SimpleDateFormat.java:837)
at java/text/SimpleDateFormat.format(SimpleDateFormat.java:809)
at com/ibm/ws/logging/TraceLogFormatter.formatTimeAndThread(TraceLogFormatter.java:125)
at com/ibm/ws/logging/TraceLogFormatter.formatHeaderBasic(TraceLogFormatter.java:223)
at com/ibm/ws/logging/TraceLogFormatter.formatBasic(TraceLogFormatter.java:439)
at com/ibm/ejs/ras/TraceLogger.doLog(TraceLogger.java:295)
at com/ibm/ejs/ras/TraceLogger.processEvent(TraceLogger.java:286)
at com/ibm/ws/logging/WsHandlerWrapper.publish(WsHandlerWrapper.java:43)
...
I also have this:
WebContainer : 6
Owns Monitor Lock on com/ibm/ejs/ras/TraceLogger@11265878/11265884
at sun/io/CharToByteSingleByte.convert(CharToByteSingleByte.java:329)
at com/ibm/misc/IOConverter.convert(IOConverter.java:111)
at sun/io/CharToByteConverter.convertAny(CharToByteConverter.java:191)
at sun/nio/cs/StreamEncoder$ConverterSE.implWrite(StreamEncoder.java:259)
at sun/nio/cs/StreamEncoder.write(StreamEncoder.java:186)
at java/io/OutputStreamWriter.write(OutputStreamWriter.java:214)
at java/io/BufferedWriter.flushBuffer(BufferedWriter.java:132)
at java/io/PrintStream.write(PrintStream.java:492)
at java/io/PrintStream.print(PrintStream.java:636)
at java/io/PrintStream.println(PrintStream.java:773)
at com/ibm/ejs/ras/TraceLogger.doLog(TraceLogger.java:300)
at com/ibm/ejs/ras/TraceLogger.processEvent(TraceLogger.java:286)
at com/ibm/ws/logging/WsHandlerWrapper.publish(WsHandlerWrapper.java:43)
at java/util/logging/Logger.log(Logger.java:526)
at com/ibm/ejs/ras/Tr.logToJSR47Logger(Tr.java:1625)
at com/ibm/ejs/ras/Tr.fireEvent(Tr.java:1587)
at com/ibm/ejs/ras/Tr.fireTraceEvent(Tr.java:1509)
at com/ibm/ejs/ras/Tr.entry(Tr.java:776)
at com/ibm/ws/rsadapter/jdbc/WSJdbcConnection.prepareStatement(WSJdbcConnection.java:2056)
at com/ibm/ws/rsadapter/jdbc/WSJdbcConnection.prepareStatement(WSJdbcConnection.java:2039)
...
and finally this:
WebContainer : 8
at java/lang/StringBuffer.append(StringBuffer.java:239)
at com/ibm/ws/logging/TraceLogFormatter.formatTimeAndThread(TraceLogFormatter.java:137)
at com/ibm/ws/logging/TraceLogFormatter.formatHeaderBasic(TraceLogFormatter.java:223)
at com/ibm/ws/logging/TraceLogFormatter.formatBasic(TraceLogFormatter.java:439)
at com/ibm/ejs/ras/TraceLogger.doLog(TraceLogger.java:295)
at com/ibm/ejs/ras/TraceLogger.processEvent(TraceLogger.java:286)
at com/ibm/ws/logging/WsHandlerWrapper.publish(WsHandlerWrapper.java:43)
at java/util/logging/Logger.log(Logger.java:526)
at com/ibm/ejs/ras/Tr.logToJSR47Logger(Tr.java:1625)
at com/ibm/ejs/ras/Tr.fireEvent(Tr.java:1587)
at com/ibm/ejs/ras/Tr.fireTraceEvent(Tr.java:1509)
at com/ibm/ejs/ras/Tr.exit(Tr.java:796)
at com/ibm/ejs/j2c/PoolManager.processPoolRequestStats(PoolManager.java:6879)
at com/ibm/ejs/j2c/PoolManager.reserve(PoolManager.java:2026)
at com/ibm/ejs/j2c/ConnectionManager.allocateMCWrapper(ConnectionManager.java:940)
at com/ibm/ejs/j2c/ConnectionManager.allocateConnection(ConnectionManager.java:614)
at com/ibm/ws/rsadapter/jdbc/WSJdbcDataSource.getConnection(WSJdbcDataSource.java:449)
at com/ibm/ws/rsadapter/jdbc/WSJdbcDataSource.getConnection(WSJdbcDataSource.java:418)
...
