0

In WebSphere 8.5.5.13 I have some out of memory errors and the database connections get maxed out. It seems to me this is due to thread starvation (I have some processes trying to do something with a timeout of 10 seconds and other tasks that usually take ~200ms that actually take ~10200ms). But I think the last one could even be a deadlock. I have a ~100 threads waiting like this

3XMTHREADINFO      "WorkManager.DefaultWorkManager : 648" J9VMThread:0x000000000F2AA300, omrthread_t:0x00007FE38D060D78, java/lang/Thread:0x000000018ACD99E8, state:B, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x68C86, isDaemon:true)
3XMTHREADINFO1            (native thread ID:0xF8DE, native priority:0x5, native policy:UNKNOWN, vmstate:B, vm thread flags:0x00000201)
3XMTHREADINFO2            (native stack address range from:0x00007FE09C92F000, to:0x00007FE09C96F000, size:0x40000)
3XMCPUTIME               CPU usage total: 2.131995383 secs, current category="Application"
3XMTHREADBLOCK     Blocked on: com/ibm/ws/util/ThreadPool@0x000000011CD4B888 Owned by: "WorkManager.DefaultWorkManager : 689" (J9VMThread:0x00000000011B3000, java/lang/Thread:0x00000001B148B9A8)
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at com/ibm/ws/util/ThreadPool.getTask(ThreadPool.java:1083(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1916(Compiled Code))

and the stack of WorkManager.DefaultWorkManager : 689 looks like this

3XMTHREADINFO      "WorkManager.DefaultWorkManager : 689" J9VMThread:0x00000000011B3000, omrthread_t:0x00007FE1A41A70D0, java/lang/Thread:0x00000001B148B9A8, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x68CCD, isDaemon:true)
3XMTHREADINFO1            (native thread ID:0x11410, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00001001)
3XMTHREADINFO2            (native stack address range from:0x00007FE1EFF3E000, to:0x00007FE1EFF7E000, size:0x40000)
3XMCPUTIME               CPU usage total: 1.663139688 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/ThreadLocal$ThreadLocalMap.set(ThreadLocal.java:502(Compiled Code))
4XESTACKTRACE                at java/lang/ThreadLocal$ThreadLocalMap.access$100(ThreadLocal.java:311(Compiled Code))
4XESTACKTRACE                at java/lang/ThreadLocal.setInitialValue(ThreadLocal.java:197(Compiled Code))
4XESTACKTRACE                at java/lang/ThreadLocal.get(ThreadLocal.java:183(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/util/objectpool/TwoTierObjectPool.purgeThreadLocal(TwoTierObjectPool.java:264(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/buffermgmt/impl/WsByteBufferPool.purgeThreadLocal(WsByteBufferPool.java:173(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/buffermgmt/impl/WsByteBufferPoolManagerImpl.purgeThreadLocals(WsByteBufferPoolManagerImpl.java:1169(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/runtime/component/WSBBPoolListener.threadDestroyed(WSBBPoolListener.java:62(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/runtime/component/ThreadPoolMgrImpl.threadDestroyed(ThreadPoolMgrImpl.java:459(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/util/ThreadPool.fireThreadDestroyed(ThreadPool.java:1593(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/util/ThreadPool.workerDone(ThreadPool.java:1005(Compiled Code))
5XESTACKTRACE                   (entered lock: com/ibm/ws/util/ThreadPool@0x000000011CD4B888, entry count: 1)
4XESTACKTRACE                at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1929(Compiled Code))

As a reference, a thread that is idle (and not waiting for something to be freed) would look like this

  at sun/misc/Unsafe.park(Native Method)
  at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:222)
  at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2127)
  at com/ibm/ws/util/BoundedBuffer$GetQueueLock.await(BoundedBuffer.java:285)
  at com/ibm/ws/util/BoundedBuffer.waitGet_(BoundedBuffer.java:424)
  at com/ibm/ws/util/BoundedBuffer.take(BoundedBuffer.java:817)
  at com/ibm/ws/util/ThreadPool.getTask(ThreadPool.java:934)
  at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1704)

or

  at java/lang/Object.wait(Native Method)
  at java/lang/Object.wait(Object.java:231)
  at com/ibm/ws/util/BoundedBuffer.waitGet_(BoundedBuffer.java:192)
  at com/ibm/ws/util/BoundedBuffer.take(BoundedBuffer.java:543)
  at com/ibm/ws/util/ThreadPool.getTask(ThreadPool.java:819)
  at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1544)

and none of mine looks like those.

Thanks!

Frabac
  • 3
  • 1
  • Were you able to resolve the issue I am having similar issues on my WebSphere 8.5.5.13 system. – Travis Jul 28 '21 at 17:52

1 Answers1

0

A general example of a deadlock is like

  • thread 1 holds resource A and needs resource B to continue
  • thread 2 holds resource B and needs resource A to continue

In this scenario, neither thread can make progress, hence a deadlock exists.

The snippets you have posted do not match that pattern, so I do not think this is a deadlock.

With the caveat that I am not familiar with the particular code shown in the posted snippets, it looks to me like the first thread shown is just waiting to get a task from the WorkManager queue, which is probably empty.

Also BTW, in an IBM Java thread dump (which is what your snippets appear to be from), deadlocked threads are detected during the process of creating the dump, and are flagged with a DEADLOCK marker. So you can search for that in your java thread dump, to save you the time required to match up all the possible thread/resource combinations to find the deadlock manually.

Hope this helps.

Gary DeVal
  • 16