0

I'm newbie for supporting WAS(Websphere Application Server), currently I'm having issue with my WAS, my WAS was installed under AIX under 2 servers/nodes.

While investigating it, I found in our application log that there are some activity which is "Performing Cache Maintenance"::

2017-01-14 01:31:52,619: [Cache Maintenance] com.ibm.srm.util.db.ServerCache refreshed
2017-01-14 01:31:53,314: [Cache Maintenance] Memory: available=[6884mb] used=[9500mb] %used avail=[58%] max=[16384mb] %used max=[58%] total=[16384mb] free=[6884mb] used by doMaintenance=[-251,201,3
92bytes] Time=[22,818ms]
2017-01-14 01:51:53,325:  -------- Performing Cache Maintenance --------
2017-01-14 01:51:53,325: null    : QN=319 Select * from perform.cache_timestamps where row_class_name not like '%Cache' and row_class_name not like '%(SRM 6.0)'
2017-01-14 01:51:53,333: Returning 19 data records, QN=319,2 columns, Time: 8ms  conn/query time: 5ms
2017-01-14 01:51:53,333: [Cache Maintenance] Memory: available=[5492mb] used=[10892mb] %used avail=[66%] max=[16384mb] %used max=[66%] total=[16384mb] free=[5492mb] used by doMaintenance=[532kb] Time=[8ms]

After this activity triggered, I found that mpmstats value for 'bsy' are keep increasing until reach MaxClient maximum value which is '4000'::

[Sat Jan 14 01:38:58 2017] [notice] mpmstats: rdy 166 bsy 234 rd 0 wr 234 ka 0 log 0 dns 0 cls 0
[Sat Jan 14 01:38:58 2017] [notice] mpmstats: bsy: 234 in mod_was_ap22_http.c
[Sat Jan 14 01:48:58 2017] [notice] mpmstats: rdy 195 bsy 505 rd 0 wr 505 ka 0 log 0 dns 0 cls 0
[Sat Jan 14 01:48:58 2017] [notice] mpmstats: bsy: 505 in mod_was_ap22_http.c
[Sat Jan 14 01:58:58 2017] [notice] mpmstats: rdy 180 bsy 720 rd 0 wr 720 ka 0 log 0 dns 0 cls 0
[Sat Jan 14 01:58:58 2017] [notice] mpmstats: bsy: 720 in mod_was_ap22_http.c
[Sat Jan 14 02:08:59 2017] [notice] mpmstats: rdy 105 bsy 895 rd 1 wr 894 ka 0 log 0 dns 0 cls 0
[Sat Jan 14 02:08:59 2017] [notice] mpmstats: bsy: 894 in mod_was_ap22_http.c
[Sat Jan 14 02:18:59 2017] [notice] mpmstats: rdy 112 bsy 1088 rd 1 wr 1087 ka 0 log 0 dns 0 cls 0
[Sat Jan 14 02:18:59 2017] [notice] mpmstats: bsy: 1087 in mod_was_ap22_http.c
[Sat Jan 14 02:28:59 2017] [notice] mpmstats: rdy 158 bsy 1242 rd 1 wr 1241 ka 0 log 0 dns 0 cls 0

[..]

[Sat Jan 14 04:55:34 2017] [notice] mpmstats: rdy 0 bsy 4000 rd 0 wr 4000 ka 0 log 0 dns 0 cls 0
[Sat Jan 14 04:55:34 2017] [notice] mpmstats: bsy: 4000 in mod_was_ap22_http.c
[Sat Jan 14 04:57:04 2017] [notice] mpmstats: reached MaxClients (4000/4000)
[Sat Jan 14 04:57:04 2017] [notice] mpmstats: rdy 0 bsy 4000 rd 0 wr 4000 ka 0 log 0 dns 0 cls 0
[Sat Jan 14 04:57:04 2017] [notice] mpmstats: bsy: 4000 in mod_was_ap22_http.c
[Sat Jan 14 04:58:34 2017] [notice] mpmstats: reached MaxClients (4000/4000)
[Sat Jan 14 04:58:34 2017] [notice] mpmstats: rdy 0 bsy 4000 rd 0 wr 4000 ka 0 log 0 dns 0 cls 0
[Sat Jan 14 04:58:34 2017] [notice] mpmstats: bsy: 4000 in mod_was_ap22_http.c

It seem WAS are not processing the client request until it reached the Maximum value.

Questions is:

  1. Is there any log that I can check about why WAS are not processing the Client request until it reached to the max value?

  2. Does the "Cache Maintenance" activity hold WAS from processing the Client request? Because as mentioned from our developer this activity should not lead to this issue.

  3. What is the procedure that I can take to identify/resolve this issue?

473183469
  • 1,350
  • 1
  • 12
  • 23
MrAZ
  • 131
  • 5

1 Answers1

0

Step one would be to get a javacore during the time during and shortly after the cache maintenance is done. It will likely tell you what resource is misused that causes the application server to basically lockup. Then take it to the proper support team for the com.ibm.srm application.

covener
  • 1,665
  • 9
  • 15