| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Buffer Pool Problem? |
« View previous topic :: View next topic » |
| Author |
Message
|
| PeterPotkay |
 Posted: Fri May 14, 2010 5:06 am Post subject: Buffer Pool Problem? |
 |
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
We have an IMS application reporting slowness. It uses MQ.
For this one Buffer Pool:
QPSTBPHR (Buffer Pool Hit Ratio) is frequently dropping from 100% to 0. Before 1 PM yesterday its always averaged in the high 90s% or 100%.
QPSTSTLA is going over 0 frequently. Before 1 PM it never went > 0. We have never gotten alerted on this in 5 years of monitoring it, but yesterday we started to, and continue to.
Page Set 2 (mapped to Buffer Pool #2) shows as being half full of persistent pages, yet the only q in Page 2 with messages has 103 tiny messages (66 bytes), that have been there for a couple of years.
Accessing this one queue to browse it, with any tool, takes forever. Accessing other queues on Page Set 2 performs normally.
Available buffers for this pool shows > 49,850 out of 50,000.
There are no uncommitted messages on this queue according to MO71.
The ****MSTR is clean.
Other Buffer Pools and Page Sets for this QM are normal.
Nothing changed from an MQ configuration perspective.
MQ 7.0.0 (since last year).
PMR has been opened, but any advice what else we can check in the meantime? Its very odd that accessing this one queue is so slow, when others on the same Page Set are fine. And its odd that the Page Set shows half full, when there are only 103 messages 66 bytes in length each.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
 |
| PeterPotkay |
| Posted: Fri May 14, 2010 11:56 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Well, its working now. Baffling. Here is the write up I gave the Incident Manager.
| Quote: |
We had an IMS application reporting slowness. It uses MQ.
We noticed that one queue on the MQXX Queue Manager was performing very slowly. It would take tens of seconds to look at the messages on the queue, when normailly that should take < 1 second. This queue is called QueueNameHere.
The messages are tiny, only 66 bytes in length. There are 103 of them on the queue, some dating back to 2006. Otherwise new messages go in and out of the queue at about 250 per hour between 9 AM and 5 PM.
The queue has not been altered since 2001.
The queue maps to Buffer Pool 2 which maps to Page Set 2.
Buffer pools are areas of WebSphere MQ virtual storage reserved to satisfy the buffering requirements for WebSphere MQ queues. Each buffer pool contains an installation defined number of 4 KB virtual storage pages or buffers. Page sets are VSAM linear data sets and each page set is associated with a buffer pool.
For this Buffer Pool #2:
QPSTBPHR (Buffer Pool Hit Ratio) was frequently dropping from 100% to 0. Before 1 PM yesterday its always averaged in the high 90s% or 100%.
QPSTSTLA (Hash Chain Changed during Buffer steal) was going over 0 frequently. Before 1 PM it never went > 0. We have never gotten alerted on this in 5 years of monitoring it, but yesterday we started to, and continue to during the IMS online day.
Page Set 2 (mapped to Buffer Pool #2) shows as being half full of persistent pages, yet the only queue in Page 2 with messages is QueueNameHere with the 103 tiny messages (66 bytes).
Buffer Pool 2 has 50,000 buffers in it, and available buffers was consistently at 49,850 or higher.
Accessing this one queue to browse it, with any tool, takes forever. Accessing other queues on Page Set 2 performs normally.
There are no uncommitted messages on this queue.
The Queue Manager log (MQXXMSTR) is clean - no errors.
Other Buffer Pools and Page Sets for this QM are normal.
Nothing changed from an MQ configuration perspective.
At the 9 AM conf call we decided to clear the queue of these old messages. The clear q command failed as expected, because the queue is in use during the day. I then tried to delete a block of 15-20 of these old messages. That succeeded, but took a long time. I then tried to delete another 15 or so, and that executed very quickly.
At this point access to the queue was normal. App area confirmed that IMS queueing dropped to zero. QPSTBPHR went back up to 100% (good). QPSTSTLA dropped to zero (good). The Page Set utilization dropped to nearly zero, correctly reflecting that there were hardly any messages sitting in the queues.
It has been stable ever since.
Its odd that the Page Set showed half full, when there are only 103 messages of 66 bytes in length each. Queues can hold millions of messages that are very big for a very long time. We do not understand why this one queue with 103 little messages that have been there for years all of the sudden became "slow" at 1 PM on 05-13-2010. Its also puzzling why deleting one or more of them helped.
PMR has been opened with IBM. Perhaps they can help us determine root cause or know of similiar scenarios on other accounts. |
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
