| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Messages accumulating SYSTEM.BROKER.AGGR.* QUEUES |
« View previous topic :: View next topic » |
| Author |
Message
|
| JRome |
 Posted: Thu Oct 20, 2011 11:09 pm Post subject: Messages accumulating SYSTEM.BROKER.AGGR.* QUEUES |
 |
|
Novice
Joined: 01 Jul 2011
Posts: 15
|
In Broker Queue manager A
SYSTEM.BROKER.AGGR.UNKNOWN, 1 message, 0.0% full
In Broker Queue manager B
SYSTEM.BROKER.AGGR.REPLY, 30409 messages, 30.4% full
SYSTEM.BROKER.AGGR.REQUEST, 30461 messages, 30.4% full
SYSTEM.BROKER.AGGR.TIMEOUT, 60836 messages, 60.8% full
SYSTEM.BROKER.AGGR.UNKNOWN, 30427 messages, 30.4 full
In Broker Queue manager C
SYSTEM.BROKER.AGGR.REPLY, 30653 messages, 30.6% full
SYSTEM.BROKER.AGGR.REQUEST, 30653 messages, 30.6% full
SYSTEM.BROKER.AGGR.TIMEOUT, 61343 messages, 61.3% full
SYSTEM.BROKER.AGGR.UNKNOWN, 30692 messages, 30.6 full
Afew question on the above observations:
1) Why are messages accumulating on these queues?
2) Can having some triggered alert on this queue depth of any of the above queues be a way to indicate early signals of Broker health issues?
3) How could I find out any offending or malformed messages in the queue?. |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Fri Oct 21, 2011 6:39 am Post subject: Re: Messages accumulating SYSTEM.BROKER.AGGR.* QUEUES |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| JRome wrote: |
In Broker Queue manager A
SYSTEM.BROKER.AGGR.UNKNOWN, 1 message, 0.0% full
In Broker Queue manager B
SYSTEM.BROKER.AGGR.REPLY, 30409 messages, 30.4% full
SYSTEM.BROKER.AGGR.REQUEST, 30461 messages, 30.4% full
SYSTEM.BROKER.AGGR.TIMEOUT, 60836 messages, 60.8% full
SYSTEM.BROKER.AGGR.UNKNOWN, 30427 messages, 30.4 full
In Broker Queue manager C
SYSTEM.BROKER.AGGR.REPLY, 30653 messages, 30.6% full
SYSTEM.BROKER.AGGR.REQUEST, 30653 messages, 30.6% full
SYSTEM.BROKER.AGGR.TIMEOUT, 61343 messages, 61.3% full
SYSTEM.BROKER.AGGR.UNKNOWN, 30692 messages, 30.6 full
Afew question on the above observations:
1) Why are messages accumulating on these queues?
2) Can having some triggered alert on this queue depth of any of the above queues be a way to indicate early signals of Broker health issues?
3) How could I find out any offending or malformed messages in the queue?. |
The base assumption here is that fan out and corresponding fan in flows are always in a single execution group.
The second (most important) assumption here is that the aggregation timeout on any of the aggregations does not exceed 5 mins.
The growth of those queues indicates a sure way that the broker is not able to keep up with the load... and that any msg older than 5 mins (max timeout across) is forfeit. This can happen in case of extreme peak loads or after an incident (not necessarily on the broker but may be in the feeding system) that caused a backup.
What to do? - Make sure io and cpu for the broker are at acceptable levels
- find out which service/provider feeding a fan in is down and why. Getting it restarted and performing.
- find out if a fan in flow is not running or in a loop...
- I remember there having been a discussion about being able to specify your own aggregation system queues...
It might be of interest to separate your aggregation so that if there is a problem in one it won't affect the others. Check out if that is possible (guess you'd have to be at least at V7.0.0.2 if the option is available)
- wait for the broker to catch up => spare capacity is needed for that
- Manually clear the queues of anything older than the max timeout across all aggregations (mqget)
- clear periodically (daily) aggregation timeout and unknown queues (you should not see more than the odd message there on a daily basis)
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| JRome |
| Posted: Tue Oct 25, 2011 2:07 am Post subject: Backup of timeout messages |
|
|
Novice
Joined: 01 Jul 2011
Posts: 15
|
Thanks for the answers much appretiated.
Just One more question, if one of the service providers to the composite/aggregated message is a CICS transaction, if there was case that Mainframe had to be IPLed for some other reasons, this could cause time-out messages be backed up multiple times in the SYSTEM.BROKER.AGGR.TIMEOUT queue ?, is there a way of making these timend out messages expire and no longer be in the SYSTEM.BROKER.AGGR.TIMEOUT queue, probably this could only be possible through the application.
Thanks again |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Oct 25, 2011 6:16 am Post subject: Re: Backup of timeout messages |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| JRome wrote: |
Thanks for the answers much appretiated.
Just One more question, if one of the service providers to the composite/aggregated message is a CICS transaction, if there was case that Mainframe had to be IPLed for some other reasons, this could cause time-out messages be backed up multiple times in the SYSTEM.BROKER.AGGR.TIMEOUT queue ?, is there a way of making these timend out messages expire and no longer be in the SYSTEM.BROKER.AGGR.TIMEOUT queue, probably this could only be possible through the application.
Thanks again |
If you don't want your messages to expire, you will have to clean the queue manually at some predetermined interval. Best is to use a destructive get.
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
