| Author |
Message
|
| ctallchris |
 Posted: Mon Feb 22, 2010 8:13 am Post subject: MQ Queue depths in ESQL |
 |
|
Newbie
Joined: 22 Feb 2010
Posts: 4
|
Hi,
I'm looking for a way to evaluate the depth of a queue in an instance of message broker.
Specifically we receive a single message on an input queue which is turned into a few thousand output messages by the time it makes it through our broker to the system.broker.aggr.reply queue.
In certain circumstances this begins to back up to the extent where the aggregation begins to slow the whole broker down. I am looking to put a check on the input process to check that the queue depth is not above a threshold before continuing as it is better to have 500 messages backed up and filtering through than quater of a million on the aggregate reply.
Many thanks |
|
| Back to top |
|
 |
| shashivarungupta |
| Posted: Mon Feb 22, 2010 8:32 am Post subject: |
|
|
Grand Master
Joined: 24 Feb 2009
Posts: 1343
Location: Floating in space on a round rock.
|
"ESQL is a language derived from SQL Version 3, particularly suited to manipulating both database and message data."
As definition says and as what I remember is we can not do that specifically.
Most of the times I used to use esql to deal with message and its translation and/or transformation.
You probably can use the Java compute node or external java program to do that in your project.
_________________
*Life will beat you down, you need to decide to fight back or leave it. |
|
| Back to top |
|
|
| nathanw |
| Posted: Mon Feb 22, 2010 8:34 am Post subject: |
|
|
Knight
Joined: 14 Jul 2004
Posts: 550
|
if it is throughput you are after then why not set individual queues for the aggr flows and have multiple instances of the flows in multiple execution groups with individual queues for each flow, that way the responses will not all back up on 1 queue and if all teh flows read of the same initial input you will be spreading the work out over a wider area
_________________
Who is General Failure and why is he reading my hard drive?
Artificial Intelligence stands no chance against Natural Stupidity.
Only the User Trace Speaks The Truth  |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Feb 22, 2010 9:12 am Post subject: Re: MQ Queue depths in ESQL |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ctallchris wrote: |
| I am looking to put a check on the input process to check that the queue depth is not above a threshold before continuing as it is better to have 500 messages backed up and filtering through than quater of a million on the aggregate reply. |
How do you plan to throttle the input without impacting other broker processes?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| ctallchris |
| Posted: Mon Feb 22, 2010 9:27 am Post subject: Re: MQ Queue depths in ESQL |
|
|
Newbie
Joined: 22 Feb 2010
Posts: 4
|
| Vitor wrote: |
How do you plan to throttle the input without impacting other broker processes? |
ideally just place timer in a while loop
while queuedepth > 100,000
wait(60)
end while
to be honest for the throttling even stopping the input flow would be an acceptable solution as it would reduce recovery time once someone arrives to fix it.
It is currently a live system so any change would really need to be as small as possible so structural changes are pretty much unsuitable. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Feb 22, 2010 9:41 am Post subject: Re: MQ Queue depths in ESQL |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ctallchris wrote: |
ideally just place timer in a while loop
while queuedepth > 100,000
wait(60)
end while
to be honest for the throttling even stopping the input flow would be an acceptable solution as it would reduce recovery time once someone arrives to fix it.
|
Which is exactly what that code would do. The execution group thread will go into wait.
To avoid that, you'd need to split the flow into it's own group. Doing that, you'd be better advised to use some of the scaling suggestions elsewhere in this thread.
Also what do you mean by:
| ctallchris wrote: |
| it would reduce recovery time once someone arrives to fix it |
You have a lot of messages and your design currently causes a backlog. This is bad but your comment implies manual intervention to "fix it". What's to fix?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| ctallchris |
| Posted: Tue Feb 23, 2010 1:58 am Post subject: |
|
|
Newbie
Joined: 22 Feb 2010
Posts: 4
|
Ok the problem is that after messages are split should one of the paths fail (say 3rd party server fails) and one of the groups of messages is not received the successfuly processed messages will sit on the aggr.reply queue until their sibling turns up.
If a failure in one of these systems someone will resolve the problem within a few hours. In this time the number of messages in the reply queue can get to the level where processing the replys takes more performance from the system than the process which is processing the new messages onto the queue which is comparably a very lightweight process.
Once this happens to restore the system the input needs to be stopped until the backlog can clear and new messages fed in at intervals.
After it becomes obvious that the system has experienced a failure (ie reply queues above 50,000) it is better to allow messages to clear than to continue feeding new data onto the system and letting it choke
_________________
Oracle DB, IBM Websphere MQ, IBM Websphere Message Broker Specialist. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue Feb 23, 2010 2:10 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
If you have a halfway decent monitoring solution in your company, it should be able to monitor the output q depth. When the depth is > than your threshold, it can get disable the input queue. When the depth falls below your threshhold, it can get enable the input queue. If the output q stays above your threshhold for longer than x minutes, it can alert you.
Spend the pennies and make sure the storage supporting the output queue is adequate so you can make that queue and your threshold deep.
Moving this thread to the WMB Forum.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| ctallchris |
| Posted: Tue Feb 23, 2010 2:25 am Post subject: |
|
|
Newbie
Joined: 22 Feb 2010
Posts: 4
|
| PeterPotkay wrote: |
If you have a halfway decent monitoring solution in your company, it should be able to monitor the output q depth. When the depth is > than your threshold, it can get disable the input queue. When the depth falls below your threshhold, it can get enable the input queue. If the output q stays above your threshhold for longer than x minutes, it can alert you.
Spend the pennies and make sure the storage supporting the output queue is adequate so you can make that queue and your threshold deep.
Moving this thread to the WMB Forum. |
Thanks. probably the simplest solution
_________________
Oracle DB, IBM Websphere MQ, IBM Websphere Message Broker Specialist. |
|
| Back to top |
|
|
|
|
