| Author |
Message
|
| StephenDub |
 Posted: Wed Apr 11, 2012 8:45 pm Post subject: WMB 6.1 Message Flow not always reading Grouped messages |
 |
|
Newbie
Joined: 11 Apr 2012
Posts: 3
Location: Wellington, New Zealand
|
I have a WMB 6.1 message flow which reads "grouped" messages from a queue and processes them as a group. However, sometimes all the messages for a group sit on the queue without getting processed. New messages which form part of other groups are processed ok, but for some reason the original group stays on the queue. All messages have the same GroupId, have Sequences starting at 1 and incrementing as they go, and have a "Last Message In Group" flag on the last message in the group.
MQInput properties are : TransactionMode=Auto, OrderMode=Default, LogicalOrder=yes, AllMessagesAvailable=no, CommitByMessageGroup=no
Could someone please point out any glaring errors in the setup, or suggest new courses of action that I could take? |
|
| Back to top |
|
 |
| Esa |
| Posted: Wed Apr 11, 2012 11:14 pm Post subject: Re: WMB 6.1 Message Flow not always reading Grouped messages |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| StephenDub wrote: |
All messages have the same GroupId, have Sequences starting at 1 and incrementing as they go, and have a "Last Message In Group" flag on the last message in the group.
|
Am I reading you correctly, do you really mean that all your groups have the same GroupId?
| StephenDub wrote: |
I have a WMB 6.1 message flow which reads "grouped" messages from a queue and processes them as a group.
...
MQInput properties are : TransactionMode=Auto, OrderMode=Default, LogicalOrder=yes, AllMessagesAvailable=no, CommitByMessageGroup=no |
Your MQInput properties do not seem correct for processing messages as a group. So i assume the flow gets the first message of the group and then uses MQGet to fetch the rest of the group?
Have you checked the backout count of the first message of the group that is stuck in the queue? What kind of exception handling do you have? |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Apr 11, 2012 11:38 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
All Messages Available needs to be set to Yes, so that you won't start processing any group until it is complete on the queue.
At least that is the usual way to use group processing. Otherwise the flow would get suspended part way through one incomplete group even when there are other complete groups ready on the queue.
You don't need to use MQGET node - the flow will be driven for each message in the group and you can detect the last one in the group (if you want to) by looking at the MQMD flags.
Always worth reading the WMQ Application Programming Guide/Ref (or the WMQ infocenter) because WMB is just encapulsating the MQI calls. |
|
| Back to top |
|
|
| Esa |
| Posted: Wed Apr 11, 2012 11:54 pm Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| zpat wrote: |
| All Messages Available needs to be set to Yes, so that you won't start processing any group until it is complete on the queue. |
Exactly. I suspect that the flow has additional instances and the MQGet loop is written in such a way that it ends up in an endless loop if one of the messages is missing...
@StephenDub: does the problematic group get processed if you kill the execution group?
| zpat wrote: |
| You don't need to use MQGET node - the flow will be driven for each message in the group and you can detect the last one in the group (if you want to) by looking at the MQMD flags. |
Unless the OP is transforming a group of messages into one. His mention of 'reading "grouped" messages' makes me think he is using message grouping for segmentation. Instead of sending segmented messages and letting MQInput assemble them automatically  |
|
| Back to top |
|
|
| StephenDub |
| Posted: Sun Apr 15, 2012 12:24 pm Post subject: |
|
|
Newbie
Joined: 11 Apr 2012
Posts: 3
Location: Wellington, New Zealand
|
| Thanks for the assistance and speculation. The flow is meant to "merge" messages comign from z/OS, as z/OS doesn't allow normal segmentation, so it's doing 1 MQInput and a "loop" of MQGets. I'll change the settings initially, and experiment with creating a new flow which doesn't have the MQGet loop. |
|
| Back to top |
|
|
| Vitor |
| Posted: Sun Apr 15, 2012 2:58 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| StephenDub wrote: |
| so it's doing 1 MQInput and a "loop" of MQGets. |
There are a number of posts on the dangers of looping MQGet.
| StephenDub wrote: |
| experiment with creating a new flow which doesn't have the MQGet loop. |
Start with a Collector node & experiment from there.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Esa |
| Posted: Sun Apr 15, 2012 11:10 pm Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| Vitor wrote: |
There are a number of posts on the dangers of looping MQGet. |
There is also a correct way to loop with MQGet. It is essential to write the loop in ESQL (or java) and propagate to MQGet from within the loop. You can make the loop perform very well, if you understand how the propagation model of Message Broker works. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Apr 16, 2012 4:39 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Esa wrote: |
| You can make the loop perform very well, if you understand how the propagation model of Message Broker works. |
And you can bring down the execution group if you don't understand how the model works.
Hence my use of the word "danger". You can easily repalce the worn brake components of your car if you understand how the mechanism works. If you don't understand the mechanism, or worse still think you do when in fact you don't, there is some danger.
So I pointed the OP at the previous threads outlining these dangers so they may be avoided.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| StephenDub |
| Posted: Mon Apr 16, 2012 3:28 pm Post subject: |
|
|
Newbie
Joined: 11 Apr 2012
Posts: 3
Location: Wellington, New Zealand
|
I thought my days of being impressed by WMB 6.1 were long gone, but I was wrong.
The flow has been re-designed to use a Collector node, and to use the "OutputRoot.BLOB.BLOB[1]" method of concatenating the messages. And it's working like a dream.
Thanks to all who helped. |
|
| Back to top |
|
|
| kimbert |
| Posted: Tue Apr 17, 2012 12:51 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
So...the Collector node can be a good alternative to using an MQGet-in-a-loop?
I think that is worth highlighting, for the benefit of future readers of this thread. I've seen several users with hard-to-solve stack or heap exhaustion problems when they use MQGet in a loop. Even when they followed the good advice give by Esa. |
|
| Back to top |
|
|
| Esa |
| Posted: Tue Apr 17, 2012 1:46 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
Interesting. This is something I haven't thought about in this context before.
Even if you loop within a compute or java compute node and only propagate to the MQGet to fetch the next message, you will eventually get exhausted if you process thousands of messages. Not because of the call stack, but because of orphaned parsers!
So you will actually have to propagate to a passtrough node with DELETE DEFAULT after each message from MQGet.
I guess that will work even if the message from MQGet is in LocalEnvironment, won't it, Kimbert? |
|
| Back to top |
|
|
|
|
