| Author |
Message
|
| zpat |
 Posted: Thu Aug 25, 2011 7:31 am Post subject: Message group processing on mqinput node |
 |
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
WMQ 7, WMB 7, AIX
I have a group of messages on a MQ queue (using the MQMD.groupid field), it is marked complete when the last message in group flag is set.
Now, the MQinput node has the "All messages available" property which starts the flow when a complete MQ group is on the queue.
My question is if I want to read all the messages for a given group inside the flow - does the MQinput node repeatedly supply the same flow transaction with each message (like the fileinput node can do with records in a file).
Or is a new transaction started for each message in the group? Which would mean that I would have to use the MQGET node to get the remaining messages from the group inside the same transaction.
My requirement is to transform a group of MQ messages into a single file. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Thu Aug 25, 2011 7:36 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| I'd be tempted to string them through the MQInput node without using the group id, and then feed them to a Collector that does use GroupID. |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Aug 25, 2011 9:22 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| OK, does that flow process all the messages in the group as part of a single transaction, or as one transaction per message? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Aug 25, 2011 9:34 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| it would process each message in the group as an individual message going INTO collector, and then process them all as a single transaction coming OUT of collector. |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Aug 26, 2011 12:25 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Does that mean that all the messages in the group are accumulated into memory and then propagated out as one item from the collector node when the group is complete?
The reason I ask is that the total size of all the messages is rather large.
I really want to process each message separately but inside one flow execution.
Another related question, can you send records to a fileoutput node and then finish file in another flow? I assume it has to be finished in the same flow execution? |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Aug 26, 2011 1:21 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
The collector node stores them on a Queue.
Well two queues actually.
The first is where the parts of the collection is stored.
The second holds complete collections that are waiting to be processed.
SYSTEM.BROKER.EDA.COLLECTIONS
SYSTEM.BROKER.EDA.EVENTS
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Aug 26, 2011 9:44 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| OK, but when it is complete - what happens - is it all read into memory at once? |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Aug 26, 2011 9:59 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| zpat wrote: |
| is it all read into memory at once? |
The collection is presented as a single message tree. The usual discussion about handling large messages applies.
But if you want them all in a single file I don't see a way round it - you can't (to answer your question from earlier) issue a Finish to a FileOutput node from a different flow.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Aug 26, 2011 12:57 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
There is an alternative - by using MQinput to get the first message when the group is ready, then propagating a MQget node from a compute node - multiple times to get the other messages in the group, and for each message propagate to the fileoutput node to accumulate the file data, and then finish file after the last message.
I just wanted to make sure that the collector node couldn't be used without the entire file being in memory (and it's rather large). |
|
| Back to top |
|
|
| Vitor |
| Posted: Sat Aug 27, 2011 4:19 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| zpat wrote: |
| There is an alternative - by using MQinput to get the first message when the group is ready, then propagating a MQget node from a compute node - multiple times to get the other messages in the group, and for each message propagate to the fileoutput node to accumulate the file data, and then finish file after the last message. |
It's certainly an option. You'd need to keep in mind the many discussions we've had about MQGet inside a loop causing memory problems, especially if you're going to be looping as many times as it seems.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sat Aug 27, 2011 5:59 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| zpat wrote: |
| There is an alternative - by using MQinput to get the first message when the group is ready, then propagating a MQget node from a compute node - multiple times to get the other messages in the group, and for each message propagate to the fileoutput node to accumulate the file data, and then finish file after the last message. |
It's certainly an option. You'd need to keep in mind the many discussions we've had about MQGet inside a loop causing memory problems, especially if you're going to be looping as many times as it seems. |
 .
The first thing is to make sure you build the loop in the proper way...
The other thing to keep in mind is that, since you are trying to keep all of these messages in the same MQ transaction, that you then need to ensure that your Broker's qmgr logs have sufficient space for this transaction to occur - amongst all the rest of the transactions that are going on...
Secondly, I want to nit-pick at Vitor's answer about closing a file from a separate flow. It really depends on what you mean by "a separate flow"... You have to close the file using the finish file terminal on the same FileOutput node that has received all the input records... but that doesn't have to in any way have been invoked from the same input node that passed in all of the input records.
So you can, for example:
MQInput--(out)-------+->FileOutput
MQInput--(finishFile)-|
It's still the same .msgflow file, but it's an entirely separate instance of the message flow that can do or respond to entirely different and unrelated things. |
|
| Back to top |
|
|
| Vitor |
| Posted: Sun Aug 28, 2011 5:56 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| Secondly, I want to nit-pick at Vitor's answer about closing a file from a separate flow. It really depends on what you mean by "a separate flow"... You have to close the file using the finish file terminal on the same FileOutput node that has received all the input records... but that doesn't have to in any way have been invoked from the same input node that passed in all of the input records. |
It's a valid nit-pick. I stand by you can't close a file in Flow A from Flow B, but you could combine both functions into Flow A as my most worthy associate suggests.
FWIW faced with your scenario I'd use (and have used) a Collector node.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
