Author |
Message
|
zpat |
Posted: Wed May 06, 2009 4:51 am Post subject: How to move multiple input MQ messages to a single file? |
|
|
 Jedi Council
Joined: 19 May 2001 Posts: 5866 Location: UK
|
OK. I want to process messages from a queue and write them as records to a file. The file should be closed when the queue is empty or when 500 records are written.
I assume that the flow starts with MQInput for the first message and I send the record to a FileOutput node, then loop around to a MQGet node for the next message (same queue) until MQRC 2033 or reaching 500 records when I close the file (end of data).
Is this a correct design and can it all operate under a unit of work?
WMB 6.1, AIX. |
|
Back to top |
|
 |
mqjeff |
Posted: Wed May 06, 2009 4:55 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 17447
|
It's a correct design if you really really really really need to maintain a single unit of work here.
The most straightforward way to do this, without enforcing a single unit of work, is to wire MQInput->compute->FileOutput, and use the compute node to decide when to close the file.
Or you could use Collector. |
|
Back to top |
|
 |
zpat |
Posted: Wed May 06, 2009 5:26 am Post subject: |
|
|
 Jedi Council
Joined: 19 May 2001 Posts: 5866 Location: UK
|
I wasn't sure if multiple, separate invocations of the message flow (for each message) would allow deferring the file close.
Does the compute node have to maintain the message/record count in a shared variable for your design to work?
How would the queue empty mqrc 2033 cause the file to be closed in your design?
The collector node is interesting. |
|
Back to top |
|
 |
mqjeff |
Posted: Wed May 06, 2009 5:33 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 17447
|
zpat wrote: |
I wasn't sure if multiple, separate invocations of the message flow (for each message) would allow deferring the file close. |
Yes, there's an input terminal to say that the file should be closed.
zpat wrote: |
Does the compute node have to maintain the message/record count in a shared variable for your design to work? |
Yep. Unless you can tell from the message data. But that's very lightweight, and you'd need to do something like it with your design anyway, to know when to close the file in the first place.
zpat wrote: |
How would the queue empty mqrc 2033 cause the file to be closed in your design? |
Right, that's a bit of a sticky wicket.... Could do an MQGet with Browse to detect that. Again, once for each flow through, and stick the results somewhere they get discarded.
zpat wrote: |
The collector node is interesting. |
yah, again the question of how to detect the 2033 to close the collection. And it's a bit more heavyweight. |
|
Back to top |
|
 |
Vitor |
Posted: Wed May 06, 2009 5:41 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
zpat wrote: |
How would the queue empty mqrc 2033 cause the file to be closed in your design? |
Wire the "No Message" terminal of the MQGet to the End Of Data on the FileOutput? With an intermediate Compute to reset record count? _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
zpat |
Posted: Wed May 06, 2009 5:47 am Post subject: |
|
|
 Jedi Council
Joined: 19 May 2001 Posts: 5866 Location: UK
|
Thanks (both).
The collector expiry option might also work, I don't need the file closed the instant the queue is empty, but I also don't want the pending file waiting around a long time for more messages to arrive.
Presumably until the file is closed, it cannot be accessed accidentally from say an FTP script, because it's in the mqsitransit directory? |
|
Back to top |
|
 |
Vitor |
Posted: Wed May 06, 2009 5:49 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
zpat wrote: |
Presumably until the file is closed, it cannot be accessed accidentally from say an FTP script, because it's in the mqsitransit directory? |
Provided the FTP script is polling / pulling from where it should be pulling from. _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
mqjeff |
Posted: Wed May 06, 2009 6:33 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 17447
|
zpat wrote: |
The collector expiry option might also work, I don't need the file closed the instant the queue is empty, but I also don't want the pending file waiting around a long time for more messages to arrive. |
You could use a simple Timeout node without using Collector, and have it reset in the compute every time a message passes through.
zpat wrote: |
Presumably until the file is closed, it cannot be accessed accidentally from say an FTP script, because it's in the mqsitransit directory? |
As my esteemed colleague says, this is entirely dependant on the FTP script being well behaved. |
|
Back to top |
|
 |
|