| Author |
Message
|
| sandeep9678 |
 Posted: Wed May 23, 2012 12:23 am Post subject: Timeout Control node and Batch Processing |
 |
|
Apprentice
Joined: 04 Aug 2008
Posts: 41
Location: India
|
we are trying to use Timeout Control & Timeout Notification nodes to process the Messages in Batch mode.
As Timeout Control node stores the messages before propagating it to Timeout Notification node, how is it's impact on Memory. If there are multiple of 10K messages present those need to be processed as a batch at a specific time, will Timeout Control node will be cause memory issue on Broker?
Also we would like to process all the messages when the batch starts. So we are setting below in request. But it is only processing single message at a time and next after every 1 hour which is Interval. What we should specify to process all the messages that were gathered by TimeControl Notification node. We do not know the count.... It can be any
<TimeoutRequest>
<Identifier>BatchProcess</Identifier>
<Action>SET</Action>
<StartDate>TODAY</StartDate>
<StartTime>13:00:00</StartTime>
<Count>-1</Count>
<Interval>3600</Interval>
</TimeoutRequest>
We looked at http://www.ibm.com/developerworks/websphere/library/techarticles/0603_schutz/0603_schutz.html, but seems something missing
_________________
Cheers,
Sandeep |
|
| Back to top |
|
 |
| Esa |
| Posted: Wed May 23, 2012 1:12 am Post subject: Re: Timeout Control node and Batch Processing |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
I have been working with a similar case. The messages are stored in SYSTEM.BROKER.TIMEOUT.QUEUE, so there is no impact in memory. But each message is stored and triggered separately.
TimeoutNotification node (in controlled mode) can be used for batch processing, but its best suited for delayed batch processing.
If you want to collect messages and process all of them in a batch every full hour, you should consider starting the consumer flow with cron instead. You can use a flow that starts with a TimeoutNotification node running in automatic mode, but there has been a lot of discussion about it and the consencus is that is it somewhat problematic and less trustworthy than cron.
In any case, if your aim is to process messages in a batch, you should not send the payloads along with timeoutControl messages, but put them in a separate queue, from where the messages can be consumed by the triggered flow. |
|
| Back to top |
|
|
| sandeep9678 |
| Posted: Wed May 23, 2012 1:29 am Post subject: Re: Timeout Control node and Batch Processing |
|
|
Apprentice
Joined: 04 Aug 2008
Posts: 41
Location: India
|
Thanks Esa for quick reply. I am doing same and have payload messages in separate queue. But how to process all the messages from the queue as event triggers.
1. We can specify the count value as the queue depth and then Timeout Notification node will trigger that many times and MQGet can retrieve messages. Once all messages has been retrieved. But then next time we will again need to reset the count for queue depth.
2. Have MQGet node in loop until we reach No Messages
which one is good or any other thoughts / suggestions?
_________________
Cheers,
Sandeep |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed May 23, 2012 1:56 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I would use a collector node.
I would then use the timeout every hour to signal the collector node that the collection is complete and start a new one. |
|
| Back to top |
|
|
| sandeep9678 |
| Posted: Wed May 23, 2012 2:06 am Post subject: |
|
|
Apprentice
Joined: 04 Aug 2008
Posts: 41
Location: India
|
| mqjeff wrote: |
I would use a collector node.
I would then use the timeout every hour to signal the collector node that the collection is complete and start a new one. |
Thanks jeff. I have question, where exactly these messages (payloads) will be stored by collector node or Timeout Control node.
In Timeout Control node, is SYSTEM.BROKER.TIMEOUT.QUEUE used to save payload as well?
_________________
Cheers,
Sandeep |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed May 23, 2012 2:12 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
everything is stored in queues, yes.
Should be mentioned in the info center somewhere.
Oh, and a slight edit to what I said before.
I'd use the timeout node to start a new collection and then complete the old one, not the other way around as I said before. |
|
| Back to top |
|
|
| Esa |
| Posted: Wed May 23, 2012 2:42 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| I think Collector node can be used for batching a limited number of relatively small messages. The problem with it is that the compound message propagated by it contains all the collected messages may need to allocate a considerable amount of memory... |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed May 23, 2012 3:09 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Esa wrote: |
| I think Collector node can be used for batching a limited number of relatively small messages. The problem with it is that the compound message propagated by it contains all the collected messages may need to allocate a considerable amount of memory... |
There's no indication in this thread that it is possible to process the batch in pieces.
Ergo, there's no indication that some solution for assembling all of the messages in a batch in memory in the same instance of a flow is not required.
Ergo, collector.
If it's possible to process each message in the batch individually, but they must all be done "at the same time", within reason, then yes. I agree. Don't use collector, use an MQGet node in a loop.
Or use the Admin API to start a message flow that uses an MQInput node - this one is harder to stop than the MQGet node...
In either option, there's a risk of processing messages that arrived after the batch was started. So some thought needs to go into it more, if there's a requirement to only process messages that arrived before the batch started. |
|
| Back to top |
|
|
|
|
