| Author |
Message
|
| Partha.Baidya |
 Posted: Thu Dec 20, 2018 2:53 pm Post subject: Collector Node use case |
 |
|
Voyager
Joined: 05 Nov 2009
Posts: 97
|
We have a use case where we need to group message from 3 different queues based on trigger message from a 4th Input queue. The trigger message will have the correlation id to create the collections.
We thought of using Collector node to collect messages based on the correlation id.
But we found out that in order to flash out the collections we need to either give a timeout.
If we set time out we can not get all the messages put in those queue as messages can come any time, so some related messages may skip from the collection if timeout has happened.
We found out that either we have to set timeout or quantity of how many messages to be collected.
For out case trigger message has the the quantity of each input messages and it is dynamic.
I am wondering whether the above use case is valid for collector node or we need to go for MQ Get and fetch each messages separately. |
|
| Back to top |
|
 |
| timber |
| Posted: Thu Dec 20, 2018 3:50 pm Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
You are not thinking clearly about this. If a collection can take a long time to complete then a message flow containing an MQGet node would be a really bad solution. That message flow thread might be hanging around for minutes, hours or perhaps even days waiting for a collection to complete. If the integration server went down, how would it recover its state after the restart?
( which leads to one of the rules of message flow design...'If you think you need an MQGet node, think again' )
I don't think the Collector node can do what you need (collection size set by the first message in the collection). You should look for a solution that can process each message in the collection as it arrives. By putting some state information into a database you can often meet the requirements without actually dealing with a thing called a 'collection'. One advantage of this approach is that you avoid the single-threaded bottleneck that is inevitable whenever you use a Collector node. |
|
| Back to top |
|
|
| Partha.Baidya |
| Posted: Thu Dec 20, 2018 6:19 pm Post subject: |
|
|
Voyager
Joined: 05 Nov 2009
Posts: 97
|
We are trying to use of the trigger message which will have correlation id to be used for collection and no of message to be picked up from each Input source.
But what we have found that the quantity field in Collector Configuration table can not be override dynamically.
We do not want keep the state information in Database as it may slow down the process.
As the entire processing is in real time, there is less chance that collection will take long time. Because of this we are thinking of using MQ Get Node and loop through each queue based on the no of messages (no of messages will be present in trigger message for each message type/queue type). I am not sure whether this looping will work or not. Because we will have 3 Queues to loop through until build a single collection and there will continuous flow of message and multiple collection to be built as well. |
|
| Back to top |
|
|
| gfrench |
| Posted: Fri Dec 21, 2018 2:33 am Post subject: |
|
|
Acolyte
Joined: 10 Feb 2002
Posts: 71
|
You might be able to do something smart with MQ and treating messages in groups. Make sure the trigger messages arrives first in the input queue and you can start a group of messages for each proper data message with the correct correlation id. Once the last one comes in mark it as the last in group.
If these groups are put on a second intermediate queue these can be processed with the flag all messages available configured on another flow so it will only process with all messages are there and I guess you can use an MQGet or collection in that flow to join them. This wont leave a thread tied up for a long period.
That just leaves the task of processing the trigger message and maintaining the count for each correlation id and marking the last message as last message in group.
Its a very contact admin solution that should really be fixed architecturally from the top rather than fixing it at flow level. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Dec 21, 2018 5:31 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Partha.Baidya wrote: |
| Because of this we are thinking of using MQ Get Node and loop through each queue based on the no of messages (no of messages will be present in trigger message for each message type/queue type). I am not sure whether this looping will work or not. Because we will have 3 Queues to loop through until build a single collection and there will continuous flow of message and multiple collection to be built as well. |
And it's a fairly well discussed point in this forum that an MQGet node in a loop is the best and fastest way to run an execution group out of memory. It may work, but it won't scale.
I agree with my associate - this is a fairly bad solution. If you don't want to store the intermediate messages in a database due to speed, there are other solutions. One which springs to mind (and I accept that there are others) would be to use the eXtreme scale supplied with IIB to store the messages as you read them off their respective queues with an MQInput node.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
