ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker SupportCollector Node use case

Post new topicReply to topic
Collector Node use case View previous topic :: View next topic
Author Message
Partha.Baidya
PostPosted: Thu Dec 20, 2018 2:53 pm Post subject: Collector Node use case Reply with quote

Voyager

Joined: 05 Nov 2009
Posts: 96

We have a use case where we need to group message from 3 different queues based on trigger message from a 4th Input queue. The trigger message will have the correlation id to create the collections.
We thought of using Collector node to collect messages based on the correlation id.
But we found out that in order to flash out the collections we need to either give a timeout.
If we set time out we can not get all the messages put in those queue as messages can come any time, so some related messages may skip from the collection if timeout has happened.

We found out that either we have to set timeout or quantity of how many messages to be collected.
For out case trigger message has the the quantity of each input messages and it is dynamic.

I am wondering whether the above use case is valid for collector node or we need to go for MQ Get and fetch each messages separately.
Back to top
View user's profile Send private message
timber
PostPosted: Thu Dec 20, 2018 3:50 pm Post subject: Reply with quote

Guardian

Joined: 25 Aug 2015
Posts: 991

You are not thinking clearly about this. If a collection can take a long time to complete then a message flow containing an MQGet node would be a really bad solution. That message flow thread might be hanging around for minutes, hours or perhaps even days waiting for a collection to complete. If the integration server went down, how would it recover its state after the restart?
( which leads to one of the rules of message flow design...'If you think you need an MQGet node, think again' )

I don't think the Collector node can do what you need (collection size set by the first message in the collection). You should look for a solution that can process each message in the collection as it arrives. By putting some state information into a database you can often meet the requirements without actually dealing with a thing called a 'collection'. One advantage of this approach is that you avoid the single-threaded bottleneck that is inevitable whenever you use a Collector node.
Back to top
View user's profile Send private message
Partha.Baidya
PostPosted: Thu Dec 20, 2018 6:19 pm Post subject: Reply with quote

Voyager

Joined: 05 Nov 2009
Posts: 96

We are trying to use of the trigger message which will have correlation id to be used for collection and no of message to be picked up from each Input source.
But what we have found that the quantity field in Collector Configuration table can not be override dynamically.
We do not want keep the state information in Database as it may slow down the process.
As the entire processing is in real time, there is less chance that collection will take long time. Because of this we are thinking of using MQ Get Node and loop through each queue based on the no of messages (no of messages will be present in trigger message for each message type/queue type). I am not sure whether this looping will work or not. Because we will have 3 Queues to loop through until build a single collection and there will continuous flow of message and multiple collection to be built as well.
Back to top
View user's profile Send private message
gfrench
PostPosted: Fri Dec 21, 2018 2:33 am Post subject: Reply with quote

Acolyte

Joined: 10 Feb 2002
Posts: 68

You might be able to do something smart with MQ and treating messages in groups. Make sure the trigger messages arrives first in the input queue and you can start a group of messages for each proper data message with the correct correlation id. Once the last one comes in mark it as the last in group.

If these groups are put on a second intermediate queue these can be processed with the flag all messages available configured on another flow so it will only process with all messages are there and I guess you can use an MQGet or collection in that flow to join them. This wont leave a thread tied up for a long period.

That just leaves the task of processing the trigger message and maintaining the count for each correlation id and marking the last message as last message in group.

Its a very contact admin solution that should really be fixed architecturally from the top rather than fixing it at flow level.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Vitor
PostPosted: Fri Dec 21, 2018 5:31 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25685
Location: Ohio, USA

Partha.Baidya wrote:
Because of this we are thinking of using MQ Get Node and loop through each queue based on the no of messages (no of messages will be present in trigger message for each message type/queue type). I am not sure whether this looping will work or not. Because we will have 3 Queues to loop through until build a single collection and there will continuous flow of message and multiple collection to be built as well.


And it's a fairly well discussed point in this forum that an MQGet node in a loop is the best and fastest way to run an execution group out of memory. It may work, but it won't scale.

I agree with my associate - this is a fairly bad solution. If you don't want to store the intermediate messages in a database due to speed, there are other solutions. One which springs to mind (and I accept that there are others) would be to use the eXtreme scale supplied with IIB to store the messages as you read them off their respective queues with an MQInput node.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexWebSphere Message Broker SupportCollector Node use case
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.