| Author |
Message
|
| psiracusa |
 Posted: Thu Apr 15, 2010 9:20 am Post subject: Variable # of Messages for Aggregation/Collection |
 |
|
Apprentice
Joined: 17 Nov 2006
Posts: 34
|
| I've been looking at the collector and aggregator nodes to try and solve a problem we are going to soon have, and that is how to aggregate X number of replies. X won't be known until runtime. Sometimes it will be 3, sometimes 10 etc. and it seems to me like the collection and aggregation suite won't combine the messages based on a dynamic number. Using the timout isn't an option either because the response needs to be sub-second. Anyone have a good solution for this? Any help is much appreciated. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu Apr 15, 2010 10:17 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
I'd have thought both would handle this because both know at runtime how many requests they've said. I'd certainly have thought the collector (which is less fiddly than aggrigation) could have handled that.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| psiracusa |
| Posted: Thu Apr 15, 2010 10:39 am Post subject: |
|
|
Apprentice
Joined: 17 Nov 2006
Posts: 34
|
| Let me rephrase that then and say X number of requests (and therefore X number of replies). In the collection definition, you have to define the quantity of messages before the node aggregates the data. In this case we don't know the quantity until runtime and the quantity field won't take a variable. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Apr 15, 2010 10:52 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Ok, I'll confess here and now I've not worked a lot with the Collector node so I might be talking out of something other than my mouth.
But surely an unknown number of reponses is still a complete collection and the node can be set to assemble the responses until the collection is complete? It might not be possible to have a collection that's made up of the next X responses, but presumably that's a feature not a problem if there are multiple request sets in play? You'd actually want the 2 sets kept separate by id?
Anybody want to comment on what I've got wrong, just jump right in... 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Apr 15, 2010 10:59 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Thinking about it, doesn't even the aggrigation node manage by setting the request up with a given name (folder name?) and the request msg id as correlation? So that would handle the unknown number of response problem as well surely?
I'm clearly missing something and/or have the wrong end of the stick. It happens, and I'm braced & ready to be embarrased when someone corrects me.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Apr 15, 2010 12:15 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Vitor wrote: |
Thinking about it, doesn't even the aggrigation node manage by setting the request up with a given name (folder name?) and the request msg id as correlation? So that would handle the unknown number of response problem as well surely?
I'm clearly missing something and/or have the wrong end of the stick. It happens, and I'm braced & ready to be embarrased when someone corrects me. |
I thought the aggregation node expected only a single response from each aggregation source. Don't know how that would play with a group...( probably all msgs in group with logical msg?)... but those settings would have to be put onto the MQInput node...
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| psiracusa |
| Posted: Thu Apr 15, 2010 12:25 pm Post subject: |
|
|
Apprentice
Joined: 17 Nov 2006
Posts: 34
|
Right. Same thing with the collector. One response per output. If I figure something out I'll let you know.  |
|
| Back to top |
|
|
| bdebruin |
| Posted: Thu Apr 15, 2010 12:48 pm Post subject: |
|
|
Novice
Joined: 14 Apr 2010
Posts: 12
|
Do you have the WTX node?
You can use WTX to concatenate vs the collector node. In the map your first input is wired to card 1 and output would have one wired card with two items/fields in a group. The first field passes the input from card 1 and the second item/field would do a GET to get the file from a directory or message from a queue.
We had a process that took several hours using the collector node, but with WTX it runs in a few minutes. It seems to be much less complicated, but I'm more of a WTX guy. |
|
| Back to top |
|
|
| munnster |
| Posted: Thu Apr 15, 2010 7:18 pm Post subject: |
|
|
Newbie
Joined: 10 Mar 2010
Posts: 5
|
| fjb_saper wrote: |
| Vitor wrote: |
Thinking about it, doesn't even the aggrigation node manage by setting the request up with a given name (folder name?) and the request msg id as correlation? So that would handle the unknown number of response problem as well surely?
I'm clearly missing something and/or have the wrong end of the stick. It happens, and I'm braced & ready to be embarrased when someone corrects me. |
I thought the aggregation node expected only a single response from each aggregation source. Don't know how that would play with a group...( probably all msgs in group with logical msg?)... but those settings would have to be put onto the MQInput node...
Have fun |
The aggregation node can handle multiple request/response pairs for each aggregation source. It just repeats the specified folder as many times as necessary. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Apr 15, 2010 11:08 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| munnster wrote: |
| fjb_saper wrote: |
I thought the aggregation node expected only a single response from each aggregation source. Don't know how that would play with a group...( probably all msgs in group with logical msg?)... but those settings would have to be put onto the MQInput node...
Have fun |
The aggregation node can handle multiple request/response pairs for each aggregation source. It just repeats the specified folder as many times as necessary. |
I had never tried it that way. Thanks for the clarification.
I guess the multiple responses need all to come in before the slowest source or timeout. When the 1st message from the slowest source or the timeout occurs the aggregation will propagate...
I am assuming here that we are not talking about a message group being sent by one of the sources.
In case of a message group I can see the aggregation waiting until the group is complete or timeout and passing multiple folders one per message in the group... Does it pass all of the group or none of the group or would it pass a partial group if the timeout occurred in the middle of the group?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| hallmark |
| Posted: Fri Apr 16, 2010 12:44 am Post subject: |
|
|
Voyager
Joined: 10 Mar 2005
Posts: 76
|
Hi
Just out of interest are all the replies from the same system. Are you able to use MQ grouping to logically group your replies?
EDIT: Should have read the above post first, sorry
Rob
_________________
Rob |
|
| Back to top |
|
|
| er_pankajgupta84 |
| Posted: Tue Feb 21, 2012 12:51 pm Post subject: |
|
|
Master
Joined: 14 Nov 2008
Posts: 203
Location: charlotte,NC, USA
|
I also have a similar situation.
Here is the requirement:
It is advance version of pub/sub where each subscriber would want some transformation to be done to its message.
Number of publisher =1.
Number of subscriber = n (not known untill runtime)
So one message will come to broker from an external client (CLIENT A) and same message should go to all the subscriber in the corresponding format. Each subscriber will reply with an acknowledgement to the message it got. Once all the messages have been sent to all the subscribers then collate all the acknowledgement from all the subscriber and send a consolidated response to client (CLIENT A).
I can implement this in a sequential manner i.e create one request and then send it to the subscriber and store the response in environment and repeat the process for each subcriber. Then at last combine the response from the environment into one message and send it back to original client.
But this way I am limiting my processing with the response time of the subscribers.
Is there better way of doing this?
How about using Aggregation nodes in recursion i.e starting a new aggregation for each third subscriber for example:
MqInput -> Compute -> AggregationControl -> Compute -> MQOutput -> AggregationRequest
-> "After AggregationControl Node " -> Compute -> MQOutput -> AggregationRequest
-> "After AggregationControl Node " -> Filter -> MQOutput -> AggregationRequest
Where the MQOutput node attached to the last (Filter) node is the same Q as that of MqInput.
Compute Node which is attached to MqInput node will take care of these things:
1. If the incoming message is the alone subscriber than simply reply using the MQReply node (Not mentioned above)
2. Do the lookup in the db to find the list of subscriber
3. Set the properties for transformation.
4. Other stuff related to recursion.
Similarly we can have a recursive Aggregation Reply flow.
I am not trying to be over ambitious here. Just wanted some expert advice before trying this POC.[/img] |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Feb 21, 2012 1:19 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Well, I don't know about a better way but if the number of subscribers is unknown until runtime you could consider a Collector node something like this:
- lookup the list of subscribers in the db
- send a transformed message for each subscriber, each message with the same correl id, storing the list of subscribers in a shared row variable for speed (or in the db if you prefer, or solidDb, or a Java singelton, or scratched on the inside of the server, or ... )
- use a Collector node to assemble all the replies by correl id
- send the results to the publisher using the shared row / db /singleton / scratches to identify missing responses
Or something like that.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Feb 21, 2012 1:22 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
If each subscriber wants or needs a unique transformation on the message, than you are in for a world of hurt. There's no way with base MQ pub/sub for a subscriber to indicate a transformation method.
You can use Subscription Points with Broker Publication nodes if there is a fixed set of transformations that is needed. You would then perform each transformation and send it to the relevant publication node. Each subscriber would know which subscription point had the transformed content it needed.
However using any kind of pub/sub properly isolates the publisher from the subscribers. There's no way to determine that there were any more than no subscribers at all.
So you don't have any fixed information about how many people received each type of message, and thus you have no way to determine if the set of responses received is the last set.
If you are not using real publish/subscribe, but instead implementing your own pub/sub engine using distribution lists or etc, then it's a different story.
You would use standard aggregation techniques that are not recursive, and use a separate flow to receive the responses than to send the requests. This gives you the scalability you're looking for, where you send all requests asynchronously from processing the responses.
But you have a big pile of requirements that need careful analysis and design, and you're not going to get that here. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Tue Feb 21, 2012 1:39 pm Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
People should really start new threads rather than reopen ones that are two years old. THAT MEANS YOU er_pankajgupta84. (third years-old thread you have reopened this week).
The WTX comment above is degated by the fact that people should not develop new flows with WTX any more.
Reopening the thread does not serve new users well.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
|
|
