| Author |
Message
|
| francisborgio |
 Posted: Fri Nov 22, 2013 4:33 am Post subject: Query in Collector Node |
 |
|
Newbie
Joined: 22 Nov 2013
Posts: 4
|
| I need to Configure Collector Node in such a way that I need stop collecting the input messages when a condition is met. For example if the input message has a XML tag as <StopCollecting>True<StopCollecting>. Can anyone help me on this? |
|
| Back to top |
|
 |
| Vitor |
| Posted: Fri Nov 22, 2013 5:34 am Post subject: Re: Query in Collector Node |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| francisborgio wrote: |
| Can anyone help me on this? |
Yes. The Collector nodes don't have this ability.
You'll need to build some other mechanism to achieve this. Storing the "collection" in a database (or cache, depending on WMB version and the sixe of the data involved) until you detect this stop condition would seem an obvious solution.
Other solutions are undoubtably possible and may be better.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| francisborgio |
| Posted: Fri Nov 22, 2013 5:42 am Post subject: |
|
|
Newbie
Joined: 22 Nov 2013
Posts: 4
|
| Thanks for your reply. But Architect has clearly mentioned to use Collector Node in his design. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Nov 22, 2013 5:45 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| francisborgio wrote: |
| But Architect has clearly mentioned to use Collector Node in his design. |
Then ask him how to use a Collector node like this.
We must assume he knows more than I do about this. Something which happens depressingly frequently.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Nov 22, 2013 6:43 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| francisborgio wrote: |
| Thanks for your reply. But Architect has clearly mentioned to use Collector Node in his design. |
It would be nice if you would go and ask your Architect (nicely) to give us the reasons for this 'You MUST use a Collector Node' for him/her self.
Honestly, we see a lot of 'The Requirement is' (see my post with this title) and 'I've been told to <insert impossible requirement here>' statements here. Many of them are I know made by people who really don't know this product. Some of us have been using it for more than a decade. There are even a few people from IBM who work on the product who post here. However we'd really like to learn how a Collector Node can be used to do this without reverting to doing something like making the input queue 'Get Inhibited'.
If your Architect wants to come and enlighten us on this topic, we would welcome it very much.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| joebuckeye |
| Posted: Fri Nov 22, 2013 6:46 am Post subject: |
|
|
Partisan
Joined: 24 Aug 2007
Posts: 365
Location: Columbus, OH
|
| francisborgio wrote: |
| But Architect has clearly mentioned to use Collector Node in his design. |
Wouldn't be the first time an architect (and/or management) hasn't understood the details of how something is implemented. |
|
| Back to top |
|
|
| goffinf |
| Posted: Sat Nov 23, 2013 3:54 am Post subject: |
|
|
Chevalier
Joined: 05 Nov 2005
Posts: 401
|
As Victor has strongly suggested, AFAIK (and it has been a while since I used the Collector) I don't think it supports stop/start (the control terminal is used for deciding when to propagate complete collections),
So I guess you need to have a mechanism either before or after the collector (better to before so you don't un-necessarily store messages you don't want). That could mean storing some global state (a simple on/off flag) that is checked befor sending a message to the Collector (iwhere 'stop' = discard message).
Of course these sort of statefull solutions need to be thought through carefully to avoid things like (a) a race condition if your solution is running additional instances or is spread across EGs and/or Brokers and (b) what happens when your state store is inaccessible or dies unexpectedly. For (a), if your collection size is large enough (versus the rate at which messages are received and the time taken to check the state) then the fact that a few get thru before the 'stop' state is noticed probably won't matter. They will just sit around until that collection expires. For (b) you need a default state value that does the least damage.
You haven't said whether you want to toggle between start and stop, so I'll ignore the issues about starting up again for now, but there are some to think about.
One other thing. The Collector can be a tricky node to work with. Everything after the node runs in its own thread and is like running a separate flow, so none of the content within messages trees before the collector are available, so if you need any of that you need to propagate it inside your message. Also because of the threading model, it's quite easy for a bottleneck to occur when you have high thruput and this can blow your collection queue depths if they are not high enough (it's to do with how the collector notices that a collection is complete and propagates it - a longer explanation won't fit into this response, so just a heads up to test thoroughly at least to your expected peak thruput and preferably a bit beyond). Its one of the reasons I have steered clear if there's a more controllable approach, although its fair to say that the Collector has had a succession of bug fixes applied so some of these issues may be resolved (but testing is everything if you want to be sure).
HTHs
Fraser. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sat Nov 23, 2013 7:28 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
The collector node propagates a complete collection when all of the conditions that have been set on all of it's input terminals are complete.
It's simply a matter of making sure you configure the correct number of input terminals and configure the conditions on them.
There's nothing that says that one input message == one input terminal. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Nov 25, 2013 5:50 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
The collector node propagates a complete collection when all of the conditions that have been set on all of it's input terminals are complete.
It's simply a matter of making sure you configure the correct number of input terminals and configure the conditions on them.
There's nothing that says that one input message == one input terminal. |
Yes, but urgh!
I would assert that while it's possible to collect an open ended collection in the way you describe, calling that a "collection" is stretching the point.
@francisborgio - as my most worthy associate pedantically points out, it is indeed possible to bend the Collector node to do this. I stand by my advice of a) it doesn't possess this ability (as you'll discover if you attempt it) even if it can be coded to do this and b) ask the architect how he would do this & push back if the answer is "well it's a Collector node; get it to collect them somehow"
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Esa |
| Posted: Mon Nov 25, 2013 6:05 am Post subject: Re: Query in Collector Node |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| francisborgio wrote: |
| I need to Configure Collector Node in such a way that I need stop collecting the input messages when a condition is met. For example if the input message has a XML tag as <StopCollecting>True<StopCollecting>. Can anyone help me on this? |
It's still a bit unclear if you mean
- stop collecting and start throwing all messages away
or
- stop collecting this collection and start a new collection
The second alternative is quite simple to implement with a Collector node.
The first alternative is actually quite simple, too - with a preprocessor compute node. |
|
| Back to top |
|
|
| francisborgio |
| Posted: Thu Nov 28, 2013 6:45 am Post subject: |
|
|
Newbie
Joined: 22 Nov 2013
Posts: 4
|
| Is it possible to dynamically configure the Property Quantity of the Collector Node from the request message. My flow will starts with SOAPInput and I am using Collector Node next to SoapInput node. |
|
| Back to top |
|
|
| Esa |
| Posted: Thu Nov 28, 2013 11:03 pm Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| francisborgio wrote: |
| Is it possible to dynamically configure the Property Quantity of the Collector Node from the request message. My flow will starts with SOAPInput and I am using Collector Node next to SoapInput node. |
No. You could have checked this easily in the infocenter yourself. 'No' in the C column means that you can't.
So the first web service call has information about the number of calls to follow?
Hopefully the call also has some kind of identity that can be used to distinguish between collections. In case there can be several collections happening simultaneously.
You need to add a compute node (or a JCN if you like that better) in front of the collector node. There you need to maintain a static hashtable (packaged in a Singleton, preferably) or an ESQL shared row tree.
For each id you need to have two values: quantity and counter.
Set quantity to the expected number of messages for the collection and counter to 1 (or 0 if using java).
Each time a message passes the node, you increment counter. If you detect that the collection is ready, you propagate to the control terminal of the Collector node and delete the entry from the hashtable/shared row. Check yourself if you need to propagate the actual message to control terminal or to the same terminal as the other messages and then send just a trigger to control terminal, I don't have time to check that. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Nov 29, 2013 7:15 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Esa wrote: |
You need to add a compute node (or a JCN if you like that better) in front of the collector node. There you need to maintain a static hashtable (packaged in a Singleton, preferably) or an ESQL shared row tree.
For each id you need to have two values: quantity and counter. |
I put it to you that if you build this kind of mechanism (which I alluded to earlier in this post), you might as well extend it to include storage of the message payload & eliminate the Collector node completely. Once you've built this part, the Collector node is not adding that much value.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Nov 29, 2013 7:35 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
I replaced a collector node where we were dealing with an open ended collection (using the collector node timeout) after a few execution group hangs after approx a month of operation.
It was replaced with a compute node writing the message payload to a table and a timer kicking in every 30 seconds and selecting those grouped messaged that have not had anything added fro at least 10 seconds.
so far this has proved 100% reliable and has processed in excess of 10 million messages into around 1M collections. The number of entries in each collection ranges (so far) from 1 to 22.
Thankfully there is a field in the message that enables the easy grouping.
Where a collector node has proved reliable is where there is a fixed size to the collection.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Ksawery |
| Posted: Mon Jun 10, 2024 6:11 am Post subject: Using Collector Node to collect messages from the MQ queue |
|
|
Newbie
Joined: 12 Oct 2022
Posts: 6
|
Please advise me if you have ever encountered a similar problem.
I have an application where I intend to use a Collector node to collect segmented messages from the MQ Input queue. The message segments have a common denominator in the form of an MQMD/GroupId header.
The flow is fairly simple. The input is MQ Input > Collector Node > Compute Node (here I put together the body) > FileOutput. According to the documentation, set the following node properties values:
- quantity = 0 (in case we do not know the number of input messages),
- TimeOut = 30 sec (example)
- Collection Expiry = 60 sec (example).
- Correlation Path = $Root/MQMD/GroupId
In theory, this much is enough for the node to collect all messages with the same group ID and combine them into a single collection. Once it has one complete collection, it lets the terminal OUT for further processing, and so on. Practically, this doesn't work for me.
Problem 1: if I have a value set for the Collection Expiry property then ALWAYS messages (completed, or incompleted) go out with the EXPIRY terminal (Yes, this is also written in the documentation). And Collection Expiry overwrites the value of the TimeOut property.
Problem 2: For some reason, the MQ queue serves messages very slowly, exactly 3 messages each about 60 seconds.
Which makes the whole process take a very long time. If I have 120 messages on the queue you can already imagine...
I want to set collection expiry property for eventually error handling (oh boy, the collection is not completed, then let's try again!), but in this case everything goes this way, so...
Perhaps some of you have already had similar problems and can point out a possible cause and suggest a solution? I would be very grateful!
My goal is:
1) Get the messages from the queue,
2) Merge them into a collection according to the group ID match,
3) Propagate for further message processing.
I can't achieve this because
1) messages come out of the queue too slowly and not immediately,
2) collections whether full or not full always come out terminal expiry.
It is worth adding that I have tried really a lot of different combinations, but everything always ends the same, and I do not want to believe that the node does not work. |
|
| Back to top |
|
|
|
|
