| Author |
Message
|
| Deepali |
 Posted: Tue Apr 09, 2013 1:26 am Post subject: Reading messages on trigger in WMB V8 |
 |
|
Novice
Joined: 05 Mar 2013
Posts: 19
|
We have a situation where there are 2 queues Q1 and Q2. Both have multiple messages. We need to compare and match messages from Q1 and Q2 based on a data field in the messages. On matching we would remove the messages from the respective queues and send both of them to a collector node. The process can begin as soon as there is a message In Q1. We need to browse all messages from Q2 and find match with message from Q1.
We are using WMB V8 and MQ V7.
Need help for this scenario.  |
|
| Back to top |
|
 |
| McueMart |
| Posted: Tue Apr 09, 2013 2:07 am Post subject: |
|
|
Chevalier
Joined: 29 Nov 2011
Posts: 490
Location: UK...somewhere
|
Rather than 'browsing all messages on Q2 and finding a match with the messsage from Q1' , think about how you could use the MQGet node's 'Get by CorrelId' or 'Get by MsgId' functionality to optimise the process.
You might have to modify the sending application to set the correct MsgId/CorrelId, or you could write an intermediate flow to set them to what you need. |
|
| Back to top |
|
|
| Deepali |
| Posted: Tue Apr 09, 2013 2:59 am Post subject: |
|
|
Novice
Joined: 05 Mar 2013
Posts: 19
|
Thanks McueMart,
The actual situation is a bit complicated. After passing through 2 WTX map nodes the result is put into Q1. Also this is very asynchronous process. While reading one message from Q1, there may be many in Q2. And we cannot change maps. So will not be able to use correlation id.
Or if we use Id, but still we should find the matching one from Q2.
We were thinking of some indexing or using Java for implementing searching algorithm. |
|
| Back to top |
|
|
| McueMart |
| Posted: Tue Apr 09, 2013 3:17 am Post subject: |
|
|
Chevalier
Joined: 29 Nov 2011
Posts: 490
Location: UK...somewhere
|
Ok, based on the assumption that you cant change the applications putting onto Q1/Q2.....How about creating an intermediate flow which reads off Q2, locates the correlation information in the payload of the message, and then puts the message back onto a new queue (Q3), with the CorrelId set to something useful.
Then in your main flow (which reads from Q1), you would then use a MQGet node, and 'get by CorrelId' to pull the message off Q3 which you want.
This is far more efficient than having every message which is read off Q1 having to search through every message on Q2 to find the one it wants. |
|
| Back to top |
|
|
| kimbert |
| Posted: Tue Apr 09, 2013 4:07 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
As a general rule, every time you think of putting an MQGet node into a loop in a message flow you should immediately try very hard to think of something else.
It is bad for performance, will exhaust the stack if not implemented properly ( and it usually is not ) and there is almost always a better way. |
|
| Back to top |
|
|
| mgk |
| Posted: Tue Apr 09, 2013 5:05 am Post subject: |
|
|
Padawan
Joined: 31 Jul 2003
Posts: 1642
|
| Quote: |
| As a general rule, every time you think of putting an MQGet node into a loop in a message flow you should immediately try very hard to think of something else. |
Indeed. A better was to do this is to loop inside an ESQL / Java or .NET compute node instead, calling PROPAGATE (or equivalent) to invoke the MQGet node...
Kind regards,
_________________
MGK
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Apr 09, 2013 5:05 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Yes, and put another way - if you are browsing messages in an business application - you are probably doing something wrong - or at least non-optimal.
You can get exponentially expensive looping as message depths build up with this sort of design. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Apr 09, 2013 5:09 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I'm really confused.
The collector node should absolutely just "match" the necessary messages together, and propagate a collection of them.
If configured properly to understand the location of the correlating data, and to understand where one collection stops and another begins.
There should be no need to browse anything or use an MQGET in a loop or etc... It should be a matter of, at most, a couple of MQInput nodes feeding some kind of transformation node (your Compute flavor of choice) and then feeding into Collector. |
|
| Back to top |
|
|
| Deepali |
| Posted: Tue Apr 09, 2013 9:05 pm Post subject: |
|
|
Novice
Joined: 05 Mar 2013
Posts: 19
|
Thanks all,
We had gone all these thoughts.
@mqjeff, it is possible to match the fields in collector node. But the two inputs are so asynchronous and also dynamic that we cannot figure out how to configure collector node for it. For one input terminal it's fine that the quantity parameter is'1'. But for second input terminal the number of messages coming are dynamic. So in that case we must give timeout property. But then after timeout we will loose incomplete messages,  which we don't want.
So any suggestion in such situation. |
|
| Back to top |
|
|
| Esa |
| Posted: Tue Apr 09, 2013 11:05 pm Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| mgk wrote: |
| Quote: |
| As a general rule, every time you think of putting an MQGet node into a loop in a message flow you should immediately try very hard to think of something else. |
Indeed. A better was to do this is to loop inside an ESQL / Java or .NET compute node instead, calling PROPAGATE (or equivalent) to invoke the MQGet node...
|
And cache the relevant fields and msgids to avoid unnecessary looping. Have another MQGet for getting selected messages by msgid, or programmatically modify get options.
But the collector node alternative is certainly simplier to implement, and quite all right from performance poit of view, too, unless you are collecting large messages. |
|
| Back to top |
|
|
| marko.pitkanen |
| Posted: Wed Apr 10, 2013 12:05 am Post subject: |
|
|
Chevalier
Joined: 23 Jul 2008
Posts: 440
Location: Jamsa, Finland
|
Any possibility to use external database for this aggregation? Database could perhaps provide you more efficient searching methods than browsing the queue?
--
Marko |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Apr 10, 2013 3:10 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Deepali wrote: |
Thanks all,
We had gone all these thoughts.
@mqjeff, it is possible to match the fields in collector node. But the two inputs are so asynchronous and also dynamic that we cannot figure out how to configure collector node for it. For one input terminal it's fine that the quantity parameter is'1'. But for second input terminal the number of messages coming are dynamic. So in that case we must give timeout property. But then after timeout we will loose incomplete messages, which we don't want.
So any suggestion in such situation. |
Three inputs. One from the first queue, that starts a given collection. One from the second queue, that feeds in messages on a given collection until it is noticed that the 'last' message is received. This then feeds the third input which says 'the collection is complete'. |
|
| Back to top |
|
|
| Deepali |
| Posted: Wed Apr 24, 2013 1:28 am Post subject: |
|
|
Novice
Joined: 05 Mar 2013
Posts: 19
|
Thanks all,
We have solved the problem. We modified the messages to set correlation id with all messages those belongs to one set and configured the collector node properly. It worked fine. |
|
| Back to top |
|
|
|
|
