| Author |
Message
|
| girish_tharwani |
 Posted: Tue Mar 03, 2009 12:09 am Post subject: Any solution for this problem without using Temp queues |
 |
|
Voyager
Joined: 01 Aug 2001
Posts: 88
Location: Pune, India
|
I am trying to find a solution for this problem using WMB built in mechanisms like Aggregation or Collector nodes so that I do not have to use temporary queues etc. No satisfactory solution yet. If you can give any pointers, I will really appreciate.
Problem: I need to query a backend system to get transaction history of past N days. Backend system accepts requests on MQ queue and responds on MQ queue but it’s a dumb system. It does not tell upfront how many transactions exists in past N days and it can send only 50 transactions in one reply so after sending the request, if I get a reply with less than 50 transactions, I am good but if I receive a reply with 50 transaction, I need to resend the request with an offset parameter of 50. This time in reply it sends me up to 50 transactions stating with 51st transaction in history. Now again, if number of transaction in reply less than 50, merge reply one with reply two and I am done else send third request with offset parameter of 100. Repeat till number of transaction in reply are less than 50.
Last edited by girish_tharwani on Tue Mar 03, 2009 4:32 am; edited 1 time in total |
|
| Back to top |
|
 |
| MQEnthu |
| Posted: Tue Mar 03, 2009 12:36 am Post subject: |
|
|
Partisan
Joined: 06 Oct 2008
Posts: 329
Location: India
|
Why not collector node?? anyways Use DB tables to store the intermediate replies, once the transcation in reply message is less than 50 you can merrge them and send it
_________________
-----------------------------------------------
It is good to remember the past,
but don't let past capture your future |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Mar 03, 2009 2:26 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
What solutions have you tried?
Why weren't they "satisfactory"?
MQEnthu - collector can't be used in an obvious way here because of the need to examine the result to find out if there is more stuff to be added to a collection. But that doesn't mean it can't be used.
<Flow Start>-->BuildRequest-->MQOutput-->MQGet(response)-->processResponse-->Collector--><rest of flow>
Looping done using propagate between BuildRequest and processResponse. ProcessResponse will notice when no more requests need to be sent and then send a message to Collector that will close the collection. |
|
| Back to top |
|
|
| MQEnthu |
| Posted: Tue Mar 03, 2009 2:51 am Post subject: |
|
|
Partisan
Joined: 06 Oct 2008
Posts: 329
Location: India
|
| mqjeff wrote: |
| MQEnthu - collector can't be used in an obvious way here because of the need to examine the result to find out if there is more stuff to be added to a collection. But that doesn't mean it can't be used. |
_________________
-----------------------------------------------
It is good to remember the past,
but don't let past capture your future |
|
| Back to top |
|
|
| girish_tharwani |
| Posted: Tue Mar 03, 2009 4:53 am Post subject: Any solution for this problem without using Temp queues |
|
|
Voyager
Joined: 01 Aug 2001
Posts: 88
Location: Pune, India
|
| mqjeff wrote: |
What solutions have you tried?
Why weren't they "satisfactory"?
<Flow Start>-->BuildRequest-->MQOutput-->MQGet(response)-->processResponse-->Collector--><rest of flow>
|
I am treating it as a two part problem. First do a loop and collect all reqeust at some place (queue or DB as MQEnthu suggested) and then read replies from this temp location and consolidate using some parameter (like corel id).
In first part, the issue is to send a flag from "processResponse" to "BuildRequest" informing it that it needs to send the reqeust message again and I also need to tell the "BuildRequest" the offset parameter to use in the request.
| mqjeff wrote: |
Looping done using propagate between BuildRequest and processResponse
|
I may be overlooking somthing very obvious here but I am not able figure out how to do this.
Second issue is that I do not want to use queue or DB for temp storage of reply parts. Collector node may be the answer to it but I need to read more to figure out how to achive what mqjeff said
| mqjeff wrote: |
ProcessResponse will notice when no more requests need to be sent and then send a message to Collector that will close the collection.
|
|
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Mar 03, 2009 5:05 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I meant PROPAGATE as in the ESQL command PROPAGATE, or the Java API method propagate().
The rest is left as an exercise, but remember the various different message trees you have to work with. |
|
| Back to top |
|
|
| girish_tharwani |
| Posted: Tue Mar 10, 2009 11:35 pm Post subject: Any solution for this problem without using Temp queues |
|
|
Voyager
Joined: 01 Aug 2001
Posts: 88
Location: Pune, India
|
I was put on some other task by "management" for a few days but now I am back to this problem and it has become critical now
. First part of problem, looping is sorted out (Thanks mqjeff). Now to the interesting part of making the collection using collector node.
'processResponse' is sending the messages to 'In' Terminal of Collector node. 'processResponse' also sends a control message to Collector where there are no more messages to be sent. The problem here is that control message does not signal Collector node that collection is complete. It just instructs the collector node that it’s time to propagate the completed collection. Only way the Collector node will mark the collection complete is by looking at 'Quantity' property or 'Timeout' property of 'In' terminal. Both of this will not work in my case. I do not know how many messages will arrive in individual collections and I do not want to set the ‘Timeout’ because sometime it may cause Collector node mark the collection complete even before all the messages has arrived and sometime Collector may keep waiting unnecessarily even after all the messages have arrived. |
|
| Back to top |
|
|
| girish_tharwani |
| Posted: Thu Mar 12, 2009 6:43 am Post subject: Any solution for this problem without using Temp queues |
|
|
Voyager
Joined: 01 Aug 2001
Posts: 88
Location: Pune, India
|
No takers of this..
so I guess I need to drop the idea of Collector node and think of something else. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Mar 12, 2009 7:01 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
No.
You just need an additional message in the collection, that will have only one instance.
So you feed all of your message segments into an unbounded collection input terminal. Then when you know you are done, you feed an additional message into an additional bounded input terminal. That closes the collection.
That's what I meant when I said "ProcessResponse will notice when no more requests need to be sent and then send a message to Collector that will close the collection.". |
|
| Back to top |
|
|
|
|
