| Author |
Message
|
| Rajat |
 Posted: Tue Jul 03, 2012 5:53 am Post subject: Parallel processing in Message Broker |
 |
|
Novice
Joined: 26 Nov 2009
Posts: 15
|
Hello Guys,
I have a requirement where I need to call a webservice which can result in number of responses(number can vary) and for each response I have to make another webservice call to the same system and then collate all the responses and send it back to the consumer.The idealisitic approach is to make these call in parallel. Could you please suggest how can I achieve this is the best possible timeframe?
I have considered,
1) Aggregation - but this doesnt seem to be a good approach as we are calling just one system..Please suggest ..
2) MQ Request Reply pattern : Where I can create request messages from each response received and put it in a queue, from where another flow can pick up the messages and make the call to the required system and then collate all the responses based on the messageid's and send it back to the consumer.
3) Any way do this direclty via HTTP ?
Please share your views.. Many Thanks in advance.. |
|
| Back to top |
|
 |
| lancelotlinc |
| Posted: Tue Jul 03, 2012 6:01 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
If using Web Services and SOAP, you can try the SOAPAsyncRequest node.
I'm concerned that the transaction volume would exceed your logic. If so, better to use the SOAPRequest node and not do parallel.
One way to accomplish parallel is like you suggested in #2: use MQ messages and aggregation that feeds other message flows with SOAPRequest nodes. I like this approach since it simplifies your logic to keep things straight. Other people may have different opinions.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jul 03, 2012 6:03 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Aggregation does apply here. You're doing exactly what it's designed for. You would simply combine your request/reply paradigm with the aggregation nodes.
There's no inbuilt method with Broker for spawning parallel threads, other than calling multiple instances of a flow. You can certainly write Java or .NET (if you're using v8 and on windows) code that calls all of the services and collects the responses in individual threads.
Another potential design approach is to use SOAPAsyncRequest/Response nodes. Then the http calls that contain the responses are received in individual threads as fast as they occur, and you can use collector to assemble them again.
But you're still using an Aggregation pattern, where you need to retain the information about the original request to be able to send a single reply. You can either try and assemble an mq based aggregation that includes the soapasync calls or you can try sticking the original request into the collector. You need to know when the last response has been received, in order to finalize the collection, though. Aggregation is easier that way than collection. |
|
| Back to top |
|
|
| Rajat |
| Posted: Thu Jul 05, 2012 2:37 am Post subject: |
|
|
Novice
Joined: 26 Nov 2009
Posts: 15
|
Thanks a lot guys for your inputs..
Planning to go with Aggregation pattern as suggested. Will keep you all posted regarding the development.
Once again Thanks.. |
|
| Back to top |
|
|
| PRKUMAR |
| Posted: Fri Sep 27, 2013 10:47 am Post subject: |
|
|
Apprentice
Joined: 04 Sep 2012
Posts: 36
|
I am also facing the same situation that Rajat has faced. Need to call a webserivce(1st system) which can result in number of sub order id's
for a given order id. For each of these responses(sub order id's), I need to create corresponding requests and to call another webservice(2nd
system) and then to collate all the respones(from the 2nd webserivce)
I have seen all the theads in this forum and other forums related to this topic:
Two solutions that you people suggested are:
1. Mix of request-reply pattern with aggregation nodes.
2. SOAP Asynchornous nodes
As per the documentation, SOAPAsyncRequest node can use HTTP or JMS transport. But the transport we are using is HTTPS. So using of
asynchronous nodes is not a feasible solution.
Hence I thought of using combination of mq req-reply with aggregation nodes. Following are the three flows I need to design for:
Fan-out request flow(flow1): In this flow, mqinput -> compute -> aggregatecontrol nodes comes in. I thought of connectiong 3 connections of
out terminal of aggregatecontrol node to three different paths having nodes in sequence: mqoutput node(each queue name is unique in each of
the three paths) and AggregateRequest node.
Aggregate request-reply flow(flow2): It actually contains 3 flows starting with MQInput nodes(that queue names we set in the flow1) and
makes remote soap webservice call and sends the response to a queue. For all the three flows, we are sending the response to only on common
queue.
Fan-in response flow(flow 3): This flow starts with AggregateReply node which combines all of the responses. And also send the single
response to the consumer.
Following are my queries:
1. Suppose by making call to 1st webservice, it is returning 6 sub-order id's. For each of these sub-order id's i need to create requests and
send it to second webservice. This will return response for each of the request. I need to collate all these responses. But, in the flow1, we
are connected only 3 connections of out terminal of AggregateControl node. It means it can handle to place only 3 sub-order id's individual
messages in 3 queues. And inturn these 3 requests will go to multiple aggregate request-reply flows and can execute parallel.
My question is, what about remaining 3 sub-order id's request messages. Do I need to put a logic in flow1 such that , 1st sub-order id
request message is placed in 1st mq output queue..2d req message in 2nd mq output queue..3rd request message in 3rd output-queue..and
again the loop starts back..such that..4th req message in 1st mq output queue, 5th req message in 2nd mq output queue and 6th req message
in 3rd mq output queue? Do I face any challenges in this approach? Is this right approach?
2. As said above, flow 2 will actually be 3 message flows starting with MQInput node. Instead, of creating 3 flows, can I develop 1 flow and
set additional instances of the flow as 2? What is the difference in creating multiple flows and creating 1 flow with additional instances of
the flow as 2? Performance wise which one is better?
3. Please help me what are the design flaws if I follow above approach?
4. Are they any other approaches aprt from using combination of req-reply with aggregation nodes?
My ultimate goal is to reduce the processing time of calling the web service for the dynamic number of results.
Our current design is, we are propagating each suborder request message one by one. Once we got the response from the previous suborder
request message then only we are sending new suborder request message. Thereby more processing time taking place. I want performance
improve suggestions. Please help. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Fri Sep 27, 2013 10:53 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| PRKUMAR wrote: |
I am also facing the same situation that Rajat has faced. Need to call a webserivce(1st system) which can result in number of sub order id's
for a given order id. For each of these responses(sub order id's), I need to create corresponding requests and to call another webservice(2nd
system) and then to collate all the respones(from the 2nd webserivce)
I have seen all the theads in this forum and other forums related to this topic:
Two solutions that you people suggested are:
1. Mix of request-reply pattern with aggregation nodes.
2. SOAP Asynchornous nodes
As per the documentation, SOAPAsyncRequest node can use HTTP or JMS transport. But the transport we are using is HTTPS. So using of
asynchronous nodes is not a feasible solution.
Hence I thought of using combination of mq req-reply with aggregation nodes. Following are the three flows I need to design for:
Fan-out request flow(flow1): In this flow, mqinput -> compute -> aggregatecontrol nodes comes in. I thought of connectiong 3 connections of
out terminal of aggregatecontrol node to three different paths having nodes in sequence: mqoutput node(each queue name is unique in each of
the three paths) and AggregateRequest node.
Aggregate request-reply flow(flow2): It actually contains 3 flows starting with MQInput nodes(that queue names we set in the flow1) and
makes remote soap webservice call and sends the response to a queue. For all the three flows, we are sending the response to only on common
queue.
Fan-in response flow(flow 3): This flow starts with AggregateReply node which combines all of the responses. And also send the single
response to the consumer.
Following are my queries:
1. Suppose by making call to 1st webservice, it is returning 6 sub-order id's. For each of these sub-order id's i need to create requests and
send it to second webservice. This will return response for each of the request. I need to collate all these responses. But, in the flow1, we
are connected only 3 connections of out terminal of AggregateControl node. It means it can handle to place only 3 sub-order id's individual
messages in 3 queues. And inturn these 3 requests will go to multiple aggregate request-reply flows and can execute parallel.
My question is, what about remaining 3 sub-order id's request messages. Do I need to put a logic in flow1 such that , 1st sub-order id
request message is placed in 1st mq output queue..2d req message in 2nd mq output queue..3rd request message in 3rd output-queue..and
again the loop starts back..such that..4th req message in 1st mq output queue, 5th req message in 2nd mq output queue and 6th req message
in 3rd mq output queue? Do I face any challenges in this approach? Is this right approach?
2. As said above, flow 2 will actually be 3 message flows starting with MQInput node. Instead, of creating 3 flows, can I develop 1 flow and
set additional instances of the flow as 2? What is the difference in creating multiple flows and creating 1 flow with additional instances of
the flow as 2? Performance wise which one is better?
3. Please help me what are the design flaws if I follow above approach?
4. Are they any other approaches aprt from using combination of req-reply with aggregation nodes?
My ultimate goal is to reduce the processing time of calling the web service for the dynamic number of results.
Our current design is, we are propagating each suborder request message one by one. Once we got the response from the previous suborder
request message then only we are sending new suborder request message. Thereby more processing time taking place. I want performance
improve suggestions. Please help. |
>> I am also facing the same situation
No you are not. You need to create your own thread. You also need to state what version you are using. Version numbers have four digits. You also need to state what you have tried and why what you have tried is not working for you. The suggestion to the OP was to use Aggregation. Did you try it? If no, why have you not yet tried it? You should identify if any affinity exists between each request. You have not stated that, but it seems that there is none. Did you try the Web Service Aggregation sample ? http://publib.boulder.ibm.com/infocenter/wmbhelp/v8r0m0/index.jsp?topic=%2Fcom.ibm.etools.mft.samples.wsaggregation.doc%2Fdoc%2Foverview.htm
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER
Last edited by lancelotlinc on Fri Sep 27, 2013 11:10 am; edited 1 time in total |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Sep 27, 2013 11:10 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
There is a very excellent sample that COMES WITH THE PRODUCT THAT EXACTLY DESCRIBES A SOLUTION.
Go use it.
Don't reinvent this. |
|
| Back to top |
|
|
| PRKUMAR |
| Posted: Sun Sep 29, 2013 7:36 am Post subject: |
|
|
Apprentice
Joined: 04 Sep 2012
Posts: 36
|
The version that we are using is Message Broker 7.0
The web service aggregation sample mentioned in the link is using of SOAP Async nodes. Does it support HTTPS transport mechanism? |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sun Sep 29, 2013 8:15 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| PRKUMAR wrote: |
The version that we are using is Message Broker 7.0
The web service aggregation sample mentioned in the link is using of SOAP Async nodes. Does it support HTTPS transport mechanism? |
What does the Infocentre say?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Sun Sep 29, 2013 8:22 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| PRKUMAR wrote: |
| The web service aggregation sample mentioned in the link is using of SOAP Async nodes. Does it support HTTPS transport mechanism? |
No, when IBM implemented the web service support in WMB they completely overlooked the HTTPS protocol....!  
How did that question make sense in your head? Given that you thought it was a reasonable question, why did you decide to post it here rather than type "HTTPS" or "web service security" into the InfoCenter search engine? Or indeed (and this is a bit radical) look at the SOAP Async node, which has 4 parameters on the HTTP tab to configure SSL support?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Mon Sep 30, 2013 3:38 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
|
| Back to top |
|
|
|
|
