| Author |
Message
|
| bprasana |
 Posted: Tue Aug 11, 2009 12:30 am Post subject: Best way to implement Consolidation of WebService responses |
 |
|
Disciple
Joined: 18 Apr 2005
Posts: 179
|
Hi All,
Background:
I make a webservice call which will return a set of values. Depending on the number of values I make another webservice call for each value. I will have to collect all the responses and then extract a value(numeric) from them and then provide the lowest value as output. How do i implement the colletion of these request in single transaction?
I can construct and propagate webservice request in a for loop. But is there a way to consolidate these responses in single message flow.
Cheers
bprasana |
|
| Back to top |
|
 |
| Luke |
| Posted: Wed Aug 12, 2009 1:06 am Post subject: |
|
|
Centurion
Joined: 10 Nov 2008
Posts: 128
Location: UK
|
|
| Back to top |
|
|
| Luke |
| Posted: Wed Aug 12, 2009 1:19 am Post subject: |
|
|
Centurion
Joined: 10 Nov 2008
Posts: 128
Location: UK
|
Although, if you really need it all to be part of the same transaction ... do you really? It'll be horrible ... I guess you'll have to go to them all one after the other within the same flow, store away the results after each webservice ... likely to be lots of hanging around waiting in the flow.
If you can get away with making it a bit asynchronous (but with timeouts and error handling of course), I'd say better to use aggregation - build separate flows to invoke the webservices and invoke these flows via MQ, using MQReply to specify where you want the reply to go to. The aggregation nodes can then handle collecting your responses ... that's how I'd do it anyway.
Cheers |
|
| Back to top |
|
|
| bprasana |
| Posted: Wed Aug 12, 2009 6:20 am Post subject: hmm |
|
|
Disciple
Joined: 18 Apr 2005
Posts: 179
|
Hi Luke,
Thanks! that makes sense though. Let me discuss this with other designers too.
Also performance needs to be considered and it should be significantly less than the Java app. that is currently doing this work.
But will it really reduce any 'processing' time? We are just converting 'processing' time into MQ waiting time arent we? (Please correct me if Iam wrong!!)
Cheers
bprasana |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Aug 12, 2009 6:36 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Parallel processing using aggregation can reduce the total linear time taken to complete the procedure, because the total time waiting for responses is as close to time it takes for the slowest response to return as can be achieved.
So instead of waiting 3 seconds for response 1, and then 5 seconds for response 2 and then 1 second for response 3, you are only waiting the 5 seconds for response 2, and in that period of time response 1 and response 3 have already been received and processed as much as they can be. |
|
| Back to top |
|
|
| Luke |
| Posted: Wed Aug 12, 2009 6:44 am Post subject: |
|
|
Centurion
Joined: 10 Nov 2008
Posts: 128
Location: UK
|
As mqjeff says, yes for linear time, although obviously 'processing' time in terms of CPU time will not be reduced.
Another advantage of separating the webservice calls is that it's a lot easier to monitor the performance of each and therefore tune them appropriately.
You might get some people telling you to avoid aggregation by the way - it used to be a bit slow, but it's much better nowadays and worth the effort IMHO |
|
| Back to top |
|
|
| bprasana |
| Posted: Wed Aug 12, 2009 6:49 am Post subject: Am I missing something? |
|
|
Disciple
Joined: 18 Apr 2005
Posts: 179
|
Hi mqjeff,
I must confess I havent used aggregation before, but are you suggesting that i increase the instances of 'webservice calling' flow? So that they can be processed concurrently? and then collect the responses using the aggregation nodes(not sure if its possible???) again?
The other question is, Can I use aggregation when the number of calls I make to webservice is not fixed i.e., is dynamic?(I seriously need a class on Aggregation!!!  )
Cheers
bprasana |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Aug 12, 2009 6:52 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Luke wrote: |
| You might get some people telling you to avoid aggregation by the way - it used to be a bit slow, but it's much better nowadays and worth the effort IMHO |
The new Collector node is much better than the old Aggregation used to be.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Aug 12, 2009 7:21 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| Luke wrote: |
| You might get some people telling you to avoid aggregation by the way - it used to be a bit slow, but it's much better nowadays and worth the effort IMHO |
The new Collector node is much better than the old Aggregation used to be. |
But of course they solve entirely separate problems and meet entirely different business cases. You can't use Collector for fan-out/fan-in patterns, without jumping through a lot of hoops, and likewise you can't use Aggregation to compile multiple independent inputs into one message.
bprasana - it will be complicated and perhaps a bit painful to use Aggregation for a dynamic number of requests. It may not even be achievable. This is likewise difficult using Collection, but may be a bit easier to build around.
And, no, I was not suggesting anything about additional instances. Merely that using Aggregation, with new flows started from MQInput nodes, can run each individual request in parallel. This reduces the linear time taken to complete the total set - but does nothing to reduce the concrete time to complete each individual request.
In general, nothing really reduces the concrete time to complete each individual request. You may find you can get better performance out of a message broker flow than out of a plain java program, and you may not. It depends quite a bit on the individual implementations. |
|
| Back to top |
|
|
| Luke |
| Posted: Wed Aug 12, 2009 7:51 am Post subject: |
|
|
Centurion
Joined: 10 Nov 2008
Posts: 128
Location: UK
|
Re: Performance of Collector v aggregation nodes, I'd guess they'd be similar in performance as both use queues to store the data now, whereas aggregation used to use the Broker's DB, hence some performance problems.
Also, regarding the dynamic number of requests, I'm pretty sure you can do this. It's been a while since I've used aggregation, but I seem to remember this being done at a company I used to work for. Wouldn't the details of the requests sent be recorded by the AggregateRequest node, then passed in the control message? I don't see any reason logically why the number of requests sent should be fixed ...
I could be wrong though.
If you want to learn more and see if it's suitable for you, the infocenter and the suggested samples should give plenty to go on. You could even play with the samples a bit and see if the dynamic requests will work. |
|
| Back to top |
|
|
| bprasana |
| Posted: Thu Aug 13, 2009 3:12 am Post subject: Super |
|
|
Disciple
Joined: 18 Apr 2005
Posts: 179
|
Thats a whole lot of info. I will try a few things and let you guys know the result
Thanks for all your help
bprasana |
|
| Back to top |
|
|
|
|
