| Author |
Message
|
| IIB_Intel |
 Posted: Mon Nov 14, 2016 8:11 pm Post subject: IIB aggregation with http design |
 |
|
Acolyte
Joined: 07 May 2015
Posts: 64
|
I have a requirement were I have to call a service for each identifier in my input and then send a consolidated reply back.
Flow design:
HttpInput -> AggregateControl -> Compute -> Http Asynchronous Request (backend service async call) -> AggregateRequest
Within the same flow I have defined another path for response:
Http Asynchronous Respone -> AggregateReply -> compute -> HttpReply
Do you see any flaw with this design. Anything which I can do better.
P.S:
- number of identifier in my input can vary
- Backend service that I am calling can accept only one identifier at a time
- I would be running this flow with multiple instances. |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Tue Nov 15, 2016 5:06 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Because of your limitations then I don't think that your design will work.
Open ended (due to your 1...n) aggregations are a PITA
What I think you may have to do is... (off the top of my head...)
- Separate the flow into multiple flows
- Send the 1...n identified requests to a separate flow via MQ. Get the response back via MQ.
- Tag the last request and make sure that the second flow sends back the tag.
- Read the messages with an MQGET node and save the results until you get the end tag
- build the response to the caller from the saved responses.
Now that is a simplified description. Depending upon how big 'n' is, you may need to store the results in a DB while you get all the others.
You do need to send the reply back from the first flow because it keeps the HTTP Request ID etc in the memory being used by that thread.
- Multiple instances may need special coding so that you don't get the wrong data in a reply if you store it in a DB.
- If 'n' is big then the overall throughput might be poor so beware of requestor timeouts.
Because the HTTP service you are requesting is basically single threaded then you may well run into real performance issues especially if it is external and uses SSL/TLS.
Now that I think about it, this is a horrible requirement. Needs some real careful design and lots and lots and lots of testing with really high 'n' values
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Nov 15, 2016 5:42 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
Don't know why you'd be looking at the aggregation design when this looks so much more like a collector design... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| IIB_Intel |
| Posted: Tue Nov 15, 2016 5:49 am Post subject: |
|
|
Acolyte
Joined: 07 May 2015
Posts: 64
|
Let me put some more clarity on my requirements..
n will not be greater than 5 and the backend service is not single threaded, it can accept many requests at a time but cannot accept more than one identifier.
Compute node before AsyncHttpRequest is having a loop so I am calling my backend service using asyncHttpRequest one after the other.
I can also split 2 segments of my flow into two separate flows i.e. have asyncHttpResponse in separate flow.
Now on your suggestion of tagging the nth request, isn't aggregation nodes internally does some sort of tagging? |
|
| Back to top |
|
|
| IIB_Intel |
| Posted: Tue Nov 15, 2016 7:51 am Post subject: |
|
|
Acolyte
Joined: 07 May 2015
Posts: 64
|
| Quote: |
Don't know why you'd be looking at the aggregation design when this looks so much more like a collector design... |
I doubt that as collector node needs either quantity or expiry time to finish collection.
Here quantity is variable so I cannot use that and putting an expiry time means each request has to wait until that time before completion it done.
Let me know if I am missing anything on collector node here.
I can use collector node if I tag my nth request and propagate it through different source to collector node input. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Nov 15, 2016 8:33 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
| IIB_Intel wrote: |
| Quote: |
Don't know why you'd be looking at the aggregation design when this looks so much more like a collector design... |
I doubt that as collector node needs either quantity or expiry time to finish collection.
|
In fact I thought it needed key + quantity or key + expiry...
Would that not match your overall design? or is the key not present on the 1-n requests?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| IIB_Intel |
| Posted: Tue Nov 15, 2016 9:56 am Post subject: |
|
|
Acolyte
Joined: 07 May 2015
Posts: 64
|
| Key could be original http reply identifier but I don't have the quantity as fixed. I don't want to use any expiry as I don't want to wait until a fixed time for all requests. |
|
| Back to top |
|
|
| IIB_Intel |
| Posted: Tue Nov 15, 2016 10:05 am Post subject: |
|
|
Acolyte
Joined: 07 May 2015
Posts: 64
|
I also would like to add a quote from http://www.mqseries.net/phpBB2/viewtopic.php?t=66500&highlight=collector+quantity by mqjeff -
| Quote: |
"The big difference between the two (aggregation & collection) is whether or not there is a single input that starts the pattern - i.e. are you taking a single message and turning it into N requests and then assembling n replies? Or are you taking N requests and assembling N replies?
If it's one message, that's Aggregation. If it's N messages, then that's Collection" |
My scenario is single input. If we have a way to dynamically set the quantity for a collection then I can use that instead of aggregation.[/quote] |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Nov 16, 2016 8:42 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You can use an input with an unlimited # of records in a collector node.
However it requires an additional input with a fixed number of records that indicate that a given collection is complete.
So you could send in all the records from the http response to the unlimited input, and then send the identifier to the limited input.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| IIB_Intel |
| Posted: Fri Nov 18, 2016 5:06 am Post subject: |
|
|
Acolyte
Joined: 07 May 2015
Posts: 64
|
Not doable in my situation...
I will get a request with N identifier and then calling a backend service for each identifier one at a time. This backend call is asynchronous so I won't know when all calls are done. Even if I tag my nth identifier and send it to a 2nd terminal of the collector node I cannot gaurantee that all (n-1) have completed and arrived on terminal 1 of the collector node.
Do you see any challenge with the aggregation approach I have taken? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Nov 18, 2016 5:13 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Ok. So you are getting a value "n". You are then calling a webservice for each N. Presumably in a loop that does a propagate.
Once you have propagated all N, the last propagate will only return when the response message has gone to the collector input - because that's when the message flow ends for that propagate - think of the Collector input like an MQOutput node.
Your loop will end, because you know you have counted up to N.
Then send another message to the other collector input...
Yes?
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Nov 18, 2016 5:18 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| IIB_Intel wrote: |
Even if I tag my nth identifier and send it to a 2nd terminal of the collector node I cannot gaurantee that all (n-1) have completed and arrived on terminal 1 of the collector node.
|
Yes you can.
Because you are probably propagating the requests to anothe bit of your flow. i.e. the bit that does the HTTP operation then your flow knows when it has reached the end of the line. The last request can be as indicated 'tagged' to show this fact.
Then in another compute node BEFORE the collector node, to add some ESQL that looks for the end tag. Everything is passed through to the collector node and then when the end tag is found, it can send a message to the second terminal of the collector node to mark the end of the collection.
Well, that's where I'd start. You really do need to start a POC for yourself. It may not work but instead of taking thr words of us 'strangers on the internet' as being 100% correct, do some experimentation and see what transpired. This way you will get a far better understanding of the real issues.
If I had raised the initial post, I would have already started trying things out for myself instead of waiting on us to respond.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| IIB_Intel |
| Posted: Fri Nov 18, 2016 10:29 am Post subject: |
|
|
Acolyte
Joined: 07 May 2015
Posts: 64
|
| Quote: |
Ok. So you are getting a value "n". You are then calling a webservice for each N. Presumably in a loop that does a propagate.
Once you have propagated all N, the last propagate will only return when the response message has gone to the collector input - because that's when the message flow ends for that propagate - think of the Collector input like an MQOutput node.
Your loop will end, because you know you have counted up to N.
Then send another message to the other collector input...
Yes? |
This hold good assuming webservice is called sequentially for all ids.. 1..n but I don't want to wait for the response of 1st id before I can call the service for 2nd id so I have 2 options:
1. Either bring a queue in between and decouple the task.
2. Use asynchronous http request node (which I used).
In both the cases, If I am calling a webservice (which is multi-threaded in its own domain) then there is a chance that response to nth request may come before n-1 request.
Because of this probability I inclined towards using aggregation instead of collection. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Nov 18, 2016 10:36 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You can use a counter in a shared variable or global cache or something. Add 1 each time you send an asynch http call, subtract one each time you get a response.
Propagate to the other collection terminal when your counter is 0 again.
Aggregation is perfectly reasonable here, though.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| IIB_Intel |
| Posted: Fri Nov 18, 2016 10:41 am Post subject: |
|
|
Acolyte
Joined: 07 May 2015
Posts: 64
|
| Quote: |
Yes you can.
Because you are probably propagating the requests to anothe bit of your flow. i.e. the bit that does the HTTP operation then your flow knows when it has reached the end of the line. The last request can be as indicated 'tagged' to show this fact.
Then in another compute node BEFORE the collector node, to add some ESQL that looks for the end tag. Everything is passed through to the collector node and then when the end tag is found, it can send a message to the second terminal of the collector node to mark the end of the collection.
Well, that's where I'd start. You really do need to start a POC for yourself. It may not work but instead of taking thr words of us 'strangers on the internet' as being 100% correct, do some experimentation and see what transpired. This way you will get a far better understanding of the real issues.
If I had raised the initial post, I would have already started trying things out for myself instead of waiting on us to respond. |
My original post says I implemented the solution in a manner and it is working fine, I was looking for some issue if I may encounter with that design since this is the first time I was using aggregation nodes.
To your other point on tagging the nth request. As mentioned in my last post, I still feel that a response to nth request may come before n-1 request.
So its truly a choice between these two designs
Design 1:
Flow 1
HttpInput -> AggregateControl -> Compute (has a loop to iterate n id) -> Http Asynchronous Request (backend service async call) -> AggregateRequest
Flow 2
Http Asynchronous Respone -> AggregateReply -> compute -> HttpReply
Design 2:
Flow 1
HttpInput -> Compute (has a loop to iterate n id & tag nth id by storing a flag in environement) -> Http Asynchronous Request (backend service async call)
Flow 2
Http Asynchronous Respone -> filter (to filter record to be sent to 2nd input terminal) -> CollectorNode (having 2 input terminal one with unlimitied quantity & other with 1)->Compute -> HttpReply
I went with design 1 because of the understanding that a response to nth request may come before n-1 request. |
|
| Back to top |
|
|
|
|
