| Author |
Message
|
| Vitor |
 Posted: Wed Mar 26, 2014 5:10 am Post subject: |
 |
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| christo wrote: |
| But after 30 seconds external system respose may come ,if it comes i have to use that resposne and do something . |
Why? Given that the requestor has timed out and is no longer expecting that response, why are you processing it?
As both my associate and I have said, this is a design problem not a broker problem.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
 |
| christo |
| Posted: Wed Mar 26, 2014 10:59 pm Post subject: |
|
|
Newbie
Joined: 23 Mar 2014
Posts: 8
|
| Vitor wrote: |
Why? Given that the requestor has timed out and is no longer expecting that response, why are you processing it?
As both my associate and I have said, this is a design problem not a broker problem. |
We have a bussiness requirement like this. We need to process the late response from external system. So we need to have mqinput node for the external system rather an Mqget node(i believe ,with MQget node i cant achieve this). |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Mar 27, 2014 4:34 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| christo wrote: |
| We have a bussiness requirement like this. We need to process the late response from external system. |
Then you need a better design, or a better requirement.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Tibor |
| Posted: Thu Mar 27, 2014 4:53 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
| christo wrote: |
| We need to process the late response from external system. |
What is the purpose of processing this late response messages? It seems a little bit strange if the requester application is able to handle this situation... |
|
| Back to top |
|
|
| satya2481 |
| Posted: Thu Mar 27, 2014 4:02 pm Post subject: |
|
|
Disciple
Joined: 26 Apr 2007
Posts: 170
Location: Bengaluru
|
Thank you all for your responses. As Christo mentioned in his all conversations its the project requirement. The requirement is as below. If our design does not look correct then suggest what is the best option here. We need to support high volume transactions. Hence we have ended up with this design and implementation. Which will work fine if we use separate queue for each of the timeout component.
Below I am giving the complete requirement, see if anyone can suggest a better design. You may require lots of patience to read it 
1. Request message comes with 50 Employee Numbers in single message.
2. This message needs to be split into 50 individual messages as back end application has a limitation. It can not process multiple employee numbers.
3. Collect all the responses from Backend application1 and group them in to 10 Employee numbers as one message.
4. Send the request to one more Backend application and wait for response. Once the response is received ensure all other Employee numbers got processed or not.
5. Collect all the Employee details responses from Backend application 2 and create a single response message and send to consumer.
Since we have complex orchestration instead of waiting for the thread to complete all these activities we have divided the work into several flows as given below.
Flow 1 : Receives request from consumer and splits the message
Flow 2 : Receive the request from Flow 1 and makes a call to Back end application 1 (This is having HTTP call hence single flow handles both request and response). Collect all the responses and trigger next flow
Flow 3 : Transform the request message into Backend application 2 format and send the request
Flow 4 : Receive the response from Backend application 2. Cross check if all the Employee number responses received or not.
Flow 5 : Once it is ensured all the Employee Number details are received trigger this flow. Which will create a single response back to consumer.
Here there are several scenario where in timeout scenario needs to be handled. Over all timeout starting from Flow1 and timeout during interaction with External application.
_________________
IBM Certified Solution Developer WebSphere Message Broker V6.0
IBM Certified System Administrator WebSphere MQ V6.0 |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Mar 28, 2014 4:49 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
It's still not clear from this very well put together explaination why you would process late responses from the back end system if the requester (Step 1) has timed out, or how you can if the requester has timed out.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| satya2481 |
| Posted: Fri Mar 28, 2014 7:22 am Post subject: |
|
|
Disciple
Joined: 26 Apr 2007
Posts: 170
Location: Bengaluru
|
Hi Vitor,
We are not having control over the Back End system response messages. Hence if the flow timeout happens and the responses received after timeout period we will ignore them after reading from queue.
From consumer perspective they want a response, hence flow is timing out and sending response back. So we have to timeout our flow after some time. If we have such requirement then we have to provide a solution to such requirement, this is what I believe.
Integration is between systems and should be capable to satisfy both parties....!!! |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Mar 28, 2014 8:20 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| satya2481 wrote: |
Hi Vitor,
We are not having control over the Back End system response messages. Hence if the flow timeout happens and the responses received after timeout period we will ignore them after reading from queue. |
So set expiry
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| satya2481 |
| Posted: Sun Mar 30, 2014 4:21 pm Post subject: |
|
|
Disciple
Joined: 26 Apr 2007
Posts: 170
Location: Bengaluru
|
| We are not having control over the backend system MQMD.Expiry values. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Mar 30, 2014 10:24 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| satya2481 wrote: |
| We are not having control over the backend system MQMD.Expiry values. |
Sure you do. Have the responses come to queue A, be assigned an expiry time and put to queue B (the real response queue for the flow...) ... Easy, .... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sun Mar 30, 2014 11:04 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
The topic discussed in this thread are a great example of the problems trying to cobble together a solution in a transactional manner when there is a great bit of the data flow being done in a non transactional manner. i.e. the HTTP part.
Many designers don't give this key area any thought at all. They are simply not aware of how fragile anything using HTTP is. No transactionality, No builtin retries. The same goes for SOAP and TCP/IP nodes as well.
I have design reviewed many flows using TCP/IP over the years and it is safe to say, that 99% of them no not have any retry handling. Do the designers/coders not understand now often even a LAN can have a 'glitch'. SOD's law will dictate that this will happen just as an http request is issued. Come back even 100mSecs later and all is fine.
Question: How many people reading this post have coded a flow using HTTP/SOAP that does a defined number of retries when it gets a '404' response. The problem might be transitory for example a cluster failing over and within a few seconds the endpoint is up again and ready for business. Even retrying once would help. Good defensive programming or just common sense? I don't know, you decide.
There has been a lot of discussion here about MQ Message expiry but IMHO, the real weak point is the HTTP call in the middle.
Enough of my soap box rant for one day. This is probably best discussed in another thread...
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Mar 31, 2014 1:04 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Completely agree.
This sort of poor coding is caused by (1) inexperience (2) a culture of just getting the code out the door quickly and saying "I have finished" (3) a culture of being optimistic instead of realistic.
Assume the worst may happen, and your code will work predictably under all situations. Assume the best will always happen and your code will fail on bad days.
.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| satya2481 |
| Posted: Mon Mar 31, 2014 8:38 am Post subject: |
|
|
Disciple
Joined: 26 Apr 2007
Posts: 170
Location: Bengaluru
|
Hi smdavies99 and zpat,
Thank you for sharing your views. However we are still not got the required solution or better design to meet the requirements explained in our previous replies.
| Quote: |
| Have the responses come to queue A, be assigned an expiry time and put to queue B (the real response queue for the flow...) ... Easy |
I do not think this is correct way to handle timeout scenario. Unnecessarily introducing a flow just to update MQMD.Expiry? What happens if any exception happens in this flow. One more component to track.
At present we are trying to add multiple queues for each timeout value and browse all of them using single flow in sequence. But do we have any better solution which is kind of a standard case for such timeout requirements ? |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Mar 31, 2014 9:05 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| satya2481 wrote: |
Unnecessarily introducing a flow just to update MQMD.Expiry? What happens if any exception happens in this flow. One more component to track.
|
A simple flow that reads the message as a BLOB, updates the MQMD.Expiry in a compute node and sends the message on its way with a MQOutput node will hardly bew a cause for concern (IMHO)
I really think you need to go back and revisit the design and point out all the weaknesses that have been discussed here and come up with a better design.
If you have to go with the flawed design at least you can dicument all the weaknesses so that when it fails in production you can quietly say 'I thought that might happen'.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Mar 31, 2014 9:10 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| satya2481 wrote: |
| I do not think this is correct way to handle timeout scenario. Unnecessarily introducing a flow just to update MQMD.Expiry? |
No, the correct way is to have the downstream applicaiton act like they're in a time limited limited request/response.
| satya2481 wrote: |
| What happens if any exception happens in this flow. One more component to track. |
Same thing as if the timeout in your original design goes wrong.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
