Author |
Message
|
gbaddeley |
Posted: Tue Mar 13, 2018 4:50 pm Post subject: Retry Logic |
|
|
![](images/avatars/gallery/Star Wars/movie_star_wars_yoda.gif) Jedi
Joined: 25 Mar 2003 Posts: 2498 Location: Melbourne, Australia
|
Hi, I am not an experienced broker developer or administrator.
I would like to know if there are any commonly accepted patterns for designing message flows to properly handle retry, where there is an MQ input message and call to a http request / response service. If the http response is a failure that is not due to an error in the request data and the request needs to be retried, what are the best methods of handling this in the flow design?
Eg. An interface has 1,000's of input messages per day, but sometimes the http backend service is not available or times out, due to a planned outage (could be several hours) or an issue (that may last an indefinite period of time). The flow http request timeout is typically set to a few seconds. _________________ Glenn |
|
Back to top |
|
![](templates/DAJ_Glass/images/spacer.gif) |
fjb_saper |
Posted: Tue Mar 13, 2018 9:24 pm Post subject: |
|
|
![](images/avatars/gallery/misc/2.gif) Grand High Poobah
Joined: 18 Nov 2003 Posts: 20697 Location: LI,NY
|
If the outage is planned to last only a little time say less than 5 min you could choose the retry with timer nodes.
Otherwise you should have a back-out queue and the retry will be manual when the HTTP service is fixed...
![](images/smiles/innocent.gif) _________________ MQ & Broker admin |
|
Back to top |
|
![](templates/DAJ_Glass/images/spacer.gif) |
abhi_thri |
Posted: Wed Mar 14, 2018 1:37 am Post subject: |
|
|
![](images/avatars/3159673415d820d89ea89a.jpg) Knight
Joined: 17 Jul 2017 Posts: 516 Location: UK
|
|
Back to top |
|
![](templates/DAJ_Glass/images/spacer.gif) |
joebuckeye |
Posted: Wed Mar 14, 2018 5:32 am Post subject: |
|
|
![](images/avatars/19029967664f35568c654b0.jpg) Partisan
Joined: 24 Aug 2007 Posts: 364 Location: Columbus, OH
|
|
Back to top |
|
![](templates/DAJ_Glass/images/spacer.gif) |
gbaddeley |
Posted: Wed Mar 14, 2018 4:07 pm Post subject: |
|
|
![](images/avatars/gallery/Star Wars/movie_star_wars_yoda.gif) Jedi
Joined: 25 Mar 2003 Posts: 2498 Location: Melbourne, Australia
|
Thanks for the responses so far. I will look at them. The retry needs to be automatic, reliable and not require any manual intervention. _________________ Glenn |
|
Back to top |
|
![](templates/DAJ_Glass/images/spacer.gif) |
fjb_saper |
Posted: Wed Mar 14, 2018 7:54 pm Post subject: |
|
|
![](images/avatars/gallery/misc/2.gif) Grand High Poobah
Joined: 18 Nov 2003 Posts: 20697 Location: LI,NY
|
gbaddeley wrote: |
Thanks for the responses so far. I will look at them. The retry needs to be automatic, reliable and not require any manual intervention. |
At best you are looking at a Fatah Morgana...(mirage...)
How can you say that the retry needs to be automatic, reliable and not require manual intervention, when you don't know the reason why or root cause of the retry behavior. Say the target system crashed. So you are going to retry millions of messages waiting for it to be fixed, and by the way nobody notified them that they crashed... ![](images/smiles/innocent.gif) _________________ MQ & Broker admin |
|
Back to top |
|
![](templates/DAJ_Glass/images/spacer.gif) |
gbaddeley |
Posted: Thu Mar 15, 2018 3:49 pm Post subject: |
|
|
![](images/avatars/gallery/Star Wars/movie_star_wars_yoda.gif) Jedi
Joined: 25 Mar 2003 Posts: 2498 Location: Melbourne, Australia
|
fjb_saper wrote: |
Say the target system crashed. So you are going to retry millions of messages waiting for it to be fixed, and by the way nobody notified them that they crashed... ![](images/smiles/innocent.gif) |
Sorry, I am assuming that the target systems have their own monitoring and alerting, and they will actively fix issues to restore service.
On the broker side, we have alerting for MQ queue depth thresholds and messages aging, but we wouldn't be taking any action, other than checking that the target system app support team is working on the issue. _________________ Glenn |
|
Back to top |
|
![](templates/DAJ_Glass/images/spacer.gif) |
fjb_saper |
Posted: Thu Mar 15, 2018 8:19 pm Post subject: |
|
|
![](images/avatars/gallery/misc/2.gif) Grand High Poobah
Joined: 18 Nov 2003 Posts: 20697 Location: LI,NY
|
gbaddeley wrote: |
fjb_saper wrote: |
Say the target system crashed. So you are going to retry millions of messages waiting for it to be fixed, and by the way nobody notified them that they crashed... ![](images/smiles/innocent.gif) |
Sorry, I am assuming that the target systems have their own monitoring and alerting, and they will actively fix issues to restore service.
On the broker side, we have alerting for MQ queue depth thresholds and messages aging, but we wouldn't be taking any action, other than checking that the target system app support team is working on the issue. |
Your assumptions in god's ear. Alas my experience tells me otherwise.
We had a call to a .NET service (over MQ) and MQ was the first to notice deterioration in the response time when there was a problem. Sometimes all they did was recycle their .NET servers in a rolling fashion, and the world was ok again... ![](images/smiles/innocent.gif) _________________ MQ & Broker admin |
|
Back to top |
|
![](templates/DAJ_Glass/images/spacer.gif) |
Vitor |
Posted: Fri Mar 16, 2018 5:04 am Post subject: |
|
|
![](images/avatars/905340816499abc1cdfb07.jpg) Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
gbaddeley wrote: |
Sorry, I am assuming that the target systems have their own monitoring and alerting, and they will actively fix issues to restore service. |
Hah!
![](images/smiles/sign10.gif) _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
![](templates/DAJ_Glass/images/spacer.gif) |
PeterPotkay |
Posted: Fri Mar 16, 2018 4:14 pm Post subject: Re: Retry Logic |
|
|
![](images/avatars/c7de043e3ce24eaa279a5.gif) Poobah
Joined: 15 May 2001 Posts: 7716
|
gbaddeley wrote: |
Hi, I am not an experienced broker developer or administrator. |
Neither am I. I struggle even qualifying as inexperienced.
gbaddeley wrote: |
I would like to know if there are any commonly accepted patterns for designing message flows to properly handle retry, where there is an MQ input message and call to a http request / response service. |
I'm an anonymous moron on the internet, so I can offer a smug answer: Why is this your problem? You are just a proxy for the service provider, that is having issues, that only allows a synchronous call (http), that has no answer for you, and offer no mechanism for processing your request later. "Sorry, no one's home, come back another day." I don't think its your obligation to attempt make this "better".
Just proxy the bad news right back.
The front end waits "x" seconds for your MQ reply.
At no later than x-1 seconds you should send them a reply: either a success, or details explaining the http backend is N/A - ya, know, the same thing they would be getting if they were calling the backend directly themselves. _________________ Peter Potkay
Keep Calm and MQ On |
|
Back to top |
|
![](templates/DAJ_Glass/images/spacer.gif) |
|