| Author |
Message
|
| gbaddeley |
 Posted: Tue Mar 13, 2018 4:50 pm Post subject: Retry Logic |
 |
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
Hi, I am not an experienced broker developer or administrator.
I would like to know if there are any commonly accepted patterns for designing message flows to properly handle retry, where there is an MQ input message and call to a http request / response service. If the http response is a failure that is not due to an error in the request data and the request needs to be retried, what are the best methods of handling this in the flow design?
Eg. An interface has 1,000's of input messages per day, but sometimes the http backend service is not available or times out, due to a planned outage (could be several hours) or an issue (that may last an indefinite period of time). The flow http request timeout is typically set to a few seconds.
_________________
Glenn |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Tue Mar 13, 2018 9:24 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
If the outage is planned to last only a little time say less than 5 min you could choose the retry with timer nodes.
Otherwise you should have a back-out queue and the retry will be manual when the HTTP service is fixed...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| abhi_thri |
| Posted: Wed Mar 14, 2018 1:37 am Post subject: |
|
|
Knight
Joined: 17 Jul 2017
Posts: 516
Location: UK
|
|
| Back to top |
|
|
| joebuckeye |
| Posted: Wed Mar 14, 2018 5:32 am Post subject: |
|
|
Partisan
Joined: 24 Aug 2007
Posts: 365
Location: Columbus, OH
|
|
| Back to top |
|
|
| gbaddeley |
| Posted: Wed Mar 14, 2018 4:07 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
Thanks for the responses so far. I will look at them. The retry needs to be automatic, reliable and not require any manual intervention.
_________________
Glenn |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Mar 14, 2018 7:54 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| gbaddeley wrote: |
| Thanks for the responses so far. I will look at them. The retry needs to be automatic, reliable and not require any manual intervention. |
At best you are looking at a Fatah Morgana...(mirage...)
How can you say that the retry needs to be automatic, reliable and not require manual intervention, when you don't know the reason why or root cause of the retry behavior. Say the target system crashed. So you are going to retry millions of messages waiting for it to be fixed, and by the way nobody notified them that they crashed...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Thu Mar 15, 2018 3:49 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| fjb_saper wrote: |
| Say the target system crashed. So you are going to retry millions of messages waiting for it to be fixed, and by the way nobody notified them that they crashed... |
Sorry, I am assuming that the target systems have their own monitoring and alerting, and they will actively fix issues to restore service.
On the broker side, we have alerting for MQ queue depth thresholds and messages aging, but we wouldn't be taking any action, other than checking that the target system app support team is working on the issue.
_________________
Glenn |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Mar 15, 2018 8:19 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| gbaddeley wrote: |
| fjb_saper wrote: |
| Say the target system crashed. So you are going to retry millions of messages waiting for it to be fixed, and by the way nobody notified them that they crashed... |
Sorry, I am assuming that the target systems have their own monitoring and alerting, and they will actively fix issues to restore service.
On the broker side, we have alerting for MQ queue depth thresholds and messages aging, but we wouldn't be taking any action, other than checking that the target system app support team is working on the issue. |
Your assumptions in god's ear. Alas my experience tells me otherwise.
We had a call to a .NET service (over MQ) and MQ was the first to notice deterioration in the response time when there was a problem. Sometimes all they did was recycle their .NET servers in a rolling fashion, and the world was ok again...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Mar 16, 2018 5:04 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| gbaddeley wrote: |
| Sorry, I am assuming that the target systems have their own monitoring and alerting, and they will actively fix issues to restore service. |
Hah!
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Mar 16, 2018 4:14 pm Post subject: Re: Retry Logic |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| gbaddeley wrote: |
| Hi, I am not an experienced broker developer or administrator. |
Neither am I. I struggle even qualifying as inexperienced.
| gbaddeley wrote: |
I would like to know if there are any commonly accepted patterns for designing message flows to properly handle retry, where there is an MQ input message and call to a http request / response service. |
I'm an anonymous moron on the internet, so I can offer a smug answer: Why is this your problem? You are just a proxy for the service provider, that is having issues, that only allows a synchronous call (http), that has no answer for you, and offer no mechanism for processing your request later. "Sorry, no one's home, come back another day." I don't think its your obligation to attempt make this "better".
Just proxy the bad news right back.
The front end waits "x" seconds for your MQ reply.
At no later than x-1 seconds you should send them a reply: either a success, or details explaining the http backend is N/A - ya, know, the same thing they would be getting if they were calling the backend directly themselves.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
|
|
