| Author |
Message
|
| Bichu |
 Posted: Wed Oct 04, 2017 11:49 pm Post subject: HTTP webservice retry without losing the message order |
 |
|
Centurion
Joined: 16 Oct 2011
Posts: 124
Location: London
|
Hi Guys,
Need a help. I have a requirement where i received messages from a source, do some processing and need to POST data to destination.
I need to maintain the order of messages as well. My flow will be like
| Code: |
MQ input - Compute - HTTP request
Failure will go to error q where retry message flow needs to be designed
success will got to audit log |
The problem is when the destination web service is not accessible, i can have a retry mechanism for retry sending the message to the destination.
But once the webservice comes available, both normal as well as backed out queues will be processed and I will lose the order.
How can I maintain the order here. Any ideas please. |
|
| Back to top |
|
 |
| joebuckeye |
| Posted: Thu Oct 05, 2017 4:34 am Post subject: Re: HTTP webservice retry without losing the message order |
|
|
Partisan
Joined: 24 Aug 2007
Posts: 365
Location: Columbus, OH
|
| Bichu wrote: |
| How can I maintain the order here. Any ideas please. |
Wow, message affinity (ie order) and retry in one solution? That will be some ugly, fragile code that will cause headaches for years ahead. Bad design.
But lets think about what sort of Rube Goldberg type solution would work here.
So the retry queue needs to be processed first before any message can be taken from the actual input queue to maintain order.
But even then your retry queue has to be processed in order also. So if you retry a call and it still fails then it needs to go to the front of the retry queue and not just placed back on the retry queue because then it would be out of order.
Interesting dilemma.
Of course any solution you find would also have to run in a single instance so I hope performance is not important. |
|
| Back to top |
|
|
| Bichu |
| Posted: Thu Oct 05, 2017 5:03 am Post subject: |
|
|
Centurion
Joined: 16 Oct 2011
Posts: 124
Location: London
|
| Otherwise I need to compromise with the order of message... |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Oct 05, 2017 5:06 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Bichu wrote: |
| Otherwise I need to compromise with the order of message... |
That's not a compromise.
Message affinity (where messages have to be processed in a specific order) is a bad thing as you've discovered and as my associate has correctly pointed out. "Compromising" on this would be an improvement.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Bichu |
| Posted: Thu Oct 05, 2017 5:16 am Post subject: |
|
|
Centurion
Joined: 16 Oct 2011
Posts: 124
Location: London
|
Hi Vitor,
We have a requirement where an incident created needs to go first before incident update goes..hence we need to maintain the order... |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Oct 05, 2017 5:24 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Bichu wrote: |
Hi Vitor,
We have a requirement where an incident created needs to go first before incident update goes..hence we need to maintain the order... |
You might want to have a look at how SAP solves this dilemma when applied to IDocs... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Oct 05, 2017 5:25 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Bichu wrote: |
| We have a requirement where an incident created needs to go first before incident update goes..hence we need to maintain the order... |
No you don't.
If an update arrives for a non-existent incident, you can easily create a skeleton incident (assuming that the update doesn't contain all the information in the create). You can then update the skeleton when the create arrives.
You obviously need some logic so that the create doesn't overwrite the information contained in the update, and some governance to ensure there are no skeletons left "orphaned" because the create never actually arrived.
But I put it to you that this kind of code is more robust and easier to maintain than the kind of (to borrow my associate's descriptive phrase) Rube Goldberg code you're going to end up with trying to mix retry logic with maintaining order.
You'll also be able to scale it by using multiple instances.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| joebuckeye |
| Posted: Thu Oct 05, 2017 7:06 am Post subject: |
|
|
Partisan
Joined: 24 Aug 2007
Posts: 365
Location: Columbus, OH
|
| Vitor wrote: |
| Bichu wrote: |
| We have a requirement where an incident created needs to go first before incident update goes..hence we need to maintain the order... |
No you don't.
If an update arrives for a non-existent incident, you can easily create a skeleton incident (assuming that the update doesn't contain all the information in the create). You can then update the skeleton when the create arrives.
You obviously need some logic so that the create doesn't overwrite the information contained in the update, and some governance to ensure there are no skeletons left "orphaned" because the create never actually arrived.
But I put it to you that this kind of code is more robust and easier to maintain than the kind of (to borrow my associate's descriptive phrase) Rube Goldberg code you're going to end up with trying to mix retry logic with maintaining order.
You'll also be able to scale it by using multiple instances. |
Or if your update fails because it is processed before the create (for whatever reason), just throw the failed updates into a retry queue.
There could still be an issue of multiple updates for the same entry however. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Oct 05, 2017 7:24 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| joebuckeye wrote: |
| Vitor wrote: |
| Bichu wrote: |
| We have a requirement where an incident created needs to go first before incident update goes..hence we need to maintain the order... |
No you don't.
If an update arrives for a non-existent incident, you can easily create a skeleton incident (assuming that the update doesn't contain all the information in the create). You can then update the skeleton when the create arrives.
You obviously need some logic so that the create doesn't overwrite the information contained in the update, and some governance to ensure there are no skeletons left "orphaned" because the create never actually arrived.
But I put it to you that this kind of code is more robust and easier to maintain than the kind of (to borrow my associate's descriptive phrase) Rube Goldberg code you're going to end up with trying to mix retry logic with maintaining order.
You'll also be able to scale it by using multiple instances. |
Or if your update fails because it is processed before the create (for whatever reason), just throw the failed updates into a retry queue.
There could still be an issue of multiple updates for the same entry however. |
The cat can be skinned in a number of different ways according to personal taste.
I was attempting to highlight that most "requirements" for strict message order wilt under scrutiny and the actual requirement (in this case an incident which includes updated information) can be met in other ways.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Bichu |
| Posted: Thu Oct 05, 2017 7:49 am Post subject: |
|
|
Centurion
Joined: 16 Oct 2011
Posts: 124
Location: London
|
Thanks.
I thought of a design where if an http post to the backend fails, message will be written to the backout queue after backout threshold. Along with this, the particular message flow alone will be stopped. Another message flow which tries retrying from the backout queue will ensure connectivity check to the backend. If the connection works, the output should start the stopped message flow previously.
Now critics please!!  |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Oct 05, 2017 7:59 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Bichu wrote: |
| Now critics please!! |
That's a lot of moving parts.
I'm never a fan of flows which have administrative authority to start and stop other flows. What's to stop another (possibly malicious) flow using the same technique?
Consider the scenario where someone's trying to perform maintenance on the server and/or the broker. The endpoint shuts down, causing messages on the retry queue before close down. Once maintenance is finished, the admin innocently starts all the flows and both first time and retry messages flow mixed up. And don't tell me that can't happen because of procedures and maintenance windows. Procedures get overlooked and hardware fails.
That's just what immediately comes to mind. You've designed a Rube Goldberg machine where the bowling ball falls on the switch to start the fan that blows the model boat with the pool queue towards the switch on the table lamp. Not only is there an easier way to switch on a table lamp, sooner or later the bowling ball will crush the fan or the pool cue will knock the table lamp over.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Sun Oct 08, 2017 4:04 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| Bichu wrote: |
Thanks.
I thought of a design where if an http post to the backend fails, message will be written to the backout queue after backout threshold. Along with this, the particular message flow alone will be stopped. Another message flow which tries retrying from the backout queue will ensure connectivity check to the backend. If the connection works, the output should start the stopped message flow previously.
Now critics please!! |
Interesting design.
Another technique is to GET INHIBIT the input queue, so that the main flow cannot consume messages until after all the retry messages have been processed, and then the retry flow sets the main input queue to GET ENABLED.
There is another very simple solution that makes it clear to all parties : "IIB / MQ does not guarantee message order". This moves the responsibility to the backend application, to marshall the requests and process them into the required order. It will need logic to cater for missing and duplicate requests.
_________________
Glenn |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Sun Oct 08, 2017 4:45 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Bichu,
You assume the messages in your input queue are in the "correct" order. Irrespective of what you end up doing in your flow, will it handle the scenarios where the messages in the queue are not in the "correct" order? How ill you know they are not in the queue in the right order?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Oct 09, 2017 1:12 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
I once saw a very complex design with "retry" and "holding" queues set up.
All they actually needed was a means to stop the flow until the backend service was restored. MQ could do all the rest (keeping everything safe and in order) on the single input queue.
If no useful work can be done with the back-end down, then why do anything other than keep retrying (with a sleep interval), or stop the flow/inhibit the queue until it the backend is restored?
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Oct 09, 2017 4:10 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| zpat wrote: |
I once saw a very complex design with "retry" and "holding" queues set up.
All they actually needed was a means to stop the flow until the backend service was restored. MQ could do all the rest (keeping everything safe and in order) on the single input queue.
If no useful work can be done with the back-end down, then why do anything other than keep retrying (with a sleep interval), or stop the flow/inhibit the queue until it the backend is restored? |
It's possible that the flow is talking to more than one actual endpoint - webserver, etc - for different customers/bps/eggs/whatever.
One would then want to push messages to another place when one endpoint was down, and then reprocess them in order once that endpoint was back up.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
|
|
