| Author |
Message
|
| whydieanut |
 Posted: Wed Aug 24, 2011 2:49 am Post subject: Proposed Error Handling Flow Design |
 |
|
Disciple
Joined: 02 Apr 2010
Posts: 186
|
Hi all,
Like the millions of others here, I am looking at creating a reusable Error Handling Subflow.
I have various scenarios and different types of interfaces in my system and would like to reuse the Error Handler throughout the system. I am looking for critiques to my proposed flow.
I have read through quite a lot of articles, threads on this topic and here is what I could finally come up with, that could fulfill all my requirements.
Kindly let me know what you think of it.
I plan to have 2 sub flows - one for handling exceptions that happen in the downstream flow
and the other for failures (that might happen when the message fails in the Input node itself).
Exception Handler Subflow:
This will be connected to the Catch terminal of the Input node.
| Code: |
| InputNode -> ComputeNode (Process ExceptionList) -> TraceNode (Logs the exception) -> MQOutputNode (Puts the message into ERROR.Q) -> ThrowNode (To rollback changes) |
Failure Handler Subflow:
This will be connected to the Failure terminal of the Input node.
| Code: |
| InputNode -> FilterNode (Checks the Backout Count; If > 0 then discard the message) -> ComputeNode (Process ExceptionList) -> TraceNode (Logs the exception) -> MQOutputNode (Puts the message into FAILURE.Q) |
What I am expecting out of this is:
When an exception happens in the flow, after the Input node, I want the Exception Handler to deal with it and Throw at the end for Rollback.
When an error happens in the Input node itself (parse error perhaps, don't know what else could cause it), then I want the Failure Handler to deal with it.
But because of the Throw at the end of the (Catch) Exception Handler, the message goes to the Failure terminal of the Input node which is wired to the Failure Handler.
I don't want the Failure Handler to process such messages which are already dealt with in the (Catch) Exception Handler.
Hence I am checking the Backout Count to determine if the message has already been through the Input node earlier (and hence through the Catch terminal's Exception Handler)
Kindly let me know if there are any glaring mistakes in this design, and if so what better way to deal with this.
Also I'd like to know what are some good ways of logging errors. Here I have used the Trace nodes which I plan to replace with an MQOutput node or so.
In such a case since there are 2 messages being written to 2 queues (Exception message into Exception.Q and the actual message into ERROR.Q or FAILURE.Q),
should they be wired in sequence or is it better to use a Flow Order Node?
Thanking in anticipation. |
|
| Back to top |
|
 |
| lancelotlinc |
| Posted: Wed Aug 24, 2011 4:27 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Aug 24, 2011 4:40 am Post subject: Re: Proposed Error Handling Flow Design |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| whydieanut wrote: |
| Kindly let me know if there are any glaring mistakes in this design, and if so what better way to deal with this. |
Why have 2 subflows? Why differentiate between the error's origin point?
| whydieanut wrote: |
| Also I'd like to know what are some good ways of logging errors. Here I have used the Trace nodes which I plan to replace with an MQOutput node or so. |
Queues are not good places to log things. If you want to store the original failing message for later replay then obviously a queue, but the exception?
If you're Java literate, log4j is an option. If you need to persist the exceptions (e.g. for reporting or analysis purposes) then use a database. You can associate the message with the error by putting it in a column and extracting it for replay.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Aug 24, 2011 4:43 am Post subject: Re: Proposed Error Handling Flow Design |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| if you're Java literate, log4j is an option. If you need to persist the exceptions (e.g. for reporting or analysis purposes) then use a database. You can associate the message with the error by putting it in a column and extracting it for replay. |
Yes, and it's a reasonable design idea to write a single flow that handles this logging or inserting of data into a database, and have it read messages that come from a queue...
Rather than, for example, relying on a flow that's already failing to be able to establish a new database connection... |
|
| Back to top |
|
|
| whydieanut |
| Posted: Thu Aug 25, 2011 12:04 am Post subject: Re: Proposed Error Handling Flow Design |
|
|
Disciple
Joined: 02 Apr 2010
Posts: 186
|
| Vitor wrote: |
| Why have 2 subflows? Why differentiate between the error's origin point? |
If not, whats the other way around? Handle all Exceptions in the Failure flow itself? But then information in the ExceptionList tree is lost.
| Vitor wrote: |
| Queues are not good places to log things. If you want to store the original failing message for later replay then obviously a queue, but the exception? |
Agreed. How about logging the entire message in a Queue and storing the Exception info in a Log file/DB with a reference to the message in the Queue through MsgId?
| Vitor wrote: |
| If you're Java literate, log4j is an option. If you need to persist the exceptions (e.g. for reporting or analysis purposes) then use a database. You can associate the message with the error by putting it in a column and extracting it for replay. |
Unfortunatley, am not. Still will have a look at the option.
| mqjeff wrote: |
Yes, and it's a reasonable design idea to write a single flow that handles this logging or inserting of data into a database, and have it read messages that come from a queue...
Rather than, for example, relying on a flow that's already failing to be able to establish a new database connection... |
This again would make it necessary to write out the Exception Info into the queue for the flow to get error info, wouldn't it? |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Aug 25, 2011 4:07 am Post subject: Re: Proposed Error Handling Flow Design |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| whydieanut wrote: |
| Vitor wrote: |
| Why have 2 subflows? Why differentiate between the error's origin point? |
If not, whats the other way around? Handle all Exceptions in the Failure flow itself? But then information in the ExceptionList tree is lost. |
Tell me again the structural difference between a message leaving the Catch terminal of an input node, and a message leaving the Failure node of an input terminal.
| whydieanut wrote: |
| How about logging the entire message in a Queue and storing the Exception info in a Log file/DB with a reference to the message in the Queue through MsgId? |
How about it? Why do an expensive operation like finding a message in a potentially large queue by MsgId when you can store it in the DB and retrieve it and the error with a single operation? Via software designed to find specific items (which WMQ is not)
| whydieanut wrote: |
| Vitor wrote: |
| If you're Java literate, log4j is an option. If you need to persist the exceptions (e.g. for reporting or analysis purposes) then use a database. You can associate the message with the error by putting it in a column and extracting it for replay. |
Unfortunatley, am not. Still will have a look at the option. |
I should have been clearer. Using log4j requires Java. Using a database does not.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| whydieanut |
| Posted: Thu Aug 25, 2011 10:06 pm Post subject: |
|
|
Disciple
Joined: 02 Apr 2010
Posts: 186
|
| Vitor wrote: |
| Tell me again the structural difference between a message leaving the Catch terminal of an input node, and a message leaving the Failure node of an input terminal. |
You mean Failure terminal of the Input node.
In short Failure terminal flow loses out on the ExceptioList Tree info whereas the flow wired to the Catch terminal retains it.
Consider the following 2 scenarios:
1] Catch terminal wired to a common Error Handler, Failure terminal not wired.
1.1] Exception occurs in the flow connected to the Out terminal:
The message returns to the Input node, and is forwarded to the Catch terminal's Error Handler flow.
The ExceptionList Tree is preserved, and can be used to log the error.
1.2] Internal error occurs on the Input node:
Message is backed out to the Backout Queue/DLQ.
No information about the error (from the ExceptionList Tree) can be logged.
2] Catch terminal not wired. Failure terminal wired to Common Error Handler
2.1] Exception occurs in the flow connected to the Out terminal:
The message returns to the Input node, and is forwarded to the Failure terminal's Error Handler.
The ExceptionList Tree doesn't contain information about the errors that happened downstream.
2.2] Internal error occurs on the Input node:
The message is forwarded to the Failure terminal's Error Handler.
ExceptionList Tree contains information about the node's internal errors.
Given all this, the only way to record error data in case of Exceptions AND Input node Internal Errors seems to have flows connected to both terminals.
Correct me if I am wrong.
I'll come back to how to actually log the error after this 2 flows vs 1 flow thing is cleared. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Aug 26, 2011 4:09 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| whydieanut wrote: |
Given all this, the only way to record error data in case of Exceptions AND Input node Internal Errors seems to have flows connected to both terminals.
Correct me if I am wrong. |
You're quite correct. My point is why are these 2 different sub flows rather than 1?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| whydieanut |
| Posted: Sun Aug 28, 2011 11:38 pm Post subject: |
|
|
Disciple
Joined: 02 Apr 2010
Posts: 186
|
| Vitor wrote: |
| You're quite correct. |
Great! That's a relief! 
About the 2 separate flows,
I guess the functionality can be combined.
I wanted to give a provision to rollback a message if there was an exception in the main flow. So I have a compute node at the end of the Catch flow which serves as a conditional Rollback mechanism. That is, rollback if a particular flag is set in the Environment tree.
Additionally Exceptions downstream and Failures in the Input node mean 2 different kinds of error situations, so wanted to filter them out while logging.
I know this can still be combined into a single flow, but it'll add a bit more complexity.
IS there a compelling enough reason to combine these 2 flows? |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Aug 29, 2011 4:35 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| whydieanut wrote: |
| Additionally Exceptions downstream and Failures in the Input node mean 2 different kinds of error situations, so wanted to filter them out while logging. |
Why are these 2 kinds of error situation? That's what I don't get.
| whydieanut wrote: |
| I know this can still be combined into a single flow, but it'll add a bit more complexity. |
Is it more complex than having 2 near-identical flows?
| whydieanut wrote: |
| IS there a compelling enough reason to combine these 2 flows? |
I don't see your compelling reason to have them as 2 separate flows.
But if you want 2 flows, have 2 flows. You're writing and maintaining them, I'm not. It's your decision to make. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| whydieanut |
| Posted: Mon Aug 29, 2011 9:54 pm Post subject: |
|
|
Disciple
Joined: 02 Apr 2010
Posts: 186
|
Agree to all points. Will think over it then. Shouldn't be too difficult to combine anyways.
I was saying that they're 2 different kinds of errors because a message failing in the input node itself signals mostly that the message received is itself faulty, while any exception downstream COULD also indicate some error in message flow logic.
Anyways even then there are ways to filter them out with the common flow approach.
Coming to the error logging bit.
I have searched and searched and still not found a very good approach to doing it.
Here are my requirements:
I have a couple of different scenarios. I need my Error Handling mechanism to cater to them in a standard way AS FAR AS possible. Though I know it can't be generic enough to fulfill all the scenarios.
1] MQ/File to SAP and back
| Code: |
| MQ/File -> WMB -> SAP |
This will be used in a Batch processing environment.
The main requirement is just to transform the messages and move them to SAP with minimal processing involved.
SAP then processes the message and sends out a status message for each of the message it receives, back to WMB which needs to alert the originating app depending on the status received from SAP.
My priority is just to ensure that every message entering WMB be accounted for.
There's no business logic implemented in WMB, so I don't expect too many errors in that sense.
I was thinking of logging MQ messages with errors into Queues and File messages with errors into Files; So that replaying them is easier.
Now my problem is deciding where to log the Exception Information so that it is centralized and can be easily used to identify the faulty message in the Error Queue or Error File.
2] SAP to multiple MQ/File
| Code: |
| SAP -> WMB ->> MQ/File |
I'll come back to this later. |
|
| Back to top |
|
|
| j1 |
| Posted: Mon Apr 16, 2012 11:29 am Post subject: |
|
|
Centurion
Joined: 23 Jun 2003
Posts: 139
|
| Ive seen this requirement often enough to be surprised if there isnt any sample/template error handling pattern code that can be reused. is there? |
|
| Back to top |
|
|
| inMo |
| Posted: Mon Apr 23, 2012 10:59 am Post subject: |
|
|
Master
Joined: 27 Jun 2009
Posts: 216
Location: NY
|
I agree with Vitor that you really don't want to maintain two seperate error subflows. It just opens the door to maintenance and usage problems as time goes on. If possible you should keep that door shut for as long as possible.
That said, I also agree with you that you must recognize the difference in origin of the error. An error off a catch terminal is likely to receive similar treatment to an error off of the failure terminal, but not necessarily 100% the same.
I, apparently in the minority, also strongly endorse the idea that your error subflow should give forward thought as to how to take advantage of logging facilities to streamline, expedite and simplify root cause analysis. |
|
| Back to top |
|
|
| protocol |
| Posted: Thu Apr 26, 2012 6:37 am Post subject: |
|
|
Apprentice
Joined: 25 Feb 2005
Posts: 41
|
|
| Back to top |
|
|
| whydieanut |
| Posted: Mon Dec 17, 2012 10:32 pm Post subject: |
|
|
Disciple
Joined: 02 Apr 2010
Posts: 186
|
Sorry for reviving an old thread, but thought that it's better to keep all relevant discussions at a single place.
I had developed my Error handling subflow from all your recommendations.
This flow was supposed to handle errors in flows with an MQ Input node. The flow extracted the Exception info and wrote that into an error file and then put the entire original message into an error queue.
Now my application has grown to include File Input nodes as well as SOAP Input nodes. I now have to rethink the error handling subflow design so that i can be used in flows using any kind of input protocol.
Given this, does it make sense logging data from a potentially huge file into an MQ queue? Similarly, what about logging a web service call into a queue?
Is it better in this case to have each protocol be treated differently, using some kind of a configurable UDP, which tells my subflow what to do for each type of protocol?
Or is it better to keep things simple by logging all kinds of data into a Queue or File?
I know this is more of a design question, but still, any suggestions on what are best practices in such cases, with a common error handler for flows with different kinds of input data/protocol? |
|
| Back to top |
|
|
|
|
