| Author |
Message
|
| ollyc |
 Posted: Fri Dec 04, 2009 7:23 am Post subject: Error Handling with multiple output messages |
 |
|
Apprentice
Joined: 04 Dec 2009
Posts: 27
Location: England
|
Hi,
We're new to WMB and have flows that read a single message from a queue and then break it up into multiple output messages or write to multiple targets.
When an error occurs we want to capture the input data + error message and write it to an error queue - all outputs that have been written out should be rolled back.
If the error handling worked ok, the input message should be removed from the source queue and shouldn't be written to a backout queue.
Currently it looks like we need to:
From the input node catch terminal, write to the error queue using a transaction scope of 'no'.
Then set a shared variable flag to indicate that the error has been handled
Then throw another error (to rollback the outputs)
Then in the failure terminal find the relevant shared variable flag (so that we know that the write to the error queue has succeeded), check it and delete it.
If the shared variable flag is not found (error occurred writing to the error queue) then do another throw to send the message to a back out queue or let MQ handle the problem
This seems pretty complicated - is there an easier / cleaner way to do this?
thanks,
Olly. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Fri Dec 04, 2009 7:32 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
DO NOT USE THE FAILURE TERMINAL.
If you want discard the message, then make it non-persistent and set the BOTHRESH of the queue to 0.
Then when you throw the exception out of the catch terminal, the message will be discarded because you have said it is okay.
If the message is persistent instead of non-persistent, it will get moved to the DLQ. |
|
| Back to top |
|
|
| ollyc |
| Posted: Fri Dec 04, 2009 7:38 am Post subject: |
|
|
Apprentice
Joined: 04 Dec 2009
Posts: 27
Location: England
|
Wow - quick response!
If I don't put anything on the failure terminal, how do I know that the write to the error queue worked?
i.e. if the error queue write fails then the message is pretty much lost unless I handle this. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Dec 04, 2009 7:50 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You know the write to the error queue worked because you didn't get an exception from the MQOutput node.
You just need to make sure you set the right transactionality. |
|
| Back to top |
|
|
| ollyc |
| Posted: Fri Dec 04, 2009 8:12 am Post subject: |
|
|
Apprentice
Joined: 04 Dec 2009
Posts: 27
Location: England
|
| Quote: |
| You know the write to the error queue worked because you didn't get an exception from the MQOutput node. |
This doesn't help - I need to do a throw at the end of the catch terminal process in order to rollback my multiple outputs.
I could write the original message to a backout queue after both types of error (i.e. when the error queue failed and when error queue write was successful) but I'd need to filter out the two types of error - possibly with another process that reads from the error queue and deletes the original message from the backout queue - but it's getting messy again and race conditions can occur (because writing to the error queue is outside transaction scope....) |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Dec 04, 2009 9:26 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| ollyc wrote: |
| This doesn't help - I need to do a throw at the end of the catch terminal process in order to rollback my multiple outputs. |
That's why I said
| mqjeff wrote: |
| You just need to make sure you set the right transactionality. |
What you want is eminently doable, frequently done, and straight forward.
And never requires you to connect anything to the failure terminal of the Input node. |
|
| Back to top |
|
|
| ollyc |
| Posted: Fri Dec 04, 2009 9:32 am Post subject: |
|
|
Apprentice
Joined: 04 Dec 2009
Posts: 27
Location: England
|
Sorry, can you spell it out for me, given the requirements I have - what is the right transactionality? 
As far as I can see the data input and outputs should all be "transaction scope - yes" - because I don't want to lose messages if everything crashes.
The scope of my error queue should be "transaction scope - no" because it needs to survive the rollback of my multiple targets if I get a bad message.
cheers,
Olly. |
|
| Back to top |
|
|
| ollyc |
| Posted: Mon Dec 07, 2009 3:55 am Post subject: |
|
|
Apprentice
Joined: 04 Dec 2009
Posts: 27
Location: England
|
The conclusion i'm drawing from this is that;
a. if you don't mind losing a few messages then error handling can be simple - don't need to worry about persistance, transactionality or checking writes to your error queue.
b. If losing messages is a problem (and you've got multiple messages out and want to capture the reason for the error) then error handling is complicated and messy - the current solution we have is as good as it gets.
I'll raise an enhancement request to see if anything can be developed, e.g. a 'rollback targets node' or some form of nested transactionality.
Cheers
Olly. |
|
| Back to top |
|
|
| mqseries0209 |
| Posted: Mon Dec 07, 2009 6:56 am Post subject: use try catch nodes |
|
|
Voyager
Joined: 30 Mar 2006
Posts: 90
|
I think you can use additional try and catch nodes with the catch terminal path and take care of differentiating errors (Write errors, throw errors ... etc) ...
_________________
IBM Certified Solution Developer - WebSphere Message Broker V6.1
IBM Certified Solution Developer - WebSphere Integration Developer V6.0 |
|
| Back to top |
|
|
| Herbert |
| Posted: Mon Dec 07, 2009 7:10 am Post subject: Re: Error Handling with multiple output messages |
|
|
Centurion
Joined: 05 Dec 2006
Posts: 146
Location: Leersum, The Netherlands
|
| ollyc wrote: |
From the input node catch terminal, write to the error queue using a transaction scope of 'no'.
Then set a shared variable flag to indicate that the error has been handled
Then throw another error (to rollback the outputs)
Then in the failure terminal find the relevant shared variable flag (so that we know that the write to the error queue has succeeded), check it and delete it.
If the shared variable flag is not found (error occurred writing to the error queue) then do another throw to send the message to a back out queue or let MQ handle the problem |
Use the Failure terminal to write to the ERROR queue. Then there is no need to communicate between Catch and Failure with flags (Maybe not possible at all. The Failure terminal starts with a clean Environment). The last thing in your Catch handling must indeed be a Throw.
If you want to handle failures for writing to the ERROR queue use the MQ backout config for that (BOTHRESH and BOQNAME)
With this scenario you are also save with the situation that a message can go directly to the Failure terminal without handled by the Catch Terminal before. |
|
| Back to top |
|
|
| ollyc |
| Posted: Mon Dec 07, 2009 7:13 am Post subject: |
|
|
Apprentice
Joined: 04 Dec 2009
Posts: 27
Location: England
|
thanks for the response - I've tried adding a try/catch node, but this doesn't rollback any successful MQ writes or DB updates.
To rollback those previous writes and updates, I need to go down the failure terminal. Once down the failure terminal, I need a way to distinguish between an handled and unhandled error (currently I'm using a shared row).
cheers,
Olly. |
|
| Back to top |
|
|
| Herbert |
| Posted: Mon Dec 07, 2009 7:45 am Post subject: |
|
|
Centurion
Joined: 05 Dec 2006
Posts: 146
Location: Leersum, The Netherlands
|
| ollyc wrote: |
| I need a way to distinguish between an handled and unhandled error (currently I'm using a shared row). |
A shared row is wrong, think about multi-instance etc. Use a other design where this check is not needed.
However, I have seen an implementation in the past where the backout count was used for this, the error handling was in a subflow. From my head, it was something like below:
- Catch terminal
1) error_handling_subflow
2) throw
- Failure terminal
1) FlowOrder
2) - First, FilterNode to check the backout count, when 0 then invoke the error_handling_subflow
3) - Second, write mesasge to ERROR queue.
The idea was that a message was always put on the ERROR queue and that the error_handling_subflow was only executed once. Backout threshold and backout Queue was not used in this setup. ( so only one try, if writing to the ERROR queue did fail the messages went to the DLQ ).
AFAIK it did work that way. For messages that did go tru both Catch and Failure and for messages that did go directly to Failure. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Dec 07, 2009 7:56 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
No, you do NOT need to go down the failure terminal.
As I said.
If you want to make a decision based on whether the MQOutput node that stores your log message was successful or not, you can put a try/catch node before that MQOutput node.
If you want to roll back outputs that are in the same transaction as the input, then you need to throw an exception that gets handled by the input node.
You can throw that exception on the CATCH leg of a try/catch node.
You can wire a try/catch node down the CATCH leg of the Input node.
YOU DO NOT NEED TO WIRE THE FAILURE TERMINAL OF THE INPUT NODE. |
|
| Back to top |
|
|
| ollyc |
| Posted: Mon Dec 07, 2009 8:25 am Post subject: |
|
|
Apprentice
Joined: 04 Dec 2009
Posts: 27
Location: England
|
1. we need to put the message to the failure terminal - to roll back the outputs.
2. I don't want my DLQ / backout queue filling up with handled errors - I get many of these a day (input messages for handled errors should only appear on the error queue - that's what it's there for).
3. I can't set the input node to non-persistant, non-transactional or anything else that risks losing messages if the server dies.
with the requirements above, I still can't see anyway of avoiding handled error messages appearing on a DLQ or backout q along with serious errors (where the flow has gone very bad) - unless I wire up the failure terminal....
Herbert - I've looked at the backout count, but this won't work for stuff like JMS (AFAIK). Shared row is ok as we store the Msg ID with the flag that indicates whether the write was successful so we can run multi-threaded. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Dec 07, 2009 8:59 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You appear to have conflicting requirements on the message retention.
It's required that you keep the message until you've tried it, regardless of whether the qmgr restarts or etc.
It's entirely useless once you've tried it at least once.
You can still use NP messages if you are using a supported version of MQ. You can configure the input queue to retain non-persistent messages on queue manager restart.
If you don't want an BOQ filling up, then deploy another flow that reads the BOQ and throws all messages away.... MQInput-->(nothing) will do the trick just fine. |
|
| Back to top |
|
|
|
|
