| Author |
Message
|
| vmcgloin |
 Posted: Mon Dec 07, 2009 9:12 am Post subject: |
 |
|
Knight
Joined: 04 Apr 2002
Posts: 560
Location: Scotland
|
Ollyc - you need to take a step back and listen to Jeff and others that have replied. The rebuttals to all of your points have already been covered.
http://publib.boulder.ibm.com/infocenter/wmbhelp/v6r1m0/topic/com.ibm.etools.mft.doc/ac00410_.htm
http://publib.boulder.ibm.com/infocenter/wmbhelp/v6r1m0/topic/com.ibm.etools.mft.doc/ac00414_.htm
1. You can roll back the outputs by connecting the catch terminal just as easily.
2. Your backout queue can be whatever you want it to be (it could be the same as your error queue - or better a dedicated backout queue) and allows you to keep the input message in a failure situation - for diagnosis and/or replay.
3. You state you "want to capture the input data + error message and write it to an error queue" but not a backout queue - does that mean you don't want the input messages in their unaltered state but that you are instead going to add some sort of error header to them? You don't lose the reliability by implementing proper error handling - far from it.
4. Backout count certainly does work with JMS messages.
I do not understand why you need to make a distiction between handled and unhandled errors - just write your code to handle them. If all else fails (e.g. the error queue is full) then the correct configuration and transactionality will allow MQ to let you know and write those messages to the DLQ instead.
My only other advice is that if you are going to this effort for specific flows then make it generic enough that you can use your error handling subflow for all (or most of) your message flows.
Cheers,
Vicky |
|
| Back to top |
|
 |
| ollyc |
| Posted: Mon Dec 07, 2009 9:25 am Post subject: |
|
|
Apprentice
Joined: 04 Dec 2009
Posts: 27
Location: England
|
thanks again for your patience 
| Quote: |
| It's entirely useless once you've tried it at least once. |
no - I'm putting the input msg on the error queue along with the error message to let the database / users sort it out.
| Quote: |
| You can configure the input queue to retain non-persistent messages on queue manager restart. |
but they'll disappear after a QM crash
| Quote: |
| If you don't want an BOQ filling up, then deploy another flow that reads the BOQ and throws all messages away.... MQInput-->(nothing) will do the trick just fine. |
Any time I blindly delete messages from a backout queue I could also be deleting unhandled exceptions too - I could put in many try/catch nodes that write to queues secondary queues, but the risk will still be there.
Wouldn't a blind delete of BOQ msgs be worse architecture than wiring up the fail terminal - why is that so bad anyway?
A BOQ message indicates to me that something went seriously wrong with the flow, it shouldn't be written to as a matter of course. |
|
| Back to top |
|
|
| ollyc |
| Posted: Mon Dec 07, 2009 9:32 am Post subject: |
|
|
Apprentice
Joined: 04 Dec 2009
Posts: 27
Location: England
|
Hi Vicky,
| Quote: |
| 1. You can roll back the outputs by connecting the catch terminal just as easily. |
no - this is wrong, catch does NOT rollback previous outputs.
| Quote: |
| 2. Your backout queue can be whatever you want it to be (it could be the same as your error queue - or better a dedicated backout queue) and allows you to keep the input message in a failure situation - for diagnosis and/or replay. |
agreed.
| Quote: |
| 3. You state you "want to capture the input data + error message and write it to an error queue" but not a backout queue - does that mean you don't want the input messages in their unaltered state but that you are instead going to add some sort of error header to them? You don't lose the reliability by implementing proper error handling - far from it. |
agreed - this is what I'm aiming for.
| Quote: |
| 4. Backout count certainly does work with JMS messages. |
yes, that's what I said 
| Quote: |
| I do not understand why you need to make a distiction between handled and unhandled errors - just write your code to handle them. If all else fails (e.g. the error queue is full) then the correct configuration and transactionality will allow MQ to let you know and write those messages to the DLQ instead. |
I need to get to the fail terminal to rollback the outputs, it's not enough to just do the catch terminal - once there ALL messages handled and unhandled will appear in the DLQ / BOQ - this is not good.
if I only had a single output this would be easy - but I don't so it isn't! |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Dec 07, 2009 9:39 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| It is as if I have said nothing at all. |
|
| Back to top |
|
|
| ollyc |
| Posted: Mon Dec 07, 2009 9:57 am Post subject: |
|
|
Apprentice
Joined: 04 Dec 2009
Posts: 27
Location: England
|
| Quote: |
| It is as if I have said nothing at all. |
TBH, I don't think you've understood the key issues around message persistance, transactionality and multiple outputs:
| Quote: |
Any time I blindly delete messages from a backout queue I could also be deleting unhandled exceptions too - I could put in many try/catch nodes that write to queues secondary queues, but the risk will still be there.
Wouldn't a blind delete of BOQ msgs be worse architecture than wiring up the fail terminal - why is that so bad anyway?
A BOQ message indicates to me that something went seriously wrong with the flow, it shouldn't be written to as a matter of course. |
We did have an senior IBM software services WMB guy come in 2 weeks ago to review this and other flows - he came to the same conclusion as me that our design was the only one that would work.
Sorry that I've either not explained myself very well or you've not understood the issues |
|
| Back to top |
|
|
| mqseries0209 |
| Posted: Mon Dec 07, 2009 10:47 am Post subject: |
|
|
Voyager
Joined: 30 Mar 2006
Posts: 90
|
hey ollyc,
Here are some of your assumptions, from your previous posts:
1) cannot roll back a message with out connecting Failure terminal
- Thats false assumption, you can roll back the messages with out connecting failure terminal and of-course the roll back depends on each individual node participating in co-ordinated transaction or not.
2) catch does NOT rollback previous outputs.
- yes catch it self does not rollback previous outputs, you have to use catch then finally use THROW node or THROW statement to (after you finish error handling) throw the exception.
I am assuming you are delaying parsing of input message after common login and stuff... i,e a later point in flow and the input node you are reading message as BLOB, if this is not true, give a layout of your flow and what you are trying to achieve.
Are you parsing the message on Input node ?.
refer to this link for more info on error handling on MQInput node.
http://publib.boulder.ibm.com/infocenter/wmbhelp/v6r1m0/index.jsp?topic=/com.ibm.etools.mft.doc/ac00414_.htm
_________________
IBM Certified Solution Developer - WebSphere Message Broker V6.1
IBM Certified Solution Developer - WebSphere Integration Developer V6.0 |
|
| Back to top |
|
|
| ollyc |
| Posted: Mon Dec 07, 2009 11:30 am Post subject: |
|
|
Apprentice
Joined: 04 Dec 2009
Posts: 27
Location: England
|
Thanks for taking the time to respond -
1 +2: I agree absolutely, to rollback outputs I just need to get the message to the failure terminal - I don't need to wire up the output of the failure terminal.
| Quote: |
| Are you parsing the message on Input node ?. |
Currently no, we're using Websphere TX so the input node is treating the message as a BLOB. We may do in the future (as our development team becomes more familiar with ESQL), how do you see this affecting the design?
I've followed all the links on error handling and the various outputs - the bullet proofing WMB pdf is quite useful.
However, they assume a single message output.
Another example of a serious error is ExceptionTree corruption (this is why the WMB recreates the ExceptionTree for messages coming down the fail terminal).
If this happens, I don't want to bin the input message (e.g. by sending to a BOQ and deleteing all) but save it somewhere for processing.
However, given that I have to go to the failure terminal always to rollback my multiple outputs I've got a problem separating the two types of error (handled and unhandled).
Am I starting to make sense or still talking gibberish?  |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Dec 07, 2009 12:43 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| ollyc wrote: |
I've followed all the links on error handling and the various outputs - the bullet proofing WMB pdf is quite useful.
However, they assume a single message output. |
The number of outputs doesn't matter.
Either a given output is within the transaction started by the Input node or it's not.
If it is in that transaction, it's rolled back when the input transaction is.
If it's not in that transaction, it's not rolled back. |
|
| Back to top |
|
|
| ollyc |
| Posted: Mon Dec 07, 2009 1:43 pm Post subject: |
|
|
Apprentice
Joined: 04 Dec 2009
Posts: 27
Location: England
|
Hi Jeff
the number of outputs does matter - it affects the design.
If there is only one output you don't need the transaction to fail to rollback your output, i.e. you've either not written to your one output yet or you had an error while writing to it.
As there's nothing to rollback, you don't need to include a throw node from the catch terminal (to force a flow failure). Your error handling (writing to the error queue) would be in transaction scope and you always remove the input message from the source queue.
Some, but not all of my flows will follow this single output pattern. I'm trying to understand 'worst case' (multiple outputs) at the moment. |
|
| Back to top |
|
|
| Herbert |
| Posted: Tue Dec 08, 2009 1:41 am Post subject: |
|
|
Centurion
Joined: 05 Dec 2006
Posts: 146
Location: Leersum, The Netherlands
|
| Quote: |
| Quote: |
| Are you parsing the message on Input node ?. |
Currently no, we're using Websphere TX so the input node is treating the message as a BLOB. We may do in the future (as our development team becomes more familiar with ESQL), how do you see this affecting the design? |
If you parse and validate on the input-node then it is possible that the message goes directly to the Failure terminal, bypassing the Catch Terminal.
Also, if you use a backout threshold > then zero, then it is possible that you get a Catch and no Failure because a re-prossessing goes ok.
Because of both points I use below template for error handling. This works, single-output, multi-output, even if you change input-node parsing/validation, or change the MQ backout settings.
Catch Terminal
- Store the ExceptionList somewhere with WARNING as level.
- Throw
Failure terminal
- Store the ExceptionList somewhere with ERROR as level.
- Store the original message somewhere.
"Somewhere" can be MQ queues (I prefer this), database tables, or a mix of it.
And yes, if you analysing problems, you need most of the time the last WARNING ExceptionList for the root problem problem because the ERROR ExceptionList does not say much, if you use the same key with the stores (message-id is usual suspect for this of course) then that is no problem IMHO.
Don't do more in your Catch/Failure handling. Do the rest (analysing the Exception list, sending emails, whatever) in an other process where you process the information you did store. This keeps your Catch/Failure handling in the main-flows fast and clean. |
|
| Back to top |
|
|
| ollyc |
| Posted: Tue Dec 08, 2009 2:05 am Post subject: |
|
|
Apprentice
Joined: 04 Dec 2009
Posts: 27
Location: England
|
Thanks for the response - your design is very similar to what we're doing.
| Quote: |
| Don't do more in your Catch/Failure handling. |
I agree, we just catch the exception details + message and pass it to another queue for a separate flow to sort out - the two (error handling vs. error processing) must be decoupled.
| Quote: |
| if you use the same key with the stores (message-id is usual suspect for this of course) |
We keep the ExceptionList with the message (i.e. grab the input message from the catch terminal and put it to the error queue with the exception list) - so we don't need to do the reconciliation of warning msg, error msg and input msg at a later point - but I'll review the database design and see if we could do it your way.
Ideally though, you wouldn't need to do the reconciliation and we wouldn't need to use the shared row - there'd be some simpler option that is closer to the design of the single output flow.
(Also if the MQ MsgID isn't unique on the queue this could cause problems in both of our designs).
your input is much appreciated though.
Olly. |
|
| Back to top |
|
|
|
|
