| Author |
Message
|
| GeneRK4 |
 Posted: Thu Feb 27, 2014 9:44 pm Post subject: Exception Handling in File nodes |
 |
|
Master
Joined: 08 Jul 2013
Posts: 220
|
Hi,
My flow contains FileInput node with Retry Threshold as 3.
The exception happens in one of the downstream nodes.
Hence the catch terminal receives the real exception message.
The message is retried via Filenode 3 times as the retry threshold is set as 3 and "Throw" node is given in the end of catch terminal nodes.
The problem I see here is,
If I have to catch the real exception ,then an MQOutput node has to be placed in the catch terminal itself.In this case,the error message written into the MQ node is 3 times according to the RetryThreshold.
Hence,to avoid such scenario,I placed MQOutput node in Failure terminal.
But according to "http://publib.boulder.ibm.com/infocenter/wmbhelp/v7r0m0/index.jsp?topic=%2Fcom.ibm.etools.mft.doc%2Fac00545_.htm" ,Environment variable is not copied to the tree in Failure terminal.
Hence,I placed the throw node in Failure terminal which just backs out the original file in mqsibackout folder.And I placed the MQOutput node in catch terminal which captures the actual exception message.The compramise ,I had to make here is the error message is written 3 times in the Error queue.
Any thoughts on this please? |
|
| Back to top |
|
 |
| Esa |
| Posted: Thu Feb 27, 2014 10:59 pm Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
If you had a similar scenario with MQ Input node, you could examine MQMD.backoutCount and write the error message only when it's 1.
Unfortunately files don't have a backout count and File Input node doesn't generate any subsititute for it in LocalEnvironment either.
What you could do is to maintain a backout count yourself in a shared row variable. Insert an entry for each retried file. The key can be the filename -- if it's unique enough -- or a combination of filename + LocalEnvironment.File.LastModified. Maintain the backout count in a subelement.
Write the error message only when there is no entry for the file in the shared row.
The third retry should remove the entry.
Surprisingly, sometimes a retry may be successfull. If this happens often, the size of the tree owned by the shared row will grow. You can add a timestamp subelement for each entry and run a periodical check that removes old entries. |
|
| Back to top |
|
|
| Gralgrathor |
| Posted: Fri Feb 28, 2014 1:19 am Post subject: |
|
|
Master
Joined: 23 Jul 2009
Posts: 297
|
| Esa wrote: |
| the size of the tree owned by the shared row will grow |
End the flow with a compute node to do some housekeeping?
Eg.
| Code: |
| FileInput -> Stuff happens -> MQOutput -> Compute node (to remove this particular file-retry-entry (and additionally expired ones) from shared var) |
?
_________________
A measure of wheat for a penny, and three measures of barley for a penny; and see thou hurt not the oil and the wine. |
|
| Back to top |
|
|
| Esa |
| Posted: Fri Feb 28, 2014 2:34 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| Gralgrathor wrote: |
| Esa wrote: |
| the size of the tree owned by the shared row will grow |
End the flow with a compute node to do some housekeeping?
Eg.
| Code: |
| FileInput -> Stuff happens -> MQOutput -> Compute node (to remove this particular file-retry-entry (and additionally expired ones) from shared var) |
? |
Adding a node for housekeeping doesn't add any value -- on the contrary.
Compared to doing housekeeping after propagation in the same node that maintains the retry counters. It's a good idea to keep this kind of things as self-contained as possible, though it's certainly a bit challenging with ESQL. If you ever have played with the idea of implementing an ESQL singleton, you should know.
Yes, housekeeping is certainly needed. I personally prefer periodical cleanup of expired entries. The cleanup operation may be relatively costly, especially if some external failure causes all messages to get captured in the loop. It just doesn't make sense to me to run cleanup every time you retry. |
|
| Back to top |
|
|
| Gralgrathor |
| Posted: Fri Feb 28, 2014 2:49 am Post subject: |
|
|
Master
Joined: 23 Jul 2009
Posts: 297
|
| Esa wrote: |
| Adding a node for housekeeping doesn't add any value -- on the contrary. |
What I proposed was a final node to remove the entry only after a successful transaction.
A simple node, that on conclusion of the flow (after the results have been passed on to the next step in the chain) retrieves the key for the file from the Environment, seeks it up in the SHARED table, and removes it. This would have to be in *addition* to code that removes expired entries, I agree.
I can imagine such a final step in the transaction being necessary when, for instance, you can't be certain that a new file with identical properties might not arrive before the expiration on the entry.
Would this have a serious impact on performance? What other drawbacks might there be? I'm interested in learning about the pros and cons of such patterns.
_________________
A measure of wheat for a penny, and three measures of barley for a penny; and see thou hurt not the oil and the wine. |
|
| Back to top |
|
|
| Esa |
| Posted: Fri Feb 28, 2014 4:18 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| Gralgrathor wrote: |
I can imagine such a final step in the transaction being necessary when, for instance, you can't be certain that a new file with identical properties might not arrive before the expiration on the entry.
Would this have a serious impact on performance? What other drawbacks might there be? I'm interested in learning about the pros and cons of such patterns. |
Not at all, that's just a best practice I use myself, because as it goes with the propagation model, upstream nodes have more reliable knowledge on transaction success than downstream nodes. Even if most people seem to believe it's vice versa. |
|
| Back to top |
|
|
| GeneRK4 |
| Posted: Fri Feb 28, 2014 6:20 pm Post subject: |
|
|
Master
Joined: 08 Jul 2013
Posts: 220
|
Thanks...
As this is Sharedvariable,When there is an unexpected restart happens at flow/EG/Broker,then the message which is under retry will be lost? |
|
| Back to top |
|
|
| Esa |
| Posted: Sat Mar 01, 2014 10:55 pm Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
Your problem was that within a retry loop the exception message was written trice in your error queue.
You haven't mentioned your strategy with failed payload messages. This is something you configure in the File Input node:
- Retry.Action on failing file
- Basic.Action on successfull processing
Using a shared variable doesn't make messages dissappear even if the EG crashed. In your case the input file will stay in mqsitransit and be retried another three times when the EG resumes. But the exception message would obviously get written twice in the error queue. |
|
| Back to top |
|
|
| GeneRK4 |
| Posted: Sun Mar 02, 2014 9:44 pm Post subject: |
|
|
Master
Joined: 08 Jul 2013
Posts: 220
|
Thanks for your response  |
|
| Back to top |
|
|
|
|
