| Author |
Message
|
| sandman147 |
 Posted: Mon Mar 03, 2014 1:13 pm Post subject: End Message Flow- Propagating individual msgs to File Output |
 |
|
Apprentice
Joined: 01 Sep 2013
Posts: 42
|
The File Input Node reads a file record by record and propagates individual messages to the file output upon success.
Example:
Input
A
B
C
D
E
Output
1
2
3
4
5
Say, Upon record 'C' an exception is thrown, '3' will not be written. However 'D' and 'E' are processed after a failed 'C' resulting in the output
1
2
4
5
How can the Message Flow be terminated and stop further processing of records when such an exception is to occur i.e, The output file should not create at all |
|
| Back to top |
|
 |
| Gralgrathor |
| Posted: Tue Mar 04, 2014 2:00 am Post subject: |
|
|
Master
Joined: 23 Jul 2009
Posts: 297
|
Various ways.
In all cases, it's useful to set the FileOut node to stage with append, and only move to the target folder at the finish signal.
You could rewrite the flow so that the input is processed in a single transaction, if the amount of data is reasonable. if the transaction aborts the finish signal won't be given, and the staging file won't be moved to the target folder.
You could use shared variables to record the state of processing for that particular input file (assuming 1 instance), making the flow reject the rest of the input when an exception is recorded. The finish signal isn't produced, etc.
Or you could redesign the chain, so that the flow produces an extra file with the original records that produced exceptions. The target application then knows that loss occurred, and for which records. You'd need to make sure the EOF signal is produced for both files in that case.
Any other ideas?
_________________
A measure of wheat for a penny, and three measures of barley for a penny; and see thou hurt not the oil and the wine. |
|
| Back to top |
|
|
| sandman147 |
| Posted: Tue Mar 04, 2014 7:17 am Post subject: |
|
|
Apprentice
Joined: 01 Sep 2013
Posts: 42
|
| Quote: |
| You could rewrite the flow so that the input is processed in a single transaction, if the amount of data is reasonable. if the transaction aborts the finish signal won't be given, and the staging file won't be moved to the target folder. |
While reading the Input file Record by Record. Without the use of Shared Rows or Variables, How may the information in the output tree be retained for every record read i.e, Append the output messages in the output tree ? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Mar 04, 2014 7:21 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| sandman147 wrote: |
| Quote: |
| You could rewrite the flow so that the input is processed in a single transaction, if the amount of data is reasonable. if the transaction aborts the finish signal won't be given, and the staging file won't be moved to the target folder. |
While reading the Input file Record by Record. Without the use of Shared Rows or Variables, How may the information in the output tree be retained for every record read i.e, Append the output messages in the output tree ? |
There are many many ways you can do this. Most of them are really bad ideas. |
|
| Back to top |
|
|
| sandman147 |
| Posted: Tue Mar 04, 2014 8:36 am Post subject: |
|
|
Apprentice
Joined: 01 Sep 2013
Posts: 42
|
| Quote: |
| There are many many ways you can do this. Most of them are really bad ideas. |
Understood, Memory consumption concerns. Which practice would you recommend ?
| Quote: |
| You could use shared variables to record the state of processing for that particular input file (assuming 1 instance), making the flow reject the rest of the input when an exception is recorded. The finish signal isn't produced, etc. |
This approach Initializes exception flags (variables) which decide how the rest of the flow is to managed in case of an error in processing. However the Unfinished File remains in the mqsitransit folder. |
|
| Back to top |
|
|
| Gralgrathor |
| Posted: Tue Mar 04, 2014 9:15 am Post subject: |
|
|
Master
Joined: 23 Jul 2009
Posts: 297
|
| sandman147 wrote: |
| However the Unfinished File remains in the mqsitransit folder. |
Correct: that's the intention. You'd have to figure out something else to get rid of expired partials.
| sandman147 wrote: |
| Without the use of Shared Rows or Variables, How may the information in the output tree be retained for every record read i.e, Append the output messages in the output tree ? |
A two-stage approach could work. Flow A produces a file with transformed records and a linked list of exceptions, appending both files message-by-message in the staging folder, writing them to the input folder for Flow B when done; Flow B checks the list of exceptions and decides whether or not to put the transformed data through to its final destination. It's probably a more reliable pattern than anything that works with shared variables.
_________________
A measure of wheat for a penny, and three measures of barley for a penny; and see thou hurt not the oil and the wine. |
|
| Back to top |
|
|
| sandman147 |
| Posted: Tue Mar 04, 2014 12:36 pm Post subject: |
|
|
Apprentice
Joined: 01 Sep 2013
Posts: 42
|
| Quote: |
| A two-stage approach could work. |
Sounds like a plan. But is there an effective (save memory) implementation using a single message flow ? |
|
| Back to top |
|
|
| Gralgrathor |
| Posted: Tue Mar 04, 2014 12:47 pm Post subject: |
|
|
Master
Joined: 23 Jul 2009
Posts: 297
|
Can't think of one. Unless you can get the target application to do the work of Flow B.
_________________
A measure of wheat for a penny, and three measures of barley for a penny; and see thou hurt not the oil and the wine. |
|
| Back to top |
|
|
|
|
