| Author |
Message
|
| ghoshly |
 Posted: Thu Jun 03, 2010 3:16 am Post subject: Transaction property for FileOutput node |
 |
|
Partisan
Joined: 10 Jan 2008
Posts: 333
|
I need to maintain transactional boundary in a scenario where I need to read a message from MQ queue and based on that I need to put multiple output files in multiple directory.
So simple flow of MQInput --> Compute --> FileOutput
The whole output file is the content for one file, so using " Record is Whole File " option, which has no effect on the Finish terminal msg.
Now, I need to maintain ALL or None scenario for all the output files, but even if the Transaction mode is set to 'yes' in the MQInput node, that is not maintained for output files, and in case of any error output files are not rolled back.
Please help me how I can design and implement this.
Thanks in Advance...  |
|
| Back to top |
|
 |
| ghoshly |
| Posted: Thu Jun 03, 2010 3:18 am Post subject: Environment : Broker Version |
|
|
Partisan
Joined: 10 Jan 2008
Posts: 333
|
Sorry, forget to mention the broker version and environment set up.
Broker 6.1.0.4
Windows and AIX |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Jun 03, 2010 4:46 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
The documentation on the FileInput and FileOutput nodes are extremely clear that they are NOT transactional.
This is because FILES are not transactional. |
|
| Back to top |
|
|
| ghoshly |
| Posted: Fri Jun 04, 2010 1:41 am Post subject: |
|
|
Partisan
Joined: 10 Jan 2008
Posts: 333
|
Yes, now I understand that, sorry, but in order to achieve the functionality is there any way where I can still keep lock on the output files through the FileOutput node and release them for other applications only when all are available ?
Or else my thinking process :
MQInput --> Transform_Compute --> PutFile_JCN --> Other_Func
PutFile_JCN :
try {
1) Would put files in the directory with an exclusive lock on the file (FileChannel.tryLock)
2) Update the List in Environment and increase the fileCount .
}
catch {
for all previous put success in try block ( a. Unlock the file.
b. Delete the file.
)
}
finally {
It would release lock when all the files are fine. i.e count reaches max file count.
}
My concern is it possible that the application which is reading the output directory, can read the file between Step a and Step b
when file lock is released but is not removed yet.
Thanks for help.. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jun 04, 2010 1:50 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| ghoshly wrote: |
| Yes, now I understand that, sorry, but in order to achieve the functionality is there any way where I can still keep lock on the output files through the FileOutput node and release them for other applications only when all are available ? |
Generally, no.
Generally, this is a significantly more complicated scenario than it sounds like on first draft.
You should ideally convert the receiving application to use MQ and then use a Message Group to hold the "files".
Otherwise you need to do something to take an action when your flow knows that all files are done. I suppose one POSSIBLE way to do this is to NOT use Record is Whole File, and use the Finish File terminal. Once you know that you've written out the data portion to all files (but they aren't finished) then you can do a quick loop to finish all files.
But this is still fraught with error cases and very likely to fail silently - like any solution for the general problem of "I need X files to be available as an atomic unit". |
|
| Back to top |
|
|
| fatherjack |
| Posted: Fri Jun 04, 2010 2:00 am Post subject: |
|
|
Knight
Joined: 14 Apr 2010
Posts: 522
Location: Craggy Island
|
| mqjeff wrote: |
Otherwise you need to do something to take an action when your flow knows that all files are done. I suppose one POSSIBLE way to do this is to NOT use Record is Whole File, and use the Finish File terminal. Once you know that you've written out the data portion to all files (but they aren't finished) then you can do a quick loop to finish all files.
But this is still fraught with error cases and very likely to fail silently - like any solution for the general problem of "I need X files to be available as an atomic unit". |
Absolutely concur.
Maybe you could write your files to a temporary directory and then use the Finish File terminal to kick off a process to move them.
I'd also suggest having some sort of file header and trailer records so that if other applications do happen to pick up incomplete files, they'll recognise them as such due to missing trailers, or mismatched record counts.
_________________
Never let the facts get in the way of a good theory. |
|
| Back to top |
|
|
| ghoshly |
| Posted: Sun Jun 06, 2010 11:27 pm Post subject: Possible way |
|
|
Partisan
Joined: 10 Jan 2008
Posts: 333
|
| thanks a lot, I have implemented the possible way mqjeff suggested. Can we take any precautionary cover for the possible silent failure ? |
|
| Back to top |
|
|
|
|
