| Author |
Message
|
| mpong |
 Posted: Thu Nov 12, 2015 3:15 pm Post subject: FileInput Node File processing |
 |
|
Disciple
Joined: 22 Jan 2010
Posts: 164
|
Is there any way to control file input node from picking up the files until source completes its writing.
We have issue where we are looking for *.xml files in the input directory and JDE places file with .xml and then writs it (We have already told them they should not write the file in IIB directory until jobs completes but not possible there ). As soon as IIB finds .xml file it moves to mqsitransit or backout folder(depending on retry).
We can not implement retry mechanism as we do not know how much time it may take to complete the file writing.
I can not use file read node as file name is randomly generated . There is an another tool which used for this earlier and it is able to wait until the file lock is over and then it process it, is it possible in IIB. Please suggest. |
|
| Back to top |
|
 |
| zpat |
| Posted: Thu Nov 12, 2015 11:00 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Several ways.
1. Make your writing software using file locking (something that, incredibly, is optional in the world of Unix).
2. Create the file with a different suffix, then rename it after it is written (and set the file node to look for the new suffix only).
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Last edited by zpat on Mon Apr 25, 2016 5:00 am; edited 1 time in total |
|
| Back to top |
|
|
| mpong |
| Posted: Thu Nov 12, 2015 11:20 pm Post subject: |
|
|
Disciple
Joined: 22 Jan 2010
Posts: 164
|
Thanks for the response, right now JDE creates file with .tmp and then it also creates a actual file with .xml and starts writing, issue is IIB picks up as soon as it finds .xml, from IIB perspective it is correct.
But the problem is source can not change the logic as the impact is high. |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Nov 13, 2015 3:30 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
The file node uses file locking, if the file writing software does not - then it is not really safe.
The rename approach has minimal changes needed.
Increasing the scan interval on the file node will reduce the chance of picking up a partial file.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Last edited by zpat on Mon Apr 25, 2016 5:01 am; edited 1 time in total |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Nov 13, 2015 7:33 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
some now overused rant about files and the Halting Problem.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Nov 13, 2015 7:49 am Post subject: Re: FileInput Node File processing |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mpong wrote: |
| There is an another tool which used for this earlier and it is able to wait until the file lock is over and then it process it, is it possible in IIB. |
Yes, but you need to have a real file lock (an OS one). If the product that's writing the file doesn't instantiate a lock (and many don't, to improve performance) then the file isn't actually locked and IIB will be unable to distinguish it from a completed file.
The "another tool" almost certainly isn't using file locking but one of the other "quick & dirty" methods available in Unix to try and see if the file's in use / being written to.
The rename method is your best solution as a "poor man's file lock". I echo my most worthy associates tokenized rant about files and the halting problem.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
