| Author |
Message
|
| ggriffith |
 Posted: Thu Jun 05, 2008 2:20 am Post subject: FileInput Node |
 |
|
Acolyte
Joined: 17 Oct 2007
Posts: 67
|
| Is it guaranteed that the FileInput node will not read a file from source until the file has finished being written. |
|
| Back to top |
|
 |
| AkankshA |
| Posted: Thu Jun 05, 2008 2:24 am Post subject: |
|
|
Grand Master
Joined: 12 Jan 2006
Posts: 1494
Location: Singapore
|
yes...
from the product documentation..
FileInput has a built-in mechanism that avoids processing of open files (for
example, files that are not yet fully formed because the producing application is still writing to them).
_________________
Cheers |
|
| Back to top |
|
|
| ggriffith |
| Posted: Thu Jun 05, 2008 2:32 am Post subject: |
|
|
Acolyte
Joined: 17 Oct 2007
Posts: 67
|
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Jun 05, 2008 3:26 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
With apologies to mqpaul, I think it is not a guarantee. I think it is an 'everything that can be done has been done' situation.
A simple example is the case where an application is repeatedly open/write/close the file to append records.
Or, if a file is on an NFS mount, and the writing application is on a remote machine, the lock will almost certainly not be visible to the Broker (NFS is special that way).
It's a very hard problem to ensure that some given file is not being accessed by a remote application, and to know when a file is 'ready', without having intimate knowledge of the contents of the file and the process writing to it.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| ggriffith |
| Posted: Thu Jun 05, 2008 4:58 am Post subject: |
|
|
Acolyte
Joined: 17 Oct 2007
Posts: 67
|
That's a shame, but understandable. We'll just have to work ( arrange things ) around the problem.
Thanks |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Jun 05, 2008 5:03 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
If you understand your given situation well enough, you may be able to treat it as a guarantee.
It's only when you are putting the FileInput node in a situation where "anything can happen" that you have to take extra steps to increase reliability.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| joebuckeye |
| Posted: Thu Jun 05, 2008 6:06 am Post subject: |
|
|
Partisan
Joined: 24 Aug 2007
Posts: 365
Location: Columbus, OH
|
In the UNIX world I believe it is common when one process is writing a file for another process to use that the writing process uses a filename that starts with a period (a dot-file). This creates a file "hidden" from normal directory queries. When the writing process is done it then renames/moves the file from the dot-filename to the regular filename.
Not sure if this would help you since you didn't mention what environment you are running under. And I don't know if the File Input node ignores dot files or not. |
|
| Back to top |
|
|
| ggriffith |
| Posted: Thu Jun 05, 2008 6:29 am Post subject: |
|
|
Acolyte
Joined: 17 Oct 2007
Posts: 67
|
| Interesting point, and yes we are in the unix environment. Dare say someone will comment soon on whether the FileInput node ignores dot files or not. Can't test this myself as I have only have 6.1 on Windows |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Jun 05, 2008 8:28 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
The file node will ignore dot files if you tell it to ignore dot files.
regardless, if you write your file with a pattern that doesn't match the pattern the FileInput is watching, and then rename after completion, then the FileInput will never read the file "while it is being written".
As I said - if you know the situations of your file transfer and they are controllable, then you can ensure that the FileInput will never read a file that is not complete.
It's only when you do things like receive files from unreliable external parties that you have to start thinking about seriously complicated failure scenarios. And then you should question the value of exposing Broker to such an unreliable input source anyway...
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| mqpaul |
| Posted: Mon Jun 09, 2008 1:52 am Post subject: Three ways to stop FileInput reading incomplete files |
|
|
Acolyte
Joined: 14 Jan 2008
Posts: 66
Location: Hursley, UK
|
- The FileInput node respects operating system file locks. If the process creating the file locks it for R/W use, the FileInput node won't read it until process releases the lock. However, many UNIX applications don't lock their output file.
- Create the input file with a name that doesn't match the FileInput node Pattern - for example, with a leading dot, in which case the pattern should be "?*.*" or similar (you need the "?" to require at least one character before the "."). Rename it when complete.
- Create the input file in a subdirectory of the input directory, and when complete, move it to the input directory. This is the same process as the FileOutput node's use of the mqsitransit directory. (On UNIX, it's effectively the same as the rename option.)
_________________
Paul |
|
| Back to top |
|
|
|
|
