| Author |
Message
|
| scravr |
 Posted: Thu Jul 18, 2013 2:29 pm Post subject: FileInput node - EndOfData |
 |
|
Partisan
Joined: 03 Apr 2003
Posts: 391
Location: NY NY USA 10021
|
I have a flow that does not need to read whole file. I just need to know when a file is completely ready to be processed.
So if I get a large file FTP (from other OS) that takes about 5min to FTP, I need to know when FTP is done.
I connected EndOfData (instead of out) terminal.
Does FileInput EndOfData terminal waits for the file to be completely "ready", or flow will be triggered as soon as files comes into directory? |
|
| Back to top |
|
 |
| someyears |
| Posted: Thu Jul 18, 2013 3:53 pm Post subject: |
|
|
Newbie
Joined: 17 Jul 2013
Posts: 5
|
please check v7 fixpac5. Then your problem would be resolved.
But I am using Java utility to write file. |
|
| Back to top |
|
|
| scravr |
| Posted: Thu Jul 18, 2013 4:21 pm Post subject: |
|
|
Partisan
Joined: 03 Apr 2003
Posts: 391
Location: NY NY USA 10021
|
| anyone with better ideas? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Jul 18, 2013 11:56 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Knowing that a file is ready is a Turing complete problem, as it is a direct example of the Halting Problem.
You can't know that the file is ready to be processed until you know that the program writing the file has Halted.
If you have a program that is sending the file to the broker file system using FTP, then the program sending the file should either rename or move the file as the last step after it has completed the transfer. This will allow you to configure FileInput to only read the file from the final name/location, rather than from the interim direction.
If FileInput node is reading the file from FTP directly, then again the application that is writing the file needs to put it in an interim location/name until it is fully ready to be read, and then move it to a new location for FileInput to read.
FileInput node will not propagate anything out of the In terminal until it has completed the transfer of the file over FTP, if it is reading the file over FTP.
FileInput node End of Data terminal is only fired when the file is not being read as a single whole file. If FileInput is using Whole File mode, then it will not call EndOfData terminal.
If the only purpose of this flow is to move the flie from one system to another, you should look at an alternate solution, like Managed File Transfer or etc. |
|
| Back to top |
|
|
| scravr |
| Posted: Fri Jul 19, 2013 5:03 am Post subject: |
|
|
Partisan
Joined: 03 Apr 2003
Posts: 391
Location: NY NY USA 10021
|
| So when "out" terminal is not connected, but "EndOfData" is connected and configured as RecordDetection as FixedLength of 100GB: Flow will be triggered as soon as file arrives from FTP even when FTP is still in transition for about 5min? |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Jul 19, 2013 8:35 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Why don't you try it with say a 1Mb file and see what happens?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sat Jul 20, 2013 3:07 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| scravr wrote: |
| So when "out" terminal is not connected, but "EndOfData" is connected and configured as RecordDetection as FixedLength of 100GB: Flow will be triggered as soon as file arrives from FTP even when FTP is still in transition for about 5min? |
I'm not sure I follow.
The FileInput node will process things in this manner:
1) establish an FTP connection to the remote server
2) scan for files that match the relevant pattern
3) for each file that matches the relevant pattern,
3a) start downloading the file to a temporary location on the broker's file system
3b) process the file on the basis of the configuration of the FileInput node.
I could be wrong about 3a and 3b. It could be that the FileInput node will not process the file until the transfer from the remote side is fully complete.
If you have configured the FileInput node to process the file with a record Detection of Fixed Length and a Record Size of 100GB, then the in terminal should not get fired until a complete record of 100Gb is downloaded to the temporary location.
If the file is smaller than 100GB, then no complete record will ever be available. In that case, the FileInput node will handle the record in the same way that it is documented to handle incomplete records of any type.
What is the real goal that you are trying to accomplish?
You are trying to do *something*, and you have decided that you should be able to accomplish this by abusing the FileInput node using the EndOfData terminal only.
Again, there is nothing that any use of FileInput node can do to protect you from an application that is writing out the input file in a way that prevents other systems from knowing if the file is complete or not. Again, HALTING PROBLEM.
I ran into an issue with a very old copy of Microsoft's FTP server, where it would deliver incoming files to the destination directory by creating the final file at the final name and preallocating the file space to the final file size. The only way to tell if the file was complete was to check if the modification date of the file had changed recently, for some arbitrary value of 'recently'. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sat Jul 20, 2013 5:23 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| mqjeff wrote: |
I ran into an issue with a very old copy of Microsoft's FTP server, where it would deliver incoming files to the destination directory by creating the final file at the final name and preallocating the file space to the final file size. The only way to tell if the file was complete was to check if the modification date of the file had changed recently, for some arbitrary value of 'recently'. |
That is SOP for copying files on Windows unlike Linux/Unix who grow the file as it is copied.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| scravr |
| Posted: Tue Jul 23, 2013 11:01 am Post subject: |
|
|
Partisan
Joined: 03 Apr 2003
Posts: 391
Location: NY NY USA 10021
|
What I need is a way to trigger FileInput ONLY after a file was completely finished FTP (from other OS, or other network) into local directory, and when a file was finished copy from DIR-A into this file-input-firectory. All this without asking my current multi-numerous clients to change file name, or send MQ message reference, or other change to current doing business.
Testing with FTP, the EndOfData was triggered as soon as file arrived, and while FTP was going-on for more that 5min.
So if “out” is not connected, but only EndOfData is connected: flow is triggered immediately. This is not “true” EndOfData !!!
Again: Node is triggered on /apps/myapp/WmbFileTrigger directory (gets files from for both local apps and files from numerous client’s FTP). Record Detection: Fixed Length and Record Size: 100GB. In that case EndOfData should check to see if file has any open handles, or any associated TaskIDs, etc. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jul 23, 2013 11:19 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| scravr wrote: |
| What I need is a way to trigger FileInput ONLY after a file was completely finished FTP |
How would you do this, within the constraints you mention, if the application processing the file was not WMB but (for example) a Java or C++ application? How you you signal that application that the FTP was complete?
Use the same method here.
Then share it.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jul 23, 2013 11:14 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
If FileInput node is performing the FTP, then you don't need to do anything other than wire up the In terminal.
If FileInput is NOT performing the FTP, then you need to change whatever *is* performing the FTP to only put the file into the place that FileInput is looking, with the name FileInput is looking for, AFTER THE FILE IS COMPLETE.
As I said, this is a firm real life example of the HALTING problem, which FileInput can not resolve on it's own.
Use MQ ManagedFileTransfer or something else if all you're doing is moving files around.
FileInput is designed to give the data in a file to Broker for processing. Not to move files around. |
|
| Back to top |
|
|
|
|
