| Author |
Message
|
| JonathanHill |
 Posted: Wed Feb 04, 2009 6:39 am Post subject: Does FileOutput node support renaming (after FTP sent)? |
 |
|
Novice
Joined: 30 Jan 2009
Posts: 13
|
I'm trying to FTP files to host systems (which are out of my control) and guarantee that if a file is found, by a host system, that it's a complete file.
My concern arises from a section, within the IBM infocenter, regarding the FTP within the FileOutput node on WMB v6.1: -
"During the FTP operation, the FileOutput creates the destination file. However, the destination file is readable before the FTP operation is complete. Therefore, ensure that remote applications do not read the file until the FTP operation is complete."
The obvious way to fix this is to FTP the file (as something unexpected) then rename it after the transfer is complete.
Is this possible using FileOutput node? |
|
| Back to top |
|
 |
| MQEnthu |
| Posted: Thu Feb 05, 2009 12:44 am Post subject: |
|
|
Partisan
Joined: 06 Oct 2008
Posts: 329
Location: India
|
Hi JonathanHill,
| Quote: |
"During the FTP operation, the FileOutput creates the destination file. However, the destination file is readable before the FTP operation is complete. Therefore, ensure that remote applications do not read the file until the FTP operation is complete."
|
The files created and written by the FileOut node will be stored temporarily under the subdirectory called "mqsitransit" untill the file transfer is complete. Once the writing to the file is over, file will be put into the directory mentioned on FileOutput node. Hence the target application will not have access to the files untill they are written completely by FileOutput node.
____________________________________________________ |
|
| Back to top |
|
|
| JonathanHill |
| Posted: Thu Feb 05, 2009 2:07 am Post subject: |
|
|
Novice
Joined: 30 Jan 2009
Posts: 13
|
Thanks MQEnthu.
Yes, I'm aware that "mqsitransit" is used until the transfer is complete but my problem isn't with the node itself rather with the fact that FTP, as a transfer protocol, has the issue of creating a destination file (which is readable by other applications) that is being 'filled' during the transmission process.
The result is that if you're transferring large files (that take some time to transfer) remote applications keep reading part-built files until the transfer is complete.
Unless I'm very much mistaken the FileOutput node will have no control over this issue in it's FTP step...hence the warning from IBM.
The simplest way to solve this is to FTP to a 'dummy' file (which isn't being expected by a remote application) and then perform an FTP rename once complete (which is instantly applied and so bypasses the issue).
But, from what I can see, the FileOutput node doesn't allow this 'rename' step and drops the issue onto the 'remote applications'. But, as I have no control over those, and there are multiple remote system, I'm caught between a rock-and-a-hard-place.
The only solution I can see is to write a custom node to do the rename...but wanted to see if anyone else had hit this issue. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Feb 05, 2009 2:36 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
The mqsitransit folder IS doing exactly what you want it to do.
The file being sent over FTP by FileOutput is NOT in the final output location with the final output name, where an application is looking for it, until the file contents are successfully and fully transfered across FTP.
If someone has decided to scan the mqsitransit folder for the file, then they are doing something wrong.
Any difference between writing file contents to mqsitransit or some version of /tmp are entirely illusionary. To say that writing something to some version of /tmp "puts it outside application control" is not true.
FTP is and always has been an UNRELIABLE file transfer mechanism. Never underestimate the bandwidth of a stationwagon full of dvdroms. |
|
| Back to top |
|
|
| JonathanHill |
| Posted: Thu Feb 05, 2009 2:44 am Post subject: |
|
|
Novice
Joined: 30 Jan 2009
Posts: 13
|
I like your style mqjeff...LOL
Ok, so the rename / move step is already in there via mqsitransit on the destination server.
My gap in knowledge was how robust, and indeed how that step, actually works.
I guess I can begin transferring my stationwagon of dvdroms now.
Thanks again. |
|
| Back to top |
|
|
| MQEnthu |
| Posted: Thu Feb 05, 2009 2:57 am Post subject: |
|
|
Partisan
Joined: 06 Oct 2008
Posts: 329
Location: India
|
Hi MQJeff,
The file will be created and written into transit subdirectory untill the writing is complete. Then the file will be transfered to the directory specified on node (Basic Tab). So your answer(even mine in ealrier post) holds good when the applications are trying read from broker local machine. But if I understand JonathanHillcorrectly, he wants to transfer the file to remote FTP server. So when doing the file transfer from broker machine to the remote FTP server, applications can start reading files which are written fully.
________________________________ |
|
| Back to top |
|
|
| MQEnthu |
| Posted: Thu Feb 05, 2009 3:06 am Post subject: |
|
|
Partisan
Joined: 06 Oct 2008
Posts: 329
Location: India
|
Hi JonathanHill,
You can mention the the directory name as "\\<ip address of the remote machine>\<Shared_directory_name>" in the basic tab of FileOutput node. And share the directory in the remote server and give write access to the broker user id. This way broker creates the mqsitransit folder in remote machine and start writing the file to the remote server transit folder. And once the writing is over it will be transferred to the desired directory. I know sharing folder on remote server may not be secure.
____________________________________________ |
|
| Back to top |
|
|
| JonathanHill |
| Posted: Thu Feb 05, 2009 3:06 am Post subject: |
|
|
Novice
Joined: 30 Jan 2009
Posts: 13
|
Hi MQEnthu
Yes, your understand of my question is correct.
From mqjeff's response, and your comments, the question hinges on where 'mqsitransit' exists if the FTP option is selected in the FileOutput node.
If it's local (on the Broker) then the problem still exists.
If it's remote (on the destination server) then the problem is solved.
If it's both then even better.
I feel a dig around in the IBM docs, and some prototyping, coming on. |
|
| Back to top |
|
|
| JonathanHill |
| Posted: Thu Feb 05, 2009 3:08 am Post subject: |
|
|
Novice
Joined: 30 Jan 2009
Posts: 13
|
Sorry MQEnthu...crossed responses.
I was just thinking the same thing. Time for some prototyping.
Thanks again...
 |
|
| Back to top |
|
|
| MQEnthu |
| Posted: Thu Feb 05, 2009 3:11 am Post subject: |
|
|
Partisan
Joined: 06 Oct 2008
Posts: 329
Location: India
|
Hi JonathanHill,
Yes, it will be created in the broker machine if we give the broker local directory in the Directory property of the FileOutput node. However you can use the workaroud I have mentioned in my earlier post.
_______________________________________ |
|
| Back to top |
|
|
|
|
