| Author |
Message
|
| iamfat |
 Posted: Mon Mar 12, 2012 1:16 am Post subject: How to read file from FTPServer or write file to FTPServe ?? |
 |
|
Apprentice
Joined: 08 Mar 2012
Posts: 32
Location: China
|
My project has a requirement that need to read file from ftpserver and write file to ftpserver.
I searched through the Broker 's Help Document. But it seems someting wrong with my flow.
I design two flow for test. One for ftp get. One for ftp put.
In the GetFlow, i read any file from FTPServer, and write it into a local directory.
And in the PutFlow, i read any file from Local directory and put it to the FTPServer.
But i configured my file node as the document. But it still can't work.
There is one property field made me confused.
If the node use ftp to process file, i don't know how to fill the InputDirectory Field in the Basic tab.
When i read file from local, this filed should be the file path.
But when i read it from the ftp server, i don't know how to fill this field in this case. Actually in the FTP tabl, there are one field named ServerDirectory. I think this should be the server file's path , for example: /tmp/proj/indir/
Currently , my flow can't read any file from ftpserver. And i don't know where should i get the logfile, i think some error info may be wrote to the log file.
My flow files are as blow.
| Code: |
<?xml version="1.0" encoding="UTF-8"?>
<ecore:EPackage xmi:version="2.0"
xmlns:xmi="http://www.omg.org/XMI" xmlns:ComIbmFileInput.msgnode="ComIbmFileInput.msgnode" xmlns:ComIbmFileOutput.msgnode="ComIbmFileOutput.msgnode" xmlns:ComIbmMQInput.msgnode="ComIbmMQInput.msgnode" xmlns:ComIbmMQOutput.msgnode="ComIbmMQOutput.msgnode" xmlns:ecore="http://www.eclipse.org/emf/2002/Ecore" xmlns:eflow="http://www.ibm.com/wbi/2005/eflow" xmlns:utility="http://www.ibm.com/wbi/2005/eflow_utility" nsURI="FtpPutFlow.msgflow"
nsPrefix="FtpPutFlow.msgflow">
<eClassifiers xmi:type="eflow:FCMComposite" name="FCMComposite_1">
<eSuperTypes href="http://www.ibm.com/wbi/2005/eflow#//FCMBlock"/>
<translation xmi:type="utility:TranslatableString" key="FtpPutFlow" bundleName="FtpPutFlow" pluginId="TestMessageFlowProject"/>
<colorGraphic16 xmi:type="utility:GIFFileGraphic" resourceName="platform:/plugin/TestMessageFlowProject/icons/full/obj16/FtpPutFlow.gif"/>
<colorGraphic32 xmi:type="utility:GIFFileGraphic" resourceName="platform:/plugin/TestMessageFlowProject/icons/full/obj30/FtpPutFlow.gif"/>
<composition>
<nodes xmi:type="ComIbmFileInput.msgnode:FCMComposite_1" xmi:id="FCMComposite_1_1" location="100,166" inputDirectory="C:\TEMP">
<translation xmi:type="utility:ConstantString" string="File Input"/>
</nodes>
<nodes xmi:type="ComIbmFileOutput.msgnode:FCMComposite_1" xmi:id="FCMComposite_1_2" location="256,166" outputDirectory="." outputFilename="*" fileFtp="true" fileFtpServer="127.0.0.1" fileFtpUser="mineftpidentity" fileFtpDirectory="/">
<translation xmi:type="utility:ConstantString" string="FTP Output"/>
</nodes>
<nodes xmi:type="ComIbmMQOutput.msgnode:FCMComposite_1" xmi:id="FCMComposite_1_3" location="368,162" queueName="TQ">
<translation xmi:type="utility:ConstantString" string="MQ Output"/>
</nodes>
<nodes xmi:type="ComIbmMQInput.msgnode:FCMComposite_1" xmi:id="FCMComposite_1_4" location="103,66" queueName="T">
<translation xmi:type="utility:ConstantString" string="MQ Input"/>
</nodes>
<connections xmi:type="eflow:FCMConnection" xmi:id="FCMConnection_1" targetNode="FCMComposite_1_2" sourceNode="FCMComposite_1_1" sourceTerminalName="OutTerminal.out" targetTerminalName="InTerminal.in"/>
<connections xmi:type="eflow:FCMConnection" xmi:id="FCMConnection_2" targetNode="FCMComposite_1_2" sourceNode="FCMComposite_1_1" sourceTerminalName="OutTerminal.EOD" targetTerminalName="InTerminal.EOD"/>
<connections xmi:type="eflow:FCMConnection" xmi:id="FCMConnection_7" targetNode="FCMComposite_1_3" sourceNode="FCMComposite_1_2" sourceTerminalName="OutTerminal.failure" targetTerminalName="InTerminal.in"/>
</composition>
<propertyOrganizer/>
<stickyBoard/>
</eClassifiers>
</ecore:EPackage>
|
| Code: |
<?xml version="1.0" encoding="UTF-8"?>
<ecore:EPackage xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:ComIbmFileInput.msgnode="ComIbmFileInput.msgnode" xmlns:ComIbmFileOutput.msgnode="ComIbmFileOutput.msgnode" xmlns:ComIbmMQOutput.msgnode="ComIbmMQOutput.msgnode" xmlns:ecore="http://www.eclipse.org/emf/2002/Ecore" xmlns:eflow="http://www.ibm.com/wbi/2005/eflow" xmlns:utility="http://www.ibm.com/wbi/2005/eflow_utility" nsURI="FtpGetFlow.msgflow" nsPrefix="FtpGetFlow.msgflow">
<eClassifiers xmi:type="eflow:FCMComposite" name="FCMComposite_1">
<eSuperTypes href="http://www.ibm.com/wbi/2005/eflow#//FCMBlock"/>
<translation xmi:type="utility:TranslatableString" key="FtpGetFlow" bundleName="FtpGetFlow" pluginId="TestMessageFlowProject"/>
<colorGraphic16 xmi:type="utility:GIFFileGraphic" resourceName="platform:/plugin/TestMessageFlowProject/icons/full/obj16/FtpGetFlow.gif"/>
<colorGraphic32 xmi:type="utility:GIFFileGraphic" resourceName="platform:/plugin/TestMessageFlowProject/icons/full/obj30/FtpGetFlow.gif"/>
<composition>
<nodes xmi:type="ComIbmFileInput.msgnode:FCMComposite_1" xmi:id="FCMComposite_1_2" location="55,150" inputDirectory="." retryMechanism="shortAndLongRetry" retryThreshold="10" shortRetryInterval="10" longRetryInterval="20" fileFtp="true" fileFtpServer="127.0.0.1" fileFtpUser="mineftpidentity" fileFtpDirectory="/" fileFtpScanDelay="5">
<translation xmi:type="utility:ConstantString" string="File Input"/>
</nodes>
<nodes xmi:type="ComIbmFileOutput.msgnode:FCMComposite_1" xmi:id="FCMComposite_1_3" location="207,150" outputDirectory="C:\TEMP" outputFilename="ftp_local_out.txt">
<translation xmi:type="utility:ConstantString" string="File Output"/>
</nodes>
<nodes xmi:type="ComIbmMQOutput.msgnode:FCMComposite_1" xmi:id="FCMComposite_1_1" location="209,81" queueName="TQ">
<translation xmi:type="utility:ConstantString" string="MQ Output"/>
</nodes>
<connections xmi:type="eflow:FCMConnection" xmi:id="FCMConnection_1" targetNode="FCMComposite_1_3" sourceNode="FCMComposite_1_2" sourceTerminalName="OutTerminal.out" targetTerminalName="InTerminal.in"/>
<connections xmi:type="eflow:FCMConnection" xmi:id="FCMConnection_2" targetNode="FCMComposite_1_3" sourceNode="FCMComposite_1_2" sourceTerminalName="OutTerminal.EOD" targetTerminalName="InTerminal.EOD"/>
<connections xmi:type="eflow:FCMConnection" xmi:id="FCMConnection_3" targetNode="FCMComposite_1_1" sourceNode="FCMComposite_1_2" sourceTerminalName="OutTerminal.failure" targetTerminalName="InTerminal.in"/>
<connections xmi:type="eflow:FCMConnection" xmi:id="FCMConnection_4" targetNode="FCMComposite_1_1" sourceNode="FCMComposite_1_2" sourceTerminalName="OutTerminal.catch" targetTerminalName="InTerminal.in"/>
</composition>
<propertyOrganizer/>
<stickyBoard/>
</eClassifiers>
</ecore:EPackage>
|
Did i missed some filed in the properties view?
Please give me some advise! Thank you! |
|
| Back to top |
|
 |
| iamfat |
| Posted: Mon Mar 12, 2012 1:18 am Post subject: |
|
|
Apprentice
Joined: 08 Mar 2012
Posts: 32
Location: China
|
And PS.
I used the Apache FtpServer as the FtpServer. and also had been tested before deploy the flow. It worked fine in command line.  |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Mar 12, 2012 2:39 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
You don't need two flows to do this. One flow makes more sense.
The fileinput node passes file data into the flow (in memory) and the file output passes flow data from the flow (in memory) to a file.
You can process one record at a time in memory or a whole file, depending how you want to handle it. You flow will also be told when it is the final record (end of data terminal).
Don't think of each of these nodes as being a file transfer in the traditional sense.
Look at the help to see what each property on a node means. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Mar 12, 2012 2:48 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
A service trace will show the ftp commands that are executed, but you will have to spend some time looking for them.
The way the FileInput node uses FTP is that it does two steps. the first is that it copies the file from the ftp directory into the local directory. then it reads the file from the local directory.
So, yes, the Server Path property is the place that it will look for the file on the ftp server, and then it will put this file into the place mentioned on the basic tab and then read it from there.
The FileOutput does the reverse, it writes the file to the local directory and then uses ftp to move it to the remote directory. |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Mar 12, 2012 3:17 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
However unless you want to retain a local copy of the files, the file nodes would normally be set up to delete these interim files when the node commits the transaction.
So yes there is a local copy for transactional purposes primarily. It would be odd (and unnecessary) to have two flows as the OP has described. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Mar 12, 2012 3:53 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| zpat wrote: |
| So yes there is a local copy for transactional purposes primarily. It would be odd (and unnecessary) to have two flows as the OP has described. |
Well, yes, except that I read the scenario as a strict file transfer setup, where the intent is to move the file from one FTP site to another.
Which, as you say, can be handled with a single FileInput...->...->FileOutput flow, but one could as well view it as two files for purposes of workload management - i.e. to isolate the processing from the in side from the out side, so that if something goes wrong with one FTP server it doesn't affect the other. |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Mar 12, 2012 5:50 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| Could lead to a build up of files on the WMB box, if the input side was not transactionally linked to the output side. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Mar 12, 2012 6:02 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| zpat wrote: |
| Could lead to a build up of files on the WMB box, if the input side was not transactionally linked to the output side. |
Sure, but presumably this would be a feature of this kind of buffered transfer, rather than a defect.
It's like you want a transmission queue... |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Mar 12, 2012 6:04 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| zpat wrote: |
| Could lead to a build up of files on the WMB box, if the input side was not transactionally linked to the output side. |
Sure, but presumably this would be a feature of this kind of buffered transfer, rather than a defect.
It's like you want a transmission queue... |
Like FTE.....? 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Mar 12, 2012 6:07 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| mqjeff wrote: |
| zpat wrote: |
| Could lead to a build up of files on the WMB box, if the input side was not transactionally linked to the output side. |
Sure, but presumably this would be a feature of this kind of buffered transfer, rather than a defect.
It's like you want a transmission queue... |
Like FTE.....? |
Yes, that's correct. As FTE becomes more available in the market place in general, there will be less need to build this kind of setup using two broker message flows. |
|
| Back to top |
|
|
| iamfat |
| Posted: Tue Mar 13, 2012 10:59 pm Post subject: |
|
|
Apprentice
Joined: 08 Mar 2012
Posts: 32
Location: China
|
Great!! Thank you for all your help!
I just run through these two flows.
The root cause is the "Input Directory" field in "Basic" tab.
It's value is "." as default. But when we use the FTP, this value should be configured as well. And this directory must exist.
[If the input directory does not exist, no files are processed, even if you are processing files over FTP or SFTP.]
So thank you again.~
It's working now.   |
|
| Back to top |
|
|
|
|
