| Author |
Message
|
| new2z |
 Posted: Mon May 20, 2013 8:05 pm Post subject: Specify end of file to FileOuput Node when using FileRFead |
 |
|
Novice
Joined: 16 May 2013
Posts: 15
|
Dear All
I have a question about FileRead Node.
Here is my requirement
1. A dummy message is dropped on to a queue
2. This triggers a message flow which reads a file from the file system
3. The file has multiple records
4. Each of these records are read and processed using the fileRead node
5. Post Processing, all the records are again flushed to a File
6. This is done using a Fileoutput Node
This is how I have put my message flow
MQInput --> FileRead --> Compute --> FileOutput
The problem that I am facing is, I am unable to let Fileoutput node know that I have reached "end of the file". If i were using FileInput node, then I would have connected "End of Data" terminal to the "Finish File" terminal of the FileOuput Node.
I am unable to think of something in similar lines with FileRead Node. Can be "Finish File In" or "Finish File Out" terminal be helpful in this regard?
Kindly advice
Thank You |
|
| Back to top |
|
 |
| Esa |
| Posted: Mon May 20, 2013 10:33 pm Post subject: Re: Specify end of file to FileOuput Node when using FileRFe |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| new2z wrote: |
I am unable to think of something in similar lines with FileRead Node. Can be "Finish File In" or "Finish File Out" terminal be helpful in this regard? |
Why don't you try wiring "File Read"."Finish file out" to "FileOutput"."Finish File" and testing yourself what happens? |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue May 21, 2013 5:03 am Post subject: Re: Specify end of file to FileOuput Node when using FileRFe |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| new2z wrote: |
| If i were using FileInput node, then I would have connected "End of Data" terminal to the "Finish File" terminal of the FileOuput Node. |
Why not use the FileInput node? It seems like all this message does is indicate the file is available; there are other methods.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| new2z |
| Posted: Tue May 21, 2013 8:13 am Post subject: |
|
|
Novice
Joined: 16 May 2013
Posts: 15
|
Thanks Esa, Vitor
I cannot use FileInput Node because I am picking up the File in the middle of the flow
Moreover, the incoming message in the MQInput acts as a trigger which tells me that it is time that I start the file processing now.
We are doing the file processing in a batch. I am getting a trigger (from external agent) to start my batch. An Autosys job sends the trigger (an MQ message) to the MQInput, then my flow starts.
Did i make sense?
Can i wire the "Finish File Out" terminal of the FileRead node to the "Finish" Terminal of the Fileoutput node?
I read the fileread node from the infocenter. It doesn't speak explicitly about the use of "Finish File Out" as an "End of Data" indicator. Can you please give me some link to read or may be some ideas? I want to read a little, then start doing experiments.
Kindly advise
Thank You |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue May 21, 2013 8:37 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| new2z wrote: |
| I cannot use FileInput Node because I am picking up the File in the middle of the flow |
Are you doing this because the processing of the data in the file requires it, or because your current design has an MQInput node on the front of it?
| new2z wrote: |
Moreover, the incoming message in the MQInput acts as a trigger which tells me that it is time that I start the file processing now.
We are doing the file processing in a batch. I am getting a trigger (from external agent) to start my batch. An Autosys job sends the trigger (an MQ message) to the MQInput, then my flow starts.
Did i make sense? |
Yes, and my point is that there are other ways a flow which processes a file can have the file's availability indicated to it. Consider this: the file is assembled in a directory other than the one being monitored by the FileInput node or with a file name pattern the FileInput node isn't looking for. Once complete and ready to be processed, the file is moved or renamed as applicable, the flow picks it up and no trigger message is required.
Other methods are possible and may be better for your specific use case.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| new2z |
| Posted: Tue May 21, 2013 9:27 am Post subject: |
|
|
Novice
Joined: 16 May 2013
Posts: 15
|
Thanks Vitor
I understand what you are saying. In a shorter term, I cannot change things. There are certain guidelines which have been laid out by my predecessors and I have to abide by it.
I will change things going ahead, but it will take time
If i had no option (to use FileInput) but to use a FileRead node, then how would I do convey "End of Data" to the FileOutput Node?
Will it work, if i wire the "Finish File Out" terminal of the FileRead node to the "Finish" Terminal of the Fileoutput node? |
|
| Back to top |
|
|
| Cogito-Ergo-Sum |
| Posted: Tue May 21, 2013 9:35 am Post subject: |
|
|
Master
Joined: 07 Feb 2006
Posts: 293
Location: Bengaluru, India
|
Your Compute node could look for values in the LocalEnvironment.File.Read to decide if end of file has happened or not.
_________________
ALL opinions are welcome.
-----------------------------
Debugging tip:When you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth.
---Sherlock Holmes |
|
| Back to top |
|
|
| Cogito-Ergo-Sum |
| Posted: Tue May 21, 2013 9:37 am Post subject: |
|
|
Master
Joined: 07 Feb 2006
Posts: 293
Location: Bengaluru, India
|
| Esa wrote: |
| Why don't you try wiring "File Read"."Finish file out" to "FileOutput"."Finish File" and testing yourself what happens? |
That is a valid suggestion except for those cases where there are some explicit steps to be taken when end of file happens (what if file had zero records, etc.).
_________________
ALL opinions are welcome.
-----------------------------
Debugging tip:When you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth.
---Sherlock Holmes |
|
| Back to top |
|
|
| Esa |
| Posted: Tue May 21, 2013 11:51 pm Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| new2z wrote: |
Will it work, if i wire the "Finish File Out" terminal of the FileRead node to the "Finish" Terminal of the Fileoutput node? |
What circumstances prevent you from testing it yourself? |
|
| Back to top |
|
|
|
|
