| Author |
Message
|
| mqbrks |
 Posted: Tue Jan 31, 2017 6:41 am Post subject: File read node problem |
 |
|
Voyager
Joined: 17 Jan 2012
Posts: 75
|
Hi ,
I have multiple files to read by file read node, so I am looping the files in a java compute node and sending each file through local environment to the file node (using propagate). I have facing parsing issues on the file read node, where the parser is set to DFDL and message (DFDL msg). Doesn't file read node take the information from file(set in local env), instead does it take the info from the incoming message ?
Thank you! |
|
| Back to top |
|
 |
| Vitor |
| Posted: Tue Jan 31, 2017 7:09 am Post subject: Re: File read node problem |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqbrks wrote: |
| I have multiple files to read by file read node, so I am looping the files in a java compute node and sending each file through local environment to the file node (using propagate). |
So what are you sending? The file names?
| mqbrks wrote: |
| I have facing parsing issues on the file read node, where the parser is set to DFDL and message (DFDL msg). Doesn't file read node take the information from file(set in local env), instead does it take the info from the incoming message ? |
If you're actually sending the local environment with the propagated message tree.
What exactly are the parsing issues? Using the wrong DFDL set?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jan 31, 2017 7:10 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
I also question the wisdom of this design. Why do you need to loop through all these files like this? Why not just read them?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqbrks |
| Posted: Tue Jan 31, 2017 7:32 am Post subject: Re: File read node problem |
|
|
Voyager
Joined: 17 Jan 2012
Posts: 75
|
| Quote: |
So what are you sending? The file names? |
Just the filename and directory , the trigger msg which has been coming form mq input node.
| Quote: |
What exactly are the parsing issues? Using the wrong DFDL set? |
may be I need to set the incoming msg as some dummy DFDL msg and drop it later. |
|
| Back to top |
|
|
| mqbrks |
| Posted: Tue Jan 31, 2017 7:34 am Post subject: |
|
|
Voyager
Joined: 17 Jan 2012
Posts: 75
|
| Vitor wrote: |
| I also question the wisdom of this design. Why do you need to loop through all these files like this? Why not just read them? |
I am assuming that file read node reads only one file, if there are multiple files, I have to loop it and send using propagate? |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jan 31, 2017 7:36 am Post subject: Re: File read node problem |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqbrks wrote: |
| Just the filename and directory , the trigger msg which has been coming form mq input node. |
That's the best way you could come up with for the halting file problem? Why not use a FileInput node and have whatever's sending the trigger MQ message move the file to the right place?
| mqbrks wrote: |
| may be I need to set the incoming msg as some dummy DFDL msg and drop it later. |
Maybe you need to explain what the parsing error is, like I asked.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jan 31, 2017 7:37 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqbrks wrote: |
| Vitor wrote: |
| I also question the wisdom of this design. Why do you need to loop through all these files like this? Why not just read them? |
I am assuming that file read node reads only one file, if there are multiple files, I have to loop it and send using propagate? |
Why do all of these files need to be read by a single (same) flow instance?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqbrks |
| Posted: Tue Jan 31, 2017 7:53 am Post subject: |
|
|
Voyager
Joined: 17 Jan 2012
Posts: 75
|
| Quote: |
Why do all of these files need to be read by a single (same) flow instance? |
All these files have same pattern, just there are multiple files with the same pattern. |
|
| Back to top |
|
|
| mqbrks |
| Posted: Tue Jan 31, 2017 7:54 am Post subject: Re: File read node problem |
|
|
Voyager
Joined: 17 Jan 2012
Posts: 75
|
| Quote: |
That's the best way you could come up with for the halting file problem? Why not use a FileInput node and have whatever's sending the trigger MQ message move the file to the right place? |
Even I was thinking about this idea, just few minutes back, will check with my peers and see if this is possible.[/quote] |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jan 31, 2017 8:24 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqbrks wrote: |
| Quote: |
Why do all of these files need to be read by a single (same) flow instance? |
All these files have same pattern, just there are multiple files with the same pattern. |
That's no reason to loop round them with a FileRead node and burn memory.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqbrks |
| Posted: Fri Feb 03, 2017 7:18 am Post subject: |
|
|
Voyager
Joined: 17 Jan 2012
Posts: 75
|
| Quote: |
That's no reason to loop round them with a FileRead node and burn memory. |
Thank you vitor for all your support! I have resolved the parsing error, the issue was with the file, I fixed it 
And we are using the design as two flows.
One which triggers the flow and moves the files to local dropzone.
Other one which polls files using file input node.
So we finally got to use the file input node, which made things a lot easier.
Thank you once again ! |
|
| Back to top |
|
|
|
|
