| Author |
Message
|
| 8corkery |
 Posted: Fri Dec 19, 2008 6:56 am Post subject: File Node |
 |
|
Newbie
Joined: 19 Dec 2008
Posts: 5
|
Wondering if someone can help me.
Im using WMB v6.1 and i had a question about the file node.
could i use the file node to parse a file with the structure similar to below and propagate out the individual <message> nodes for processing.
<root>
<Message>test1</Message>
<Message>testtwo</Message>
<Message>testthree</Message>
</root>
also what would happen is there was an error in processing one of these separate message, would they all be in error or just the individual, by this i mean are they still all related while being processed.
thanks |
|
| Back to top |
|
 |
| Vitor |
| Posted: Fri Dec 19, 2008 6:59 am Post subject: Re: File Node |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| 8corkery wrote: |
| could i use the file node to parse a file with the structure similar to below and propagate out the individual <message> nodes for processing. |
Yes. IMHO you'd need a Compute node as well.
| 8corkery wrote: |
| what would happen is there was an error in processing one of these separate message, would they all be in error or just the individual, by this i mean are they still all related while being processed. |
Depends on how you set the UOW.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| 8corkery |
| Posted: Fri Dec 19, 2008 7:05 am Post subject: Re: File Node |
|
|
Newbie
Joined: 19 Dec 2008
Posts: 5
|
| Vitor wrote: |
| 8corkery wrote: |
could i use the file node to parse a file with the structure similar to below and propagate out the individual <message> nodes for processing.
Yes. IMHO you'd need a Compute node as well. |
|
Ok i assume you are saying the file node can parse the inbound file but the compute node would have to be the one breaking the message up ?
| Vitor wrote: |
| 8corkery wrote: |
| what would happen is there was an error in processing one of these separate message, would they all be in error or just the individual, by this i mean are they still all related while being processed. |
Depends on how you set the UOW. |
|
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Dec 19, 2008 7:10 am Post subject: Re: File Node |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| 8corkery wrote: |
| Ok i assume you are saying the file node can parse the inbound file but the compute node would have to be the one breaking the message up ? |
AFAIK the File node can't propogate individual records to downstream nodes. I'll accept correction on this point.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqmatt |
| Posted: Fri Dec 19, 2008 7:36 am Post subject: Re: File Node |
|
|
Grand Master
Joined: 04 Aug 2004
Posts: 1213
Location: Hursley, UK
|
| Vitor wrote: |
| 8corkery wrote: |
| Ok i assume you are saying the file node can parse the inbound file but the compute node would have to be the one breaking the message up ? |
AFAIK the File node can't propogate individual records to downstream nodes. I'll accept correction on this point. |
FileInput can break files into separately propagated records using either fixed-length, delimiter or by using a parser.
However, it can't parse embedded records eithin a single message format, such as the <Message> elements in the original poster's example. |
|
| Back to top |
|
|
| 8corkery |
| Posted: Fri Dec 19, 2008 7:40 am Post subject: Re: File Node |
|
|
Newbie
Joined: 19 Dec 2008
Posts: 5
|
| mqmatt wrote: |
| Vitor wrote: |
| 8corkery wrote: |
| Ok i assume you are saying the file node can parse the inbound file but the compute node would have to be the one breaking the message up ? |
AFAIK the File node can't propogate individual records to downstream nodes. I'll accept correction on this point. |
FileInput can break files into separately propagated records using either fixed-length, delimiter or by using a parser.
However, it can't parse embedded records eithin a single message format, such as the <Message> elements in the original poster's example. |
i thought as much, i just wanted to check. thanks for the info |
|
| Back to top |
|
|
| 8corkery |
| Posted: Fri Dec 19, 2008 8:11 am Post subject: Re: File Node |
|
|
Newbie
Joined: 19 Dec 2008
Posts: 5
|
In this case i have been thinking of using the file node merely to detect the arrival of the file and retrieve the file location and name of the inbound file and then somehow pass this information to WTX to pick up the file and parse it. The reason i would do this is that the files maybe very very large.
Is there a way to pick up a file without reading it into memory?
For example if i parsed with XMLNSC would it read the whole tree into memory or just a single element thus being more efficient then reading the whole blog. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Dec 19, 2008 8:56 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
The FileInput node does use the streaming memory model.
I think mqmatt's comment may be a bit more precise in the use of the word "message" than you may think. |
|
| Back to top |
|
|
| 8corkery |
| Posted: Fri Dec 19, 2008 9:03 am Post subject: |
|
|
Newbie
Joined: 19 Dec 2008
Posts: 5
|
| mqjeff wrote: |
The FileInput node does use the streaming memory model.
I think mqmatt's comment may be a bit more precise in the use of the word "message" than you may think. |
So in the case we had a 500Mb file that the file handler cannot break up would it load the whole 500mb into memory ?
and if so is there a way we could stop this and only load contents as we need it like XMLNSC does (well i think thats what it does, im new to all of this) |
|
| Back to top |
|
|
| mqpaul |
| Posted: Tue Jan 06, 2009 6:04 am Post subject: Streaming means the XMLNSC parser only reads what it needs |
|
|
Acolyte
Joined: 14 Jan 2008
Posts: 66
Location: Hursley, UK
|
| Quote: |
So in the case we had a 500Mb file that the file handler cannot break up would it load the whole 500mb into memory ?
and if so is there a way we could stop this and only load contents as we need it like XMLNSC does (well i think thats what it does, im new to all of this) |
The FileInput node passes the parser an I/O stream connected to the file. So the parser might read say 32K of input, and start parsing that. Depending on what happens next, it might throw those 32K away and read the next 64K, and so on. That's for the "streaming" parsers - XMLNSC, MRM-CWF and MRM-TDS. Other parsers would read all 500 Mb.
If memory use is important, you need to take care to "prune" the message tree, as otherwise it will ultimately grow to at least as large as the source message. Look at the "Large Messaging Sample" in the toolkit to see how to do this. Of course, these days 500 Mb isn't so much.
_________________
Paul |
|
| Back to top |
|
|
|
|
