| Author |
Message
|
| fszostak |
 Posted: Thu Feb 24, 2011 7:54 am Post subject: Reading large XML files with FileInput |
 |
|
Acolyte
Joined: 09 Feb 2011
Posts: 64
Location: Curitiba, Brazil
|
Hi All!
I need read Parsed Record Sequence of XML file.
For example, the XML structure is:
| Code: |
<?xml version="1.0"?>
<rootx>
<group>
<a>111</a>
<b>222</b>
<details>
<d1>aaa</d1>
<d2>aaa</d2>
<d3>aaa</d3>
</details>
<details>
<d1>aaa</d1>
<d2>aaa</d2>
<d3>aaa</d3>
</details>
<details>
<d1>aaa</d1>
<d2>aaa</d2>
<d3>aaa</d3>
</details>
</group>
<group>
<a>111</a>
<b>222</b>
<details>
<d1>aaa</d1>
<d2>aaa</d2>
<d3>aaa</d3>
</details>
<details>
<d1>aaa</d1>
<d2>aaa</d2>
<d3>aaa</d3>
</details>
<details>
<d1>aaa</d1>
<d2>aaa</d2>
<d3>aaa</d3>
</details>
</group>
</rootx> |
I would like read one <group> for each flow execution, because my file size have around 60Mb.
I try set MRM parser and Parsed Record Sequence for record detection, but don't work.
With XMLNSC and "Whole file" for record detection works but load all file content to memory, use around 1Gb to transform the message.
Any good idea?
Thanks
Szostak
_________________
WMB 6.1.005 |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu Feb 24, 2011 8:23 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
The principles of dealing with large files are the same as were discussed in your previous post here - which is suspiciously close to a duplicate of this!
| fszostak wrote: |
| I try set MRM parser and Parsed Record Sequence for record detection, but don't work. |
Reading XML as if it's TDS never does. Read XML with XMLNSC.
| fszostak wrote: |
| Any good idea? |
Follow the advice given in your previous post, or buy more memory.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Feb 24, 2011 8:46 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
XMLNSC parser is a streaming parser.
If you read the whole file, and do not validate it and do not set parse timing to cause a full parse, you can with some care navigate through the sub-segments of the input message and propagate each out or transform it WITHOUT incurring a full parse of the message requiring enough memory for a full tree.
Look at the large message sample that ships with the product. |
|
| Back to top |
|
|
| fszostak |
| Posted: Thu Feb 24, 2011 9:21 am Post subject: |
|
|
Acolyte
Joined: 09 Feb 2011
Posts: 64
Location: Curitiba, Brazil
|
| Vitor wrote: |
The principles of dealing with large files are the same as were discussed in your previous post here - which is suspiciously close to a duplicate of this!
| fszostak wrote: |
| I try set MRM parser and Parsed Record Sequence for record detection, but don't work. |
Reading XML as if it's TDS never does. Read XML with XMLNSC.
| fszostak wrote: |
| Any good idea? |
Follow the advice given in your previous post, or buy more memory. |
Hi Vitor!
Its another situation, now i reading XML file and writing sequential cobol file, now response file transformation. 
Thanks
Szostak
_________________
WMB 6.1.005 |
|
| Back to top |
|
|
| fszostak |
| Posted: Thu Feb 24, 2011 9:35 am Post subject: |
|
|
Acolyte
Joined: 09 Feb 2011
Posts: 64
Location: Curitiba, Brazil
|
| mqjeff wrote: |
XMLNSC parser is a streaming parser.
If you read the whole file, and do not validate it and do not set parse timing to cause a full parse, you can with some care navigate through the sub-segments of the input message and propagate each out or transform it WITHOUT incurring a full parse of the message requiring enough memory for a full tree.
Look at the large message sample that ships with the product. |
Hi mqjeff,
I will try this.
Thanks
Szostak
_________________
WMB 6.1.005 |
|
| Back to top |
|
|
| fszostak |
| Posted: Thu Feb 24, 2011 12:35 pm Post subject: |
|
|
Acolyte
Joined: 09 Feb 2011
Posts: 64
Location: Curitiba, Brazil
|
Fixed! 
Thanks Mqjeff!
I'm reading file with FileInput with "Whole file" and using sample idea, the memory usage has reducing to 500mb.
_________________
WMB 6.1.005 |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Feb 24, 2011 12:39 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| fszostak wrote: |
| Thanks Mqjeff! |
You can thank me by fixing
| fszostak wrote: |
| WMB 6.1.005 |
So that it correctly reads
....
 |
|
| Back to top |
|
|
|
|
