| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Converting mutiple file records to a huge XML Document |
« View previous topic :: View next topic » |
| Author |
Message
|
| PEPERO |
 Posted: Wed Jun 14, 2017 8:01 am Post subject: Converting mutiple file records to a huge XML Document |
 |
|
Disciple
Joined: 30 May 2011
Posts: 177
|
Hi all;
I've to design a flow reading from an input file , record by record , and writting the whole records as one huge document in an output file.
The input file contains c structure records , one header and multiple (hundreds of thousands) body records. The output file contains a huge XML documnet wich contains a header element and multiple body elements.
So what is the best practice to have an efficient flow? |
|
| Back to top |
|
 |
| Vitor |
| Posted: Wed Jun 14, 2017 8:28 am Post subject: Re: Converting mutiple file records to a huge XML Document |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| PEPERO wrote: |
| So what is the best practice to have an efficient flow? |
Redesign the requirement 
There's no way you're going the parse that monster of an XML document in any kind of time or resource efficient way. You're also introducing affinity where it doesn't exist (assuming a lack of affinity in the input file).
Consider the scenario where whatever's parsing your output XML crashes; perhaps because it runs out of memory. Processing would have to restart at the top of the document and even assuming you can just opaque parse through the document to the place you failed because some external system holds the last operation, that's a lot of time. If no external system holds the data and you just have to restart at the top, you could be there for days trying to find enough memory to finish the XML.
Consider instead a series of discrete files, either the original fixed length or XML format. Not only are you much less likely to run the XML parser out of memory, it's easy to see (based on what files are unprocessed) where you are and where you need to restart.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
