| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| The output file is one XML record, 400 MB.....alrighty then |
« View previous topic :: View next topic » |
| Author |
Message
|
| PeterPotkay |
 Posted: Mon Sep 01, 2014 3:55 am Post subject: |
 |
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
I'm on vacation and doing my best to stay offline but wanted to give an update.
Progress is being made. Latest runs take less than a minute,  , but the XML output doesn't look quite right with the way the elemenets are nested. The developers are working on that.
Memory is still a problem. Like fjb_saper said:
| Quote: |
Just tackling the index issue will only make the broker run out of memory so much quicker....
|
An input record set of 150K records finishes in like 15 seconds (wow), but the DataFlowEngine process takes about 5.5GB of memory which it then releases a few seconds after the file is done, dropping down to about 300 MB.
But when they try with the 200K record set, it blows up in about 20 seconds with a heap error when the memory for the DFE goes over 6 GB. Not sure if its the overall count that is the issue (200K), or if in that larger record set there is one TheCompany that has an extremely large set of TheTransactions, and since they are currently ejecting one TheCompany at a time with all its TheTransactions, its using up too much memory.
I asked the developers if they can successfully eject one TheCompany at a time, why not take down another level and eject one TheTransaction at a time.
I'll update again when I hear more from the office.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
 |
| mqsiuser |
| Posted: Mon Sep 01, 2014 1:25 pm Post subject: |
|
|
Yatiri
Joined: 15 Apr 2008
Posts: 637
Location: Germany
|
| PeterPotkay wrote: |
| but the XML output doesn't look quite right with the way the elemenets are nested. The developers are working on that. |
Let's be honest: Keeping track (in your mind) of the references is harder than using indexes. A good naming (rIn..., rIn..., rIn..., rOut..., rOut..., rOut...) and -for me- the debugger's variables view help.
| PeterPotkay wrote: |
| An input record set of 150K records finishes in like 15 seconds (wow) |
That sounds reasonable (maybe minutes on a laptop, but seconds on a server, if there's no other bottleneck).
| PeterPotkay wrote: |
| heap error when the memory for the DFE goes over 6 GB. |
Memory optimization can be done on the input and output root:
Input root:
You can delete/free the memory of the (already read (and processed)) input message/tree. I never did so (don't ask me  ), but it is described in the "large message processing example" (or similar) and I think it is kind of "deleting previous in-children".
Output root:
(I think as you describe) emitting smaller messages.
BUT: Why not (already) split the (initial) msg then?
ExecGroups just abend (block/hang) when going out of memory, and I'd want to securely eliminate that risk.
_________________
Just use REFERENCEs |
|
| Back to top |
|
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
