| Author |
Message
|
| giorginus80 |
 Posted: Wed May 06, 2009 1:41 am Post subject: Big problem with memory, please Help |
 |
|
Centurion
Joined: 08 Jul 2008
Posts: 124
Location: Rome, Italy
|
Hi,
I have a very big problem with my memory usage, I used all the techincs descripted by IBM (http://www.ibm.com/developerworks/websphere/library/techarticles/0505_storey/0505_storey.html and another article IBM 'Memory and performance considerations when handling large
messages in the broker'). I expose my problem:
The environment is:
Aix 5.3 (level 9) (But happen with Red Hat Linux too)
Message broker 6.1.0.3
MQ 6.0.2.5
I have a big xml message (20mb on filesystem) with this strutcture:
| Code: |
<credits>
<credit>
<StructA>
<ElementA1>Value A1<ElementA1>
<ElementA2>Value A2<ElementA2>
<ElementA3>Value A3<ElementA3>
<ElementA4>Value A4<ElementA4>
<ElementA5>Value A5<ElementA5>
</StructA>
<StructB>
<ElementB1>Value B1<ElementB1>
<ElementB2>Value B2<ElementB2>
<ElementB3>Value B3<ElementB3>
<ElementB4>Value B4<ElementB4>
<ElementB5>Value B5<ElementB5>
<ElementB6>Value B6<ElementB6>
<ElementB7>Value B7<ElementB7>
</StructB>
<StructC>
<ElementC1>Value C1<ElementC1>
<ElementC2>Value C2<ElementC2>
<ElementC3>Value C3<ElementC3>
<ElementC4>Value C4<ElementC4>
<ElementC5>Value C5<ElementC5>
<ElementC6>Value C1<ElementC6>
<ElementC7>Value C2<ElementC7>
<ElementC8>Value C3<ElementC8>
<ElementC9>Value C4<ElementC9>
<ElementC10>Value C5<ElementC10>
</StructC>
</credit>
<credit>
...
...
</credit>
<credit>
...
...
</credit>
...
...
</credits>
|
I have no problem to parse. I need to split every credit ( to do post operations, like db operation) and at the end I need all generated primary keys. For this I used Aggregation. These are the steps:
| Code: |
CREATE COMPUTE MODULE SplitCredits
CREATE FUNCTION Main() RETURNS BOOLEAN
BEGIN
DECLARE InputMessage REFERENCE TO InputRoot.MRM.ns:body.ns:credits.ns:credit[1];
DECLARE Inner_No INTEGER 0;
WHILE LASTMOVE(InputMessage)
DO
IF Inner_No >= 1 THEN
DELETE PREVIOUSSIBLING OF InputMessage;
END IF;
SET Inner_No = Inner_No + 1;
SET OutputRoot.Properties = InputRoot.Properties
SET OutputRoot.XMLNSC.ns:credit = InputMessage;
PROPAGATE;
MOVE InputMessage NEXTSIBLING;
END WHILE;
RETURN FALSE;
END;
END MODULE;
|
and in the esql node after the AggregateReply I do:
| Code: |
CREATE COMPUTE MODULE aggregateCredits
CREATE FUNCTION Main() RETURNS BOOLEAN
BEGIN
CREATE LASTCHILD OF OutputRoot DOMAIN 'XMLNSC';
DECLARE next INTEGER 0;
DECLARE repliesIn REFERENCE TO InputRoot.ComIbmAggregateReplyBody.[1];
DECLARE repliesOut REFERENCE TO OutputRoot.XMLNSC;
DECLARE J INTEGER;
SET J = CARDINALITY(InputRoot.ComIbmAggregateReplyBody.*[]);
WHILE LASTMOVE(repliesIn) DO
IF next >= 1 THEN
DELETE PREVIOUSSIBLING OF repliesIn;
END IF;
SET key = repliesIn.XMLNSC.ns:creditExt.ns:progPosizione;
SET repliesOut.ns:answerCredits.ns:key[next] = key;
SET next = next + 1;
MOVE repliesIn NEXTSIBLING;
END WHILE;
RETURN TRUE;
END;
END MODULE;
|
While do this aggregation the memory of the process DataFlowEngine goes to 1.5gb of swap and then crash. (I have an abend file in /var/mqsi/common/errors) Failed to allocate Memory. If I put the environment
| Code: |
LDR_CNTRL=0x90000000
|
it's the same the process just use some more than 1.5gb and goes to use 2.5gb but crash at the end. But why all this memory used? Is there any error in my nodes? |
|
| Back to top |
|
 |
| WMBDEV1 |
| Posted: Wed May 06, 2009 1:52 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
Sorry to hear the document I sent did not help!
Looking at your ESQL you are still not using the large memory handling techniques for the output tree that you build up. This is contained in the document that I sent you.
Just as a test, can you try commenting out the code that builds the output tree and see how much memory is used then (to confirm the input tree is being handled nicely).
FYI, I managed to get a 200MB message compressed down to 30MB to be decompressed and manipulated by the broker in about 700MB of memory so I know this is doable. I could have used less memory but it then becomes a trade off between supportability and memory usage. |
|
| Back to top |
|
|
| giorginus80 |
| Posted: Wed May 06, 2009 2:01 am Post subject: |
|
|
Centurion
Joined: 08 Jul 2008
Posts: 124
Location: Rome, Italy
|
| WMBDEV1 wrote: |
Sorry to hear the document I sent did not help!
Looking at your ESQL you are still not using the large memory handling techniques for the output tree that you build up. This is contained in the document that I sent you.
Just as a test, can you try commenting out the code that builds the output tree and see how much memory is used then (to confirm the input tree is being handled nicely).
FYI, I managed to get a 200MB message compressed down to 30MB to be decompressed and manipulated by the broker in about 700MB of memory so I know this is doable. I could have used less memory but it then becomes a trade off between supportability and memory usage. |
It's very strange because I used the technics descibed in the document you sent, now I'm testing without building an output message. Anyway the original file is 20mb and it's just a test case, beacause at runtime we'll have thousand files oh 75mb and I don't know how to do if the WMB crash with 20mb!! I'm testing without buliding output now to see if the memory goes up! (initially monitoring the memory, it goes up very fast when the reply queue goes empty). Anyway the message splitted aren't large they have only 1 field with the primary key of all the credit inserted!
MODIFIED:
I just finished the test commenting the output bulding, and the memory goes to 900mb (anyway very strange use of memory) and no crash. So I can't build an output message. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed May 06, 2009 2:35 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
To consume even less memory try stuff like this: instead of
| Quote: |
| DECLARE InputMessage REFERENCE TO InputRoot.MRM.ns:body.ns:credits.ns:credit[1] |
use
| Code: |
DECLARE InputMessage REFERENCE TO InputRoot.MRM.ns:body.ns:credits;
Move InputMessage FirstChild Namespace ns name 'credit';
If lastmove(InputMessage) ....
......
MOVE InputMessage nextsibling repeat name;
|
This avoids any mention of cardinality and should avoid having to calculate cardinality. Use the same to build the output message. 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Wed May 06, 2009 3:43 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
900MB is way too high.
I've also noticed that you are trying to delete from the inputroot tree but this is read only. You need to create a copy of the bitstream in the environment and then delete the memory from there.
Again, this is contained in the document I sent and I suggest that you have another read  |
|
| Back to top |
|
|
| farubino |
| Posted: Wed May 06, 2009 5:25 am Post subject: |
|
|
Novice
Joined: 17 Mar 2009
Posts: 13
|
| WMBDEV1 wrote: |
900MB is way too high.
I've also noticed that you are trying to delete from the inputroot tree but this is read only. You need to create a copy of the bitstream in the environment and then delete the memory from there.
Again, this is contained in the document I sent and I suggest that you have another read |
Now I try to copy it in the environment. But in the example you sent, there is something tring to delete the inputroot
| Code: |
DECLARE currentRecord REFERENCE TO InputRoot.XML.TestCase.Record[1];
DECLARE previousRecord REFERENCE TO InputRoot.XML.TestCase.Record[1];
WHILE LASTMOVE(currentRecord) DO
CALL CopyMessageHeaders();
SET OutputRoot.XML.TestCase.Record = currentRecord;
PROPAGATE;
MOVE previousRecord TO currentRecord;
MOVE currentRecord NEXTSIBLING NAME 'Record';
DELETE FIELD previousRecord;
END WHILE;
RETURN FALSE;
|
|
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Wed May 06, 2009 6:05 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
| Remind me which example.... the IBM doc? if so its a little naughty! |
|
| Back to top |
|
|
| farubino |
| Posted: Wed May 06, 2009 6:14 am Post subject: |
|
|
Novice
Joined: 17 Mar 2009
Posts: 13
|
| WMBDEV1 wrote: |
| Remind me which example.... the IBM doc? if so its a little naughty! |
I took it from "Memory and performance considerations when handling large
messages in the broker"
Anyway I replaced my code with:
| Code: |
DECLARE repliesIn REFERENCE TO InputRoot.ComIbmAggregateReplyBody.[1];
CREATE FIELD Environment.Variables;
CREATE LASTCHILD OF Environment.Variables DOMAIN('XML');
DECLARE currentRecord REFERENCE TO Environment.Variables;
DECLARE counter INTEGER 1;
WHILE LASTMOVE(repliesIn) DO
SET currentRecord.XML.crediti.credito[counter] = repliesIn.XMLNSC.ns:creditoExt;
SET counter = counter + 1;
MOVE repliesIn NEXTSIBLING;
END WHILE;
|
and I consume the same memory. If from this I remove "SET currentRecord.XML.crediti.credito[counter] = repliesIn.XMLNSC.ns:creditoExt; " the memory isnt' consumed.
Maybe have I to work with BLOB domain? |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed May 06, 2009 6:17 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| farubino wrote: |
| Maybe have I to work with BLOB domain? |
Or use CREATE NEXTSIBLING / LASTCHILD rather than a counter, eliminating the cardinality.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| giorginus80 |
| Posted: Wed May 06, 2009 6:22 am Post subject: |
|
|
Centurion
Joined: 08 Jul 2008
Posts: 124
Location: Rome, Italy
|
| WMBDEV1 wrote: |
| Remind me which example.... the IBM doc? if so its a little naughty! |
Yes I took from "Memory and performance considerations when handling large
messages in the broker" |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Wed May 06, 2009 7:21 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
That looks like the example on page 5. If so, the following text introduces the code....
"Ideally what you would want to write is the following" and the tex after it explains why it cant be done. Is this the example to which you are referring? If not please tell me where it is. |
|
| Back to top |
|
|
| giorginus80 |
| Posted: Wed May 06, 2009 7:26 am Post subject: |
|
|
Centurion
Joined: 08 Jul 2008
Posts: 124
Location: Rome, Italy
|
| WMBDEV1 wrote: |
That looks like the example on page 5. If so, the following text introduces the code....
"Ideally what you would want to write is the following" and the tex after it explains why it cant be done. Is this the example to which you are referring? If not please tell me where it is. |
Yes sorry you are right.
Anyway if I read InputRoot.ComIbmAggregateReplyBody without bulding output message I don't get flood memory problem, may be that the messages in the ComIbmAggregateReplyBody are all in XMLNSC domain? every ComIbmAggregateReplyBody.[] is an XMLNSC. |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Wed May 06, 2009 7:29 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
Yeah but 900MB is way too much memory to be using. I really recommend using the memory management techniques in this instance.
I dont think the domain being used is the main issue you have here. Put it into the environment and watch the memory then! |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed May 06, 2009 7:38 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
And make sure you have the input node set to parse on demand
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| giorginus80 |
| Posted: Thu May 07, 2009 6:20 am Post subject: |
|
|
Centurion
Joined: 08 Jul 2008
Posts: 124
Location: Rome, Italy
|
| WMBDEV1 wrote: |
Yeah but 900MB is way too much memory to be using. I really recommend using the memory management techniques in this instance.
I dont think the domain being used is the main issue you have here. Put it into the environment and watch the memory then! |
Ok I solved today. I removed the Aggregation logic and at the end of the split I put my information in the Environment tree. Now with a file of 80mb my swap memory goes to 1gb, and don't crash, anyway is it very much memory usage ? |
|
| Back to top |
|
|
|
|
