| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Memory management and use of Environment tree |
« View previous topic :: View next topic » |
| Author |
Message
|
| JohnMetcalfe |
 Posted: Fri Sep 17, 2004 6:13 am Post subject: Memory management and use of Environment tree |
 |
|
Apprentice
Joined: 02 Apr 2004
Posts: 40
Location: Edinburgh, Scotland
|
We have been running into memory problems processing large (11MB) messages through flows with multiple compute nodes.
After we refer to anything in the OutputRoot from within a Compute node, WMQI seems to build a output tree, which for a 11MB message, is some 150MB in size. After this, every compute node appears to consume the 150MB, as if there is a full parse output tree created (note this is our interpretation of what is happening, based on our test results).
The net result is flows that process large message use up all the memory available to the execution group (on UNIX this is supposed to be 512MB by default, but we are seeing our execution group retarting after 390MB), and the execution group restarts.
We have been kicking about ideas as to how to reduce the memory usage of these flows. One possibility we are exploring is to take a copy of the message into the Environment tree at the start, using code like this:
| Code: |
CREATE LASTCHILD OF Environment DOMAIN 'XML' NAME 'MsgBody';
SET Environment.MsgBody = InputBody; |
All subsequent nodes then act on this message copy in the Environment area - note they don't therefore need to copy the message body across. The last compute node copies the message out of Environment and into OutputRoot, as follows
| Code: |
SET OutputRoot.XML = Environment.MsgBody;
DELETE FIELD Environment.MsgBody; |
Our theory was that this will be much more efficient in terms of memory usage, as we will should only have one message tree built i.e the one in the Environment area.
Some initial testing seems to indicate this works, but I'm concerned that there may be a hidden gotcha lurching somewhere (WMQI seems to have lots of these  )
Has anybody else use the environment tree to store large amounts of data or, as we are considering, full message copies? Do you know if there is any limit to how much data you can put in the Environment area?
Has anybody used any other approaches to successfully to reduce memory usage for a flow?
Cheers |
|
| Back to top |
|
 |
| JohnMetcalfe |
| Posted: Wed Nov 03, 2004 7:05 am Post subject: |
|
|
Apprentice
Joined: 02 Apr 2004
Posts: 40
Location: Edinburgh, Scotland
|
I thought I'd answer my own mail, and record for anyone else out there the results of our investigations....
1. Doesn't seem any problem using the Environment area in the way I originally described.
2. The reason that we are seeing the execution group restart at 390MB, rather than the 512MB, is , according to our contact at IBM:
| Quote: |
The reason for failure at 390MB rather than 512MB is that memory has likely
been allocated but not used. AIX aborts the process when the address space
requested reaches the 512MB limit but the tools only report pages that have
actually been touched. One reason for requesting memory and not using it is
the threads' heaps... each thread has a 1MB stack which only rarely will
get much use. |
3. Only use Compute Nodes when you wish to modify the Output message. We were using compute nodes where we were pulling off data from the inputRoot and sticking into the Environment area. If all you want is to read the message, then use a Database node - this does not make a copy of the message.
4. Building a XML tree in WMQI is VERY expensive in memory terms - the 15* message size was pretty much as expected according to IBM. Beware of this. We swapped to using the BLOB domain - which obviously enough uses much less memory !. We then moved all our required XML validation into a XML schema plug-in node - this meant we didn't need to parse (in WMQI) the incoming messages at all before we published them (we are using pub/sub).
5. We had a number of subscription flows that needed to spin through a repeating group structure within the XML message, and pump out a MQ message for every row found. IBM showed us a interesting approach to this that used very little memory. It uses the Environment area to hold a modifable copy of the message, plus the DELETE FIELD function to free memory as each row is processed. The net result is that the memory used was the input message bitstream, plus the fragment of the XML tree created in the Enviornment area, plus the output tree. This is documented in the WMQI 2.1 CSD4 readme. In case anyone is interested, I attach some sample code we developed to prove the concept (this code also segments a large message into smaller chunks)
| Code: |
-- Create a special folder in the output message to hold the input message tree
-- NB : InputMessage is the root element of an XML parser
CREATE LASTCHILD OF Environment.Variables DOMAIN 'XML' NAME 'InputMessage';
-- Copy the input message to this just created special folder
-- This is a root to root copy which will therefore not build trees....
SET Environment.Variables.InputMessage = InputRoot.XML;
DECLARE MyPointer REFERENCE TO Environment.Variables.InputMessage.CSL.Message.TxnGeneralList.Txn[1];
DECLARE i integer;
DECLARE j integer;
DECLARE endOfList boolean;
DECLARE counter integer;
SET counter = 1000;
SET i = 1;
SET endOfList = FALSE;
WHILE endOfList = FALSE
DO
-- initialise the output row counter
SET j = 1;
-- copy headers
SET OutputRoot.Properties = InputRoot.Properties;
SET OutputRoot.MQMD = InputRoot.MQMD;
-- setup to output 1000 rows per message
WHILE ((j <= counter) AND endOfList = FALSE) DO
-- now have access to the Txn list elements
-- create the fragment of the output tree
SET OutputRoot.XML.CSL.Message.List.Txn[j].InstCd = MyPointer.InstCd;
SET OutputRoot.XML.CSL.Message.List.Txn[j].SeqNum = MyPointer.SeqNum;
DECLARE lastProcessedRow REFERENCE TO MyPointer;
SET i = i + 1;
SET j = j + 1;
-- move to next row
MOVE MyPointer TO Environment.Variables.InputMessage.CSL.Message.TxnGeneralList.Txn[2];
--did the move of the pointer fail ?
IF LASTMOVE(MyPointer) = FALSE THEN
-- yes, so at end of list structure
SET endOfList = TRUE;
END IF;
--delete the input message row just processed, and free memory used...
DELETE FIELD lastProcessedRow;
END WHILE;
-- send out a message with j number of rows on it.
PROPAGATE;
-- outputroot trees are cleared by the propagate, so don't need to be deleted
END WHILE;
-- finally delete the temporary foleder and free up any memory it is using...
DELETE FIELD Environment.Variables.InputMessage;
RETURN FALSE; |
 |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
