MQSeries.net :: View topic

Armageddon123 · Posted: Mon Apr 28, 2014 7:19 am Post subject:

Hi Experts

I need your help on a new memory issue.
I have gone thru the memory issue carefully and have been able to recreate the memory issue with a very small flow.
using MQnput,MQGet and Compute nodes.

Here it is

MQInput- Compute1-MQGet-Compute2-Compute3

Code in Compute1

CREATE FUNCTION Main() RETURNS BOOLEAN
BEGIN
CALL CopyMessageHeaders();
-- CALL CopyEntireMessage();
DECLARE K INTEGER 1;
WHILE K < 200 DO
SET OutputRoot.XMLNSC.A.C = 'A';
PROPAGATE ;
SET K= K + 1;
END WHILE;
RETURN FALSE;
END;

Here ,I am just propagating a small , valid xml to next node
In MQGet , Message Domain is set to XMLNSC.
In Compute2 node, it has only one line of code., Saving the xml in Environment
SET Environment.Data= InputRoot.XMLNSC.Data ;
(though this is not the correct method of saving the data in Environment tree, but here we are not much bothered about the data , only the memory.)
In Compute3 node, only one line
SET Environment.Data = NULL ;

So logic is trigger the flow with MQInput with a small message(1 KB message). Propagate 200 times from the Compute1 node. Each time, get a message from the MQGet Queue.
Then save it in Environment and then later ,delete it from the environment. Each xml in MQGet Queue is 500 KB size..
Why the last two compute nodes?- In real flow, once the message from MQGet is saved in Environment, some additional functions are carried out and finally the message is removed from Environment tree.Here , for simplicity, i have removed those things. This sample flow alone is enough to recreate the memory issue

Now in this simple flow, I was expecting that, after each time the flow reaches Compute3 node, the Environment.Data is cleared and so there is no additional storage/memory required when the flow saves the message again in the Environment in the next loop

But as seen from the DataFlowEngine.exe process memory usage, the memory goes up and up and up each time it is looping and the EG reaches the max memory and sometimes abends.
It appears that the statement SET Environment.Data = NULL ; is not serving the purpose of clearing the memory as i thought.
I was of the understanding that the, SET NULL statement will issue a "free" on all its storage and these blocks will be returned to the DataFlowEngines heap for next message.
The xml which MQGet gets is 500 KB xml which is the reason for huge memory usage at first place. but doesn't SET NULL statement clear that all?

Can you please advise me what is the reason for this memory issue in this simple flow.
What code/logic needs to be implemented in the Compute3 node so that the EG memory doesnot go up and up as it goes looping. In other words, how can we code to free the storage in each loop.

Memory Statistics are below for the test run

before tirggering the flow : DataFlowEngine.exe memory - 97,176 K

after triggering the flow:DataFlowEngine.exe memory - 1041,120 K

using WMB 7.0.0.6