| Author |
Message
|
| m8 |
 Posted: Thu Aug 25, 2011 5:47 am Post subject: Memory leak with shared variable and parsing mime message |
 |
|
Newbie
Joined: 11 Feb 2011
Posts: 5
|
Hi,
I am trying to fix a memory leak in a flow 
To find it, I splitted the flow into smaller flows and deploy them in different execution groups.
For now, it seems that there are 2 memory leaks but I dont understand why it happens.
The first leak involves the use of a shared variable.
| Code: |
DECLARE activeTimers SHARED ROW;
CREATE COMPUTE MODULE FileManager_Check
CREATE FUNCTION Main() RETURNS BOOLEAN
BEGIN
DECLARE localTimers ROW;
SET localTimers.Timers = activeTimers.Timers;
-- ...
DELETE FIELD activeTimers.Timers;
SET activeTimers.Timers = localTimers.Timers;
RETURN FALSE;
END;
END MODULE;
|
The original flow is without the line DELETE FIELD activeTimers.Timers that fixed partially the leak. But why I need to explicitly delete the old field before do the copy?
The second leak concerns the parsing of a multipart mime message. My test flow is MIMEInput > Compute Node and with the esql code below :
| Code: |
CREATE COMPUTE MODULE FileManager_Extract
CREATE FUNCTION Main() RETURNS BOOLEAN
BEGIN
SET OutputLocalEnvironment = InputLocalEnvironment;
CREATE LASTCHILD OF OutputRoot DOMAIN 'XMLNSC' PARSE(InputRoot.*[<].Parts.Part[1].Data.BLOB.BLOB CCSID 1208);
RETURN FALSE;
END;
END MODULE;
|
This flow increase its memory about 400MB for 4 hours and I don't know how to fix it or if there's something to fix.
Any help or advice would be most welcome  |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Thu Aug 25, 2011 6:12 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I would open a PMR on the shared variable issue. It is a bit odd.
I think the XMLNSC parsing "leak" is perfectly normal. I suspect your XML document is fairly large, and thus you end up needing a large amount of memory to hold it fully parsed.
And Broker implements it's own memory management, so even though the EG has increased it's size by 400Mb, doesn't mean that those 400Mb are holding the same data for that entire 4 hours. |
|
| Back to top |
|
|
| m8 |
| Posted: Thu Aug 25, 2011 8:06 am Post subject: |
|
|
Newbie
Joined: 11 Feb 2011
Posts: 5
|
Every 2 minutes I send 2000 times the same 60kb message to the input node. It is a multipart MIME message, each part is a blob :
the first contains some xml data (a few bytes)
the second about 60 kb of data.
In my test flow, only the 1st part is parsed but I guess the full message is loaded in memory, ie about 120 Mb every 2min.
The original flow with the leak crashes in the production env because of out of memory so this memory increase is really suspicious for me (the first one with the shared variable as well but was fixed).
The EG with the test flow (only 2 nodes and one line of code) is still increasing (1224Mb now). |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Aug 25, 2011 8:16 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I'm sorry, I missed something in your original post.
You are entering a compute node, creating a new message and parsing some XML data in OutputRoot , and then exiting the compute node without propagating... thus leaving the OutputRoot message dangling.
So I think it's the "RETURN FALSE" that's the source of your memory leak. |
|
| Back to top |
|
|
| m8 |
| Posted: Fri Aug 26, 2011 2:52 am Post subject: |
|
|
Newbie
Joined: 11 Feb 2011
Posts: 5
|
Yes, I removed the propagate statements before the RETURN FALSE from the original code.
However, the leak is still there with a RETURN TRUE. |
|
| Back to top |
|
|
| jonatan bataz astudillo |
| Posted: Wed Aug 29, 2012 10:04 am Post subject: |
|
|
Newbie
Joined: 08 Feb 2012
Posts: 3
|
| Excuse me did you solve this issue? i have the similar? |
|
| Back to top |
|
|
|
|
