| Author |
Message
|
| ashwgupt |
 Posted: Thu Jan 12, 2012 11:07 pm Post subject: EG JVM Memory build up with every message processed |
 |
|
Novice
Joined: 31 Oct 2011
Posts: 12
|
Hi,
In one of our project, there is a strange behaviour observed when the flow processes any message and the memory associated with JVM increases by a few KBs and stays there without getting released.
The flows involved are NOT using any shared variables and that is why is it more of a surprising observation for us.
Below is the chart of the Memory usage by the particular EG
PID Mem Stage
2355282 174700 - new complete deployment
2355282 175784 - 1st msg
2355282 175808 - 2nd msg
2355282 176392 - 3rd msg
2355282 176416 - 4th msg
The other interesting thing is that,
1) at first message's execution the mem increases substantially, may be because a few Static Vars are being used
2) from 2nd message onward, the hike is not very substantial and also the hike happens only when the messages is processed by one flow which is using Publish Node.
With every message processed an execution group (EG) grabs some more memory and it does not release it back to the OS. Assuming that the EG needed that much mem in the past and might need it in the future so it holds onto it.
Now the quesiton is, such behaiour expected and normal?
Kindly comment and let me know if more info required. |
|
| Back to top |
|
 |
| Esa |
| Posted: Thu Jan 12, 2012 11:36 pm Post subject: Re: EG JVM Memory build up with every message processed |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| ashwgupt wrote: |
With every message processed an execution group (EG) grabs some more memory and it does not release it back to the OS. Assuming that the EG needed that much mem in the past and might need it in the future so it holds onto it.
Now the quesiton is, such behaiour expected and normal?
|
It's normal and well-known behaviour of execution groups that they do not release the memory they have allocated. That has to do with optimization - the eg expects to need the memory again soon so it keeps it. Besides, allocating and releasing memory has it's cost, too, not to talk about the bookkeeping required if you tried to control memory in a more advanced manner. Something like that.
But if the allocated memory increases a little bit for each message, it might be because of a memory leak. Firs of all I think your sample may be too small to tell that. I would test again with 100 - 200 messages.
I presume you are using java. Check that you call .clearMessage() for every MbMessage you instantiate in the code. The normal way is to make the call in a finally block after a try-catch to force it to be called even when you get an exception. This is because it's the underlying C code that actually allocates the memory and it cannot know when your java code is finished with it. So you code has to signal it.
Last edited by Esa on Thu Jan 12, 2012 11:59 pm; edited 1 time in total |
|
| Back to top |
|
|
| jlaisbett |
| Posted: Thu Jan 12, 2012 11:54 pm Post subject: |
|
|
Apprentice
Joined: 27 Nov 2009
Posts: 39
|
I'll assume you are using Java nodes in which case 2 common causes of memory leaks are:
Not calling clearMessage()
Using try catch blocks that handle exceptions of one of the Mb*Exception types without rethrowing it, this one only leaks memory when you get an error.
I would recommend trying to crash an execution group and seeing if you can cause a java out of memory error. This should tell you if it's java memory or native memory that's leaking, broker uses a fair bit of JNI so it's possible java can actually cause a native memory leak rather than a jvm memory leak. |
|
| Back to top |
|
|
| ashwgupt |
| Posted: Fri Jan 13, 2012 12:46 am Post subject: |
|
|
Novice
Joined: 31 Oct 2011
Posts: 12
|
Thanks for quick replies.
On your suspicion of Java code being used, I would like to confirm that, there the JCN isn't being used in the code and hence there isn't any custom java code. All we have got in terms of Java is the support pac - ia9a and Log4j
As you suggested, we are now trying with a bigger set of messages to get a better sample. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Jan 13, 2012 1:05 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| jlaisbett wrote: |
I would recommend trying to crash an execution group and seeing if you can cause a java out of memory error. This should tell you if it's java memory or native memory that's leaking, broker uses a fair bit of JNI so it's possible java can actually cause a native memory leak rather than a jvm memory leak. |
If you see this happening then it is time to raise a PMR.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Jan 13, 2012 1:09 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
The virtual memory used by this DataFlowEngine went up to 8 GB at one point!
Which looks a bit high for JVM memory?
This is WMB 7003 on AIX. |
|
| Back to top |
|
|
| Esa |
| Posted: Fri Jan 13, 2012 2:48 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| ashwgupt wrote: |
On your suspicion of Java code being used, I would like to confirm that, there the JCN isn't being used in the code and hence there isn't any custom java code. All we have got in terms of Java is the support pac - ia9a and Log4j
|
There are several IBM-provided nodes that are implemented with java, too. For example header nodes. There have been memory leaks associated with these nodes, for example this one that was fixed in 6.1.08 and 7.0.0.1 |
|
| Back to top |
|
|
|
|
