| Author |
Message
|
| EAI Developer |
 Posted: Fri Aug 15, 2008 11:38 am Post subject: Memory leakage issue on Broker v6.0.0.6 |
 |
|
Centurion
Joined: 30 Nov 2005
Posts: 101
Location: US
|
Hi All,
We have Message Broker application running on three execution groups.The execution group where main message flow got deployed have a memory leak.The memory is getting increased without paltueing.We already raised PMR with IBM for this,but still problem persists.
Frist ,we have applied latest fixpack alont with one APAR,which apprently should resolve leakage caused because socket relase problem with http request node.
Now,we are in the process of moving http1.0 to http1.1 and we also verified whether we are using clearmessage() function in java compute noedes.
So here my question is,what should be correct troubleshooting way to figure where is the memory leackge exactly and also is using trace node will cause memory leakage,if no.of trace nodes are high ??
Broker version : 6.0.0.6 and OS : AIX.
Pls let me know,If you need some more info. 
_________________
___________________
Regards,
EAI Developer. |
|
| Back to top |
|
 |
| sridhsri |
| Posted: Fri Aug 15, 2008 8:06 pm Post subject: |
|
|
Master
Joined: 19 Jun 2008
Posts: 297
|
1) Trace node is not the likely cause of memory leak. It is definitely not a cause for concern if you use a number of trace nodes.
2) Identifying a memory leak can be difficult. If you have more than one flow running on the EG, then remove all flows and keep just one at a time and try and send as many message as you can. Observe the DFE mem consumption. With that you will be able to identify which flow is the likely cause of the memory leak.
3) If there are more than one flow with a potential problem, then look for common nodes/code between them. You will need to build a simple test case (which can be given to L2/L3). Make the test case as simple as possible (fewest possible nodes and code that will result in the leak).
While you wait for a fix, perhaps the only interim solution I can think of is to restart your execution groups at frequent intervals to prevent out-of-memory exceptions. |
|
| Back to top |
|
|
| vaibhav_vy |
| Posted: Sun Aug 17, 2008 9:57 pm Post subject: |
|
|
Apprentice
Joined: 04 Aug 2008
Posts: 28
|
There are still some issues with JVM of broker. We are using Broker Runtime 6.1 & we use JavaCompute nodes. We also found the same memory leak issue & we are in process of removing JavaCompute nodes where possible. We are also in contact with IBM for this issue.
We found the memory did not get released when we use JavaCompute nodes for message processing.
Till the issue gets resolved, you can try to remove JavaCompute nodes & check whether memory leak problem persists.
As far as Trace node is concerned, there is no such problem. |
|
| Back to top |
|
|
| Gaya3 |
| Posted: Sun Aug 17, 2008 10:00 pm Post subject: |
|
|
Jedi
Joined: 12 Sep 2006
Posts: 2493
Location: Boston, US
|
| vaibhav_vy wrote: |
There are still some issues with JVM of broker. We are using Broker Runtime 6.1 & we use JavaCompute nodes. We also found the same memory leak issue & we are in process of removing JavaCompute nodes where possible. We are also in contact with IBM for this issue.
We found the memory did not get released when we use JavaCompute nodes for message processing.
Till the issue gets resolved, you can try to remove JavaCompute nodes & check whether memory leak problem persists.
As far as Trace node is concerned, there is no such problem. |
it was a known issue, and been rectified
6.1 broker is doing well, you can think of upgrading to it..
_________________
Regards
Gayathri
-----------------------------------------------
Do Something Before you Die |
|
| Back to top |
|
|
| marko.pitkanen |
| Posted: Sun Aug 17, 2008 10:17 pm Post subject: |
|
|
Chevalier
Joined: 23 Jul 2008
Posts: 440
Location: Jamsa, Finland
|
Hi,
I'm remembering from the past that the broker uses such a strategy with memory reserving / releasing that it keeps the maximum memory used, because it assumes that is likely to have same kind of messages at the future to be processed again and it tries to optimize the throughput of the message flow by decreasing memory allocations. So it is not releasing all the memory before mqsireload / eg or flow restart. This behavior can perhaps do little more complexity to researchings ( if broker still uses that strategy).
Marko |
|
| Back to top |
|
|
| vaibhav_vy |
| Posted: Sun Aug 17, 2008 11:50 pm Post subject: |
|
|
Apprentice
Joined: 04 Aug 2008
Posts: 28
|
We are using Broker 6.1 & it seems memory leaks are not completely rectified esp. when JCNs are used.
Even in Broker 6.1 the same memory allocation strategy is used. We checked it using some test flows & monitoring broker memory usage. We observed the same facts mentioned by marko.pitkanen. |
|
| Back to top |
|
|
| EAI Developer |
| Posted: Thu Aug 21, 2008 1:41 pm Post subject: |
|
|
Centurion
Joined: 30 Nov 2005
Posts: 101
Location: US
|
Hi,
We are using single 'main flow' in that execution group which is having memory leak,So break the flow is not possible and we are using java compute nodes given by our COE team and those been used by different projects and proven there is no memory leak in those JCNs.
As a next step,we are trying move from http 1.0 to http1.1 ( by changing properties of httprequest node in all the request subflows).
Do you think ,will it(moving from 1.0 to 1.1) effect the memory leakage issue in positive way ??
Thanks,
EAI Developer.
_________________
___________________
Regards,
EAI Developer. |
|
| Back to top |
|
|
| EAI Developer |
| Posted: Mon Aug 25, 2008 1:39 pm Post subject: |
|
|
Centurion
Joined: 30 Nov 2005
Posts: 101
Location: US
|
additional question:
Is there any way to monitor memory usage at node level/msg flow level in message broker apart from using tivoli omegamon.Here I am interested not in CPU usage,but in memory comsuption of individual modes in a msg flow when request and reply is going through.
Regards,
EAI Developer. |
|
| Back to top |
|
|
| sunny_30 |
| Posted: Mon Aug 25, 2008 7:30 pm Post subject: |
|
|
Master
Joined: 03 Oct 2005
Posts: 258
|
Hi,
To monitor the memory-leak in the dfe execution-grp process, you will need to look at IF the 'virtual-memory' (VM) of this process is growing.
On AIX run the "ps vg" command to get the virtual-memory for the dfe process, $6 should give you that.
you calculate if this value is growing overtime, usually represented as 'delta'
use the script in this link:
http://www.tek-tips.com/viewthread.cfm?qid=1174789
svmon can also help you to grab the VM details of the dfe process.
On Windows:
you can monitor the dfe-process's memory-delta value for virtual-memory in win-task-manager.
If this grows you have the leak
The first symptom for the memory-leak is that you 'may' have dfe-abends. contact IBM.
If the dfe process continues to run w/o abend, the virtual-memory box on the box will be fully occupied, the system halts resulting in a restart!
weekly restarts of the broker shd 'prevent' you from memory-leak.
but the cure cd be in a fix-pack or the code involved.
//sunny |
|
| Back to top |
|
|
| EAI Developer |
| Posted: Tue Aug 26, 2008 9:59 am Post subject: |
|
|
Centurion
Joined: 30 Nov 2005
Posts: 101
Location: US
|
Hi Sunny,
Thanks for the reply!!. We already came to conclusion that there is a memory leak on one of the three exe grps we have.
About def abends,we have a fixpack 6.0.0.3,which is addressing this issue and our broker is already at 6.0.0.6 .
Is there any way to know the memory consumption at message node level ??
_________________
___________________
Regards,
EAI Developer. |
|
| Back to top |
|
|
| EAI Developer |
| Posted: Thu Aug 28, 2008 1:50 pm Post subject: |
|
|
Centurion
Joined: 30 Nov 2005
Posts: 101
Location: US
|
Any responses ?? I am lost here in this issue. I just want to know,is there any way to find out the memory usage at node level,apart from Tivoli omegamon.
_________________
___________________
Regards,
EAI Developer. |
|
| Back to top |
|
|
| krypton |
| Posted: Fri Jun 04, 2010 7:47 pm Post subject: |
|
|
Disciple
Joined: 14 Mar 2010
Posts: 186
|
Me too facing memory leak issues in our broker environment, it is tough to point out to a particular flow when there are so many flows and execution groups running. But, could anyone suggest where exactly we all should look for when a memory leak occurs.
I run a command "ps aux |grep DataFlowEngine | grep 112238", and got the below result, this dataflow is running around 86 flows(they are non-complex in nature), wondering whether the below stats specify any danger.
mbmqm 1122380 0.0 6.0 892188 877184 - A 01 Jun 9:02 DataFlowEngine
Regards
_________________
Dreams are not something which you watch when you are asleep,it is something which doesn't let you sleep. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Jun 05, 2010 2:54 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
You are running WMB 6.0.0.6 ...
What is the level of your xlc route? (from memory)
| Code: |
| lslpp -l | grep -i xlc |
Make sure you are @ 9.0.0.3 or above. If you are not at the required level you need to upgrade your xlc route (part of the os). If your xlc route level is below 9.0.0.3 there are random memory problems that will plague you. 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
