| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| JVM, Custom Java Nodes and Broker v5 on Linux |
« View previous topic :: View next topic » |
| Author |
Message
|
| damianharvey |
 Posted: Mon Oct 31, 2005 6:23 pm Post subject: JVM, Custom Java Nodes and Broker v5 on Linux |
 |
|
Acolyte
Joined: 05 Aug 2003
Posts: 59
Location: Sydney, Australia
|
Hi all,
We have an issue where the JVM for Broker v5 isn't garbage collecting and our servers eventually run out of memory. We have an open PMR with IBM, however I thought I'd ask youse fellas as well.
The behaviour we are seeing is that memory usage steadily increases until it consumes all available. This only happens when we have custom java input nodes in our flows (unfortunately necessary).
We are running WMB v5 CSD6 on RedHat ES 3.0 update 2 (with LD_ASSUME_KERNEL=2.4.19).
Any ideas?
Thanks,
Damian. |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Tue Nov 01, 2005 4:12 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Fix the leaks in your java nodes.
Don't forget to delete any MB trees or elements you create.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| JT |
| Posted: Tue Nov 01, 2005 6:48 am Post subject: |
|
|
Padawan
Joined: 27 Mar 2003
Posts: 1564
Location: Hartford, CT.
|
We recently experienced the same behaviour (escalating memory comsumption) in a v5.0.2 Solaris environment. Here's what IBM found:
| Quote: |
On examining the source to this we can see the following :
MbNode myNode = new MbNode();
This creates a new java Node object every time the UtilityGetNodes
evaluate method is called. Ie everytime this node is used and called in
a message flow. We are curious as to why the customer has done this in
their plugin. Afterall, the UtilityGetNode extends MbNode and so itself
s an MbNode itself and hence can call all the methods of the MbNode
class. Therefore it should be possible to call such functions as getExec
utionGroup() etc from the evaluate function itself either directly or as
a method of the variable this.
A plugin nodes constructor is usually only called when the message flow
is deployed and created, and the MbNode object is destroyed when the
message flow is deleted, or the broker execution group is stopped. Ie a
node lasts for the life of a message flow, not for just the instance
the message flow for the current message it is processing. Therefore we
are concerned that everytime an MbNode object is created it will persist
for the life of the flow and hence a storage growth will be seen. We
would suggest that the customer remove this from their plugin since they
can use the MbNode methods within the evaluate method since the node
extends MbNode. Once they have made this change they will need to
recompile this, rebuild their jar file, place this in the jplugin
directory and restart the broker.
It should also be noted that allocations on the Java heap that are
correctly dereferenced will not be freed until Garbage collection has
been driven. Therefore it is normal for a flow that utilizes Java to
cause a gradual growth in the DFE storage usage. JVM Garbage collection
is not drive periodically and is actually driven when the heap
utilization reaches a threshold. This threshold is calculated on the min
heap size, max heap size and the current heap size. What has the
customer set the max jvm heap size to for this broker?? For example if
the Max JVM heap size is set to 2GB, then the DFE may grow to between
1.5GB and 2GB before Garbage collection is driven.
It is possible to monitor the Brokers JVM Garbage collection activity by
issuing the following in the Brokers start-up environment :
export MQSI_RUN_ATTACHED=1
export MQSIJVERBOSE=-verbose:class,gc,jni
mqsistart brokerName
If this is piped to a file then this file will capture when the JVM heap
is expanded and when Garbage collection cycles are started. This
information also shows how much of the heap is cleared when garbage
collection is complete. |
We implemented their recommendation, and all is well. |
|
| Back to top |
|
|
| damianharvey |
| Posted: Tue Nov 01, 2005 11:56 am Post subject: |
|
|
Acolyte
Joined: 05 Aug 2003
Posts: 59
Location: Sydney, Australia
|
| jefflowrey wrote: |
Fix the leaks in your java nodes.
Don't forget to delete any MB trees or elements you create. |
All being done. I should have mentioned that there is no problem with this on the Windows (Development) boxes. |
|
| Back to top |
|
|
| coxsteph |
| Posted: Tue Nov 01, 2005 1:06 pm Post subject: More detail |
|
|
Newbie
Joined: 01 Nov 2005
Posts: 3
|
After collecting trace of both the broker and garbage collection for the PMR it looks as though it's a known problem. The code calling the garbage collector runs in a seperate thread and when that thread calls the finalizer functions on the threads the execution group goes into a high cpu usage state and never returns.
The fix on AIX was to ensure clearMessage() was called. However we're calling that method, ensuring we dont have any memory leaks (thanks Jeff for that gem). However the problem is still seen. A fix was included in Broker 6 for the problem seen on AIX.
The node runs for 36 hours at a steady 196Mb on windows. The max heap size is -1, the min is 134... (defaults for the broker)
Redhat ES3 Update 6, Broker FP6, MQ FP11, DB2 8.1 FP9. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
