| Author |
Message
|
| richhall88 |
 Posted: Thu Jun 05, 2014 7:32 am Post subject: Possible memory leak from WMB API? |
 |
|
Newbie
Joined: 05 Jun 2014
Posts: 4
|
Hi folks,
I have experienced this resource problem such as: outOfMemoryError, or Resource temporarily unavailable in my message flow. I have tried to play around with the heapsize, but no luck. For a regular functional test, it works just fine. It only occurs when we run a large load of transactions. It always fails around 700-900 transactions. It is even worse that with this type of error, there is no way for me to detect in my workflow or javacompute node so that I can roll back the message back into the queue, or a way to notify the support. the message just being consumed half way through the message flow and disappeared for good. From the message flow itself, everything looks like it is working normally. We won't know it until we manually look at the /var/mqm/errors/ for the FDC file. This is danger. the message just got lost even it was set to be persistent. You can see the type of error message I posted below.
After many days of troubleshooting, and researching the internet. There was no suggestion that came close to give any hint.
Finally, with eliminating process, I was able to zero in the piece of code that causes the memory leak. By removing the line below, my message flow can process hundred of thousands of messages without any issue. If you ever have this line in your code, beware:
ExecutionGroupProxy eg = ExecutionGroupProxy.getLocalInstance();
Don't ask me why such line of code could have caused the memory leak, but that is what I have discovered. IBM supports folks can give a better explanation. I just want to share my painful experiences so you can stay away from it.
WebSphere Broker v7006[17949]: (SISLXBX6_BRK.INQUIRY_DOMAIN_STQ)[32]BIP2112E: Message broker internal error: diagnostic information '11', 'Resource temporarily unavailable'. : SISLXBX6_BRK.9f7d3e3e-4601-0000-0080-dd3a79bc9661: /build/S700_P/src/CommonServices/Linux/ImbOsThread.cpp: 115: ImbOsThread::ImbOsThread: :
or
errno 11
RecoverableException: null
File: /build/S700_P/src/DataFlowEngine/PluginInterface/com_ibm_broker_plugin_CMbService.cpp
Line: 1863
Function: ImbJavaExceptionUtils::throwableToNativeException
Type:
Name:
Label:
Catalog: BIPmsgs
Severity: 3
Number: 4395
Text: Unhandled exception in plugin method
Insert: null
Type: 5
Text: java.lang.OutOfMemoryError
Insert: null
Type: 5
Text: java.lang.Thread
Insert: null
Type: 5
Text: startImpl
Insert: null
Type: 5
Text: Thread.java
Insert: null
Type: 2
Text: -2 |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu Jun 05, 2014 8:20 am Post subject: Re: Possible memory leak from WMB API? |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| richhall88 wrote: |
| IBM supports folks can give a better explanation. |
What better explaination did they give in response to your PMR?
| richhall88 wrote: |
| I just want to share my painful experiences so you can stay away from it. |
Well that's very kind of you, but what about those of us who have no option but to use this in a high volume flow? Better to convert your pain into progress and make the next fix pack a better place.
Remember that while IBM support folks (and other IBMers) may haunt this forum they do not do so officially, and even if moved by your terrible experience to help they lack your ready ability to duplicate the problem.
And it is a problem - persistent messages should never just go quietly into that night.
I'm just glad no-one else has fallen foul of this yet. Given the amount of high volume we have running through Java in WMB installations.
Raise a PMR. Don't suffer alone. We spent 5 months trying to pin down a memory leak and IBM suffered along with us.
Something of a relief when it turned out to the the SOAPInput node leaking rather than our user code I will admit, but the point is our pain has helped make a better SOAPInput node, that will no longer leak under a convoluted but not reasonable set of circumstances.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Esa |
| Posted: Thu Jun 05, 2014 10:06 am Post subject: Re: Possible memory leak from WMB API? |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| richhall88 wrote: |
ExecutionGroupProxy eg = ExecutionGroupProxy.getLocalInstance();
|
I'm not surprised. Seems your flow was trying to administer itself. |
|
| Back to top |
|
|
| richhall88 |
| Posted: Thu Jun 05, 2014 10:18 am Post subject: Re: Possible memory leak from WMB API? |
|
|
Newbie
Joined: 05 Jun 2014
Posts: 4
|
| Esa wrote: |
| richhall88 wrote: |
ExecutionGroupProxy eg = ExecutionGroupProxy.getLocalInstance();
|
I'm not surprised. Seems your flow was trying to administer itself. |
Hi Esa, not sure what you mean. All I wanted in my code was to query the execution group name to distinguish the client that I am serving.
BTW, I have opened a PMR to IBM support. I will update if anything they find.
Thanks. |
|
| Back to top |
|
|
| Esa |
| Posted: Thu Jun 05, 2014 10:24 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| OK, did you also remove the lines where you connected to the BrokerProxy? |
|
| Back to top |
|
|
| richhall88 |
| Posted: Thu Jun 05, 2014 10:52 am Post subject: |
|
|
Newbie
Joined: 05 Jun 2014
Posts: 4
|
| Esa wrote: |
| OK, did you also remove the lines where you connected to the BrokerProxy? |
Nope. Don't have reference to this class in my code. |
|
| Back to top |
|
|
| stoney |
| Posted: Thu Jun 05, 2014 11:04 am Post subject: Re: Possible memory leak from WMB API? |
|
|
Centurion
Joined: 03 Apr 2013
Posts: 140
|
| richhall88 wrote: |
| Esa wrote: |
| richhall88 wrote: |
ExecutionGroupProxy eg = ExecutionGroupProxy.getLocalInstance();
|
I'm not surprised. Seems your flow was trying to administer itself. |
Hi Esa, not sure what you mean. All I wanted in my code was to query the execution group name to distinguish the client that I am serving.
BTW, I have opened a PMR to IBM support. I will update if anything they find.
Thanks. |
Instead of using the administration (CMP) API for this purpose, you could use the lightweight getExecutionGroup().getName() method from within your Java Compute node class? |
|
| Back to top |
|
|
| richhall88 |
| Posted: Thu Jun 05, 2014 11:25 am Post subject: Re: Possible memory leak from WMB API? |
|
|
Newbie
Joined: 05 Jun 2014
Posts: 4
|
[quote="stoney"][quote="richhall88"]
| Esa wrote: |
| richhall88 wrote: |
ExecutionGroupProxy eg = ExecutionGroupProxy.getLocalInstance();
|
Instead of using the administration (CMP) API for this purpose, you could use the lightweight getExecutionGroup().getName() method from within your Java Compute node class? |
Thanks Stoney. Yeap, that should work. I will try that separately in my playground. Because we are already running late in our project, and my current work around has already addressed the problem and accepted by performance test script, I will use that API in a subsequent version of this project.
Thanks for the tip. |
|
| Back to top |
|
|
| Esa |
| Posted: Thu Jun 05, 2014 10:57 pm Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| richhall88 wrote: |
| Esa wrote: |
| OK, did you also remove the lines where you connected to the BrokerProxy? |
Nope. Don't have reference to this class in my code. |
Ok. ExecutionGroupProxy.getLocalInstance() is to be called only from a java compute node or a user defined node. So it has a very limited use, which is presumably the reason why this bug hasn't been discovered earlier.
When calling ExecutionGroupProxy.getLocalInstance() you don't need to connect to the broker, because you are already running on it.
Programs used for administering broker objects need first to connect to the broker by instantiating a BrokerProxy object, a container for a connection to the broker. I think the BrokerProxy object does all required cleanup when it's disconnected. Or I hope that it does... admin programs are called not very often, so this kind of memory leaks could easily happen unnoticed. You could in fact test that by repeatedly restarting CMP API exerciser and examining an eg, monitoring if the eg allocates more and more memory and probably finally crashes...
Obviously ExecutionGroupProxy.getLocalInstance() causes some memory allocation under the JNI interface. That memory is not cleaned up when the ExecutionGroupProxy object goes out of scope. For the same reason you need to explicitly call clearMessage() for all MbMessage objects that you create in a JCN.
So, after you are ready with everything you need to do with the local instance of an EGP, you should explicitly call clearExecutionGroup(). If only there was such method... |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Jun 06, 2014 5:08 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Clearly the problem is with your code.
Did no see you accessing it via the BrokerProxy where the disconnect method will clean up the resources.
As such you should probably treat it as a singleton in the JVM with a synchronized retrieval method...
So your code should create a static class for accessing the egproxy.
This way you declare it only once in the eg.
The best way to access proxys should really be over the broker proxy to allow you to disconnect and achieve resource cleanup...
| Code: |
private static egproxy = null;
synchronized static ExecutionGroupProxy getegproxy() {
if (this.egproxy == null) {this.egproxy = ExecutionGroupProxy.getLocalInstance();}
return this.egproxy;
} |
What you seem to be doing is instantiate a new eg proxy for every message instance... and never releasing it... no wonder you get an out of memory!.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| stoney |
| Posted: Fri Jun 06, 2014 5:53 am Post subject: |
|
|
Centurion
Joined: 03 Apr 2013
Posts: 140
|
There's no JNI involved in the CMP API - it's all 100% Java. Behind the scenes though ExecutionGroupProxy.getLocalInstance() makes a call to BrokerProxy.getLocalInstance() which spawns a new connection to the broker, along with associated threads.
This can be cleaned up by getting the parent of the ExecutionGroupProxy, which should be the BrokerProxy, and calling disconnect on it:
| Code: |
ExecutionGroupProxy egp = ExecutionGroupProxy.getLocalInstance();
BrokerProxy bp = (BrokerProxy) egp.getParent();
bp.disconnect(); |
Also, I agree - it's a bit much to be creating one of these for every message through - a Java singleton is the right way to go if you need to use an ExecutionGroupProxy for every message. |
|
| Back to top |
|
|
|
|
