| Author |
Message
|
| MQHpanduru |
 Posted: Sun Oct 21, 2012 3:51 pm Post subject: Memory leak issue with WMB V7004 |
 |
|
Newbie
Joined: 29 Apr 2011
Posts: 6
|
Hi Everybody,
We have recently upgraded WMB to V7004 and identified memory leak issue with post upgrade. When WMB was running on lower version V7 memory usage was quite stable and the graph was horizontal. After the upgrade, the memory usage is increased like anything. As per info center, WMB V7004 actually solves the memory issue but we are facing issue with V7004. We have raised a PMR with IBM but they have not provided any solution yet. Does anybody had similar issue and have solution for it? We hardly have 2 main flows and 3 subflows running on one execution group in production.All flows are developed using Java compute Node. Any response would be highly appreciated. Please let me know if anybody knows the root cause of the issue.Thanks in advance. |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Sun Oct 21, 2012 6:21 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Moved to Message Broker forum.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| zpat |
| Posted: Sun Oct 21, 2012 9:49 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Make sure the PMR contains a test case, that can be used to reproduce the problem. Give the PMR an appropriate severity. I have always found that IBM will come up with an explanation pretty quickly.
Please post the outcome here, as memory leaks are something I would be quite concerned about.
Also you might try using WMB 7005 since it's out, and see if that changes the behaviour.
Which platform are you using? |
|
| Back to top |
|
|
| MQHpanduru |
| Posted: Mon Oct 22, 2012 7:17 am Post subject: |
|
|
Newbie
Joined: 29 Apr 2011
Posts: 6
|
PMR team has suggested us to deploy each flow in separate execution group and test them independently. Currently, our servers are configured with only one execution group, so, I am thinking of stopping all the flows and test each flow independently. Hope that might give an idea which flow is causing the memory leak.
Secondly, we are using Linux platform. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Mon Oct 22, 2012 9:44 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| MQHpanduru wrote: |
PMR team has suggested us to deploy each flow in separate execution group and test them independently. Currently, our servers are configured with only one execution group, so, I am thinking of stopping all the flows and test each flow independently. Hope that might give an idea which flow is causing the memory leak.
Secondly, we are using Linux platform. |
Your workaround of not following the direction by the PMR team will only work if you stop and start the EG between each flow test.
Memory leak may not necessarily be located inside the flow code, it could be located outside the flow code but inside the container. Some WMB objects are instantiated once at flow start time or first message, then reused after that.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| MQHpanduru |
| Posted: Tue Oct 23, 2012 7:10 am Post subject: |
|
|
Newbie
Joined: 29 Apr 2011
Posts: 6
|
When I issue the 'mqsireportproperties' in PERF as
mqsireportproperties BROKER1 -e eg1 -o ComIbmJVMManager -a
I got the below output, where jvmMinHeapSize set to '32MB' and jvmMaxHeapSize set to '-1'. what does this '-1' means? Does it mean the value of jvmMaxHeapSize is default Max value which is 256MB?
ComIbmJVMManager
uuid='ComIbmJVMManager'
userTraceLevel='none'
traceLevel='none'
userTraceFilter='none'
traceFilter='none'
resourceStatsReportingOn='inactive'
resourceStatsMeasurements='<ResourceStatsSwitches ResourceType="JVM" version='1'> <Measurement name="InitialMemoryInMB" collect="on" /> <Measurement name="UsedMemoryInMB" collect="on" /> <Measurement name="CommittedMemoryInMB" collect="on" /> <Measurement name="MaxMemoryInMB" collect="on" /> <Measurement name="CumulativeGCTimeInSeconds" collect="on" /> <Measurement name="CumulativeNumberOfGCCollections" collect="on" /> </ResourceStatsSwitches>'
jvmVerboseOption='none'
jvmDisableClassGC='false'
jvmShareClasses='false'
jvmNativeStackSize='-1'
jvmMinHeapSize='33554432'
jvmMaxHeapSize='-1'
jvmDebugPort='0'
jvmSystemProperty=''
keystoreType=''
keystoreFile=''
keystorePass='********'
truststoreType=''
truststoreFile=''
truststorePass='********' |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Tue Oct 23, 2012 8:10 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
|
| Back to top |
|
|
| MQHpanduru |
| Posted: Tue Oct 23, 2012 9:36 am Post subject: |
|
|
Newbie
Joined: 29 Apr 2011
Posts: 6
|
Could please explain why do we need to increase the jvmMaxHeapSize to 8GB here ? Issue I am facing is memory usage by DataFlowEngine
Ex:
PID USERNAME LWP PRI NICE SIZE RSS STATE TIME CPU COMMAND
20013 mqm 49 59 0 1021M 970M sleep 412:22 0.84% DataFlowEngine
Value for SIZE and RSS are increasing daily |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Oct 23, 2012 9:40 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Did you enable FP4 capability after you applied the FP? or otherwise have mqsichangebroker -f all ?
Are you 100% positive that the only change that was made to the system was the application of the FP and the restart of the broker, and that *no* deployments of any kind were made? |
|
| Back to top |
|
|
| MQHpanduru |
| Posted: Tue Oct 23, 2012 10:02 am Post subject: |
|
|
Newbie
Joined: 29 Apr 2011
Posts: 6
|
Yes, Except the WMB FP v70004 upgrade, no other WMB deployments have been done till now.
I don't know how to put the image here. if you can provide your email id, I can send you the graph which shows memory usage post upgrade
[/img] |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Oct 23, 2012 10:14 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I leave my email address as an exercise for the reader.
You can only display images here, you can't host them.
If you're positive that absolutely no deployments have been done, then the simplest thing to do is specifically lower the FP level of the broker - mqsichangebroker -f again. Set it to the last known value (7.0.0.3) and confirm that the memory issue does or does not work.
Then take a service trace - for just long enough to show the memory usage increasing once.
Then reset the -f to all and take another service trace.
There are basically two options here: one that something is wrong with the product (and a review of the fixes included in FP5 might help identify what) two that your code has a bug that didn't used to be so strictly visible. |
|
| Back to top |
|
|
| MQHpanduru |
| Posted: Tue Oct 23, 2012 10:57 am Post subject: |
|
|
Newbie
Joined: 29 Apr 2011
Posts: 6
|
I wasmonitoring the memory usage for last 3 hrs and here is the output.
-bash-3.00$ date
Tue Oct 23 16:04:09 GMT 2012
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
27350 mqm 4016K 3728K cpu21 59 0 0:00:04 0.3% prstat/1
10819 mqm 967M 710M sleep 59 0 4:30:15 0.1% DataFlowEngine/64
-bash-3.00$ date
Tue Oct 23 17:27:23 GMT 2012
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
1579 mqm 4016K 3728K cpu12 59 0 0:00:03 0.3% prstat/1
10819 mqm 970M 713M sleep 59 0 4:33:14 0.1% DataFlowEngine/64
-bash-3.00$ date
Tue Oct 23 18:43:29 GMT 2012
-bash-3.00$ prstat -a
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
1609 mqm 4016K 3728K cpu4 59 0 0:00:01 0.1% prstat/1
10819 mqm 974M 717M sleep 59 0 4:35:52 0.1% DataFlowEngine/64
No tests have been done during this period hence EG is in sleep state. But, still I could see increase in RSS value for every hour |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Tue Oct 23, 2012 10:58 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
|
| Back to top |
|
|
| Harsha_AS |
| Posted: Tue Oct 30, 2012 10:32 pm Post subject: Did u find a solution for this. |
|
|
Apprentice
Joined: 04 Dec 2011
Posts: 26
|
| Recently we even we have upgraded to higher version and even we are facing the same issue. Please let me know if any one is having a solution for this. |
|
| Back to top |
|
|
|
|
