| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| [solved]Need help on WMB flow memory !! |
« View previous topic :: View next topic » |
| Author |
Message
|
| Armageddon123 |
 Posted: Tue Feb 11, 2014 6:47 am Post subject: [solved]Need help on WMB flow memory !! |
 |
|
Acolyte
Joined: 11 Feb 2014
Posts: 61
|
Hi Gurus!!
Need help from u guys. I am a newbie to WMB. I am given a task which i am not able to proceed.
Issue: work on one WMB flow where the memory goes up during execution.
The flow is like this.
MQInput-Subflow1-Subflow2-Subflow3-Subflow3-MQputput
Subflow3 is the one which causes trouble.It is like this
Input-Compute1-MQGet-Compute2-Compute3-JavaCompute1-Compute4-MQoutput3
here Compute1 has propagate statement inside a loop. So it will fetch message from MQGet node as long as it propagates from Compute1.
(Another app puts message in the Q defined on QMget)
So flow will propagate from Compute1 with a Correlid, then get that message from MQGet node- proceeds to Compute2, 3, jaavcompute, COmpute4, and puts the transformed message to MQoutput3
Now comes back to the Compute1 to continue as above logic(this will loop for arund 20-30 times.)
In MQGet- transaction is set to No.(so there is no need to rollback the whole messages incase of any failure,)
In compute node, there is no saving of the message.
PROPAGATE TO TERMINAL 'out1';--- not using delete none option,
I read through docs and understood that javacompute doesnot perform the mesage clearing automatically, So verified that .clearMessage statement is present at the end of javacompute node as well..
So from what I understand , there is no place in code where there is additional overload on WMB.
What now happens in the system is when the flow loops first time, memory is going up. Now when the first message completes and the control comes back to the first Compute(Compute1) node in Subflow3,
the memory taken for the first message is not released!!..
For every message in the loop, the memory keeps addingup.
Now the most strange thing!!-- Even after the whole flow compeltes, the memory used by the flow is not released!, Can you guys please help me on how i can proceed on this.
1. How can i find why the memory consumed for first loop is not released when the flow starts with second message.
2. How can i find why the flow is not releasing the memory even after the full flow is completed.
Read about some monitroing tools which can help to understand memory leakage. Neither the client machine allows such installations nor is this a case of memory leakage!!. It is just that it is not releasing ANY memory which it consumed for the flow.
Any help on how to proceed will be much appreciated.!!
Last edited by Armageddon123 on Thu Feb 20, 2014 8:32 pm; edited 1 time in total |
|
| Back to top |
|
 |
| Gralgrathor |
| Posted: Tue Feb 11, 2014 6:52 am Post subject: |
|
|
Master
Joined: 23 Jul 2009
Posts: 297
|
1. How do you measure memory usage?
2. The EG will usually allocate as much memory as required to complete the transaction, and *not* release it until the EG is restarted. Allocated memory may be reused in subsequent transactions. Read this:
| Quote: |
| When the message flow has completed its processing, then it issues a "free" on all its storage and these blocks will be returned to the DataFlowEngines heap ready for allocation to any other DataFlowEngine threads. The storage is never released back to the operating system, because there is actually no programmatic mechanism to perform such an operation. The operating system will not retrieve storage from a process until the process is terminated. Therefore the user will never see the size of the DataFlowEngine process decrease, after it has increased. |
Have you checked if memory usage continues to go up when the flow is triggered many times?
3. Without code snippets, it's difficult to tell how you're using your memory. Are you using any SHARED variables, for instance? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Feb 11, 2014 8:24 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Your design is at fault.
You can't build, (as I suspect you did), a loop from one node downstream to a node upstream...
Your loop needs to have a PROPAGATE statement in it to send it's messages downstream... Read up on the matter. 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Armageddon123 |
| Posted: Tue Feb 11, 2014 9:11 pm Post subject: |
|
|
Acolyte
Joined: 11 Feb 2014
Posts: 61
|
Hi Gralgrathor and fjb_saper,
many thanks for your comments.Please find my response below.
1. Memory measurement- In the Production, it is on Linux, so using the ps command to display the CPU/memory for the EG processID.
In local machine, windows , checking the DataFlowEngine process CPU in taskmanager.
2. Thankyou for the link to IBM technote.!!, that was a new information for me!.
But i am a bit surprised too!. We have some messageflows in our project which gets triggered around 20,000 times per day and the EG hasnot been bounced in last 5 months!!!!, ps command output shows very normal usage for the EG still!!.
A possible reason might be this from the link u gave
"When the message flow has completed its processing, then it issues a "free" on all its storage and these blocks will be returned to the DataFlowEngines heap ready for allocation to any other DataFlowEngine threads."
I will test some more flows in the system to understand more about the memory consumption pattern.
Also,yes, the memory goes up when the same flow is triggered many times.
3.The code is actually very big!!,I tried to remove the client specific details from the code but it is never ending!!,, basically, it has a lot of IF conditons, SET statements etc, ASBITSTREAM function etc..
It doesnot use any SHARED variable in the code
Hi fjb_saper,
can you kindly elaborate on the point.
"You can't build a loop from one node downstream to a node upstream".
A relevant part of the code in Compute1(the first compute node) is
WHILE(I<20)
SET OutputRoot.MQMD.CorrelId =a value
PROPAGATE to TERMINAL 'out' ;
SET I=I+1 ;
END WHILE
for each value of I, get one message from MQget, then perform processing and put the message in MQoutput3. Since it has completed the whole path of propagate, the control comes back to the Compute1 and increments I and repeats the logic. Are you suggesting that this kind of looping is an issue. If so, can you please advise in this case, why the memory shoots up for each looping.
Can you please provide your valuable suggestions.! |
|
| Back to top |
|
|
| Gralgrathor |
| Posted: Tue Feb 11, 2014 9:58 pm Post subject: |
|
|
Master
Joined: 23 Jul 2009
Posts: 297
|
| Armageddon123 wrote: |
| then perform processing and put the message in MQoutput3 |
Is "MQoutput3" read by a flow in the same EG? Does that subsequent flow have multiple instances? Does any of the code in this flow or the next, or any of the flows in this chain, use SHARED or Environment variables for storing data?
Have you tested the flow to destruction? Have you been able to make it crash in a Test environment? What's the maximum amount of memory you've been able to see the EG consume, and what volume of traffic did this take?
Last edited by Gralgrathor on Wed Feb 12, 2014 12:14 am; edited 1 time in total |
|
| Back to top |
|
|
| Esa |
| Posted: Wed Feb 12, 2014 12:14 am Post subject: Re: Need help on WMB flow memory !! |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| Armageddon123 wrote: |
The flow is like this.
MQInput-Subflow1-Subflow2-Subflow3-Subflow3-MQputput
Subflow3 is the one which causes trouble.It is like this
Input-Compute1-MQGet-Compute2-Compute3-JavaCompute1-Compute4-MQoutput3
|
Subflow3 is also calling another instance of subflow3?
If you loop 20 times in the first instance, the other instance will be looping 400 times!
| Armageddon123 wrote: |
| Also,yes, the memory goes up when the same flow is triggered many times. |
Does the memory consumption continue to go up as aggressively as in the first time?
Do you have other flows running in the EG?
Message Broker version?
| Armageddon123 wrote: |
| 3.The code is actually very big!!,I tried to remove the client specific details from the code but it is never ending!!,, basically, it has a lot of IF conditons, SET statements etc, ASBITSTREAM function etc.. |
Hmm, ASBITSTREAM.... I suppose it also uses a lot of CREATE ... PARSE? |
|
| Back to top |
|
|
| Armageddon123 |
| Posted: Wed Feb 12, 2014 8:27 pm Post subject: |
|
|
Acolyte
Joined: 11 Feb 2014
Posts: 61
|
Hi Guys,,
thnx
I am rewriting the whole javacompute node into ESQL Compute .
I will update when I find some result!. |
|
| Back to top |
|
|
| Gralgrathor |
| Posted: Wed Feb 12, 2014 11:24 pm Post subject: |
|
|
Master
Joined: 23 Jul 2009
Posts: 297
|
| Armageddon123 wrote: |
| I am rewriting the whole javacompute node |
Then you've established that there is in fact a memory leak, and you've found the cause of that leak? |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Feb 13, 2014 12:19 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
|
| Back to top |
|
|
| Gralgrathor |
| Posted: Thu Feb 13, 2014 2:19 am Post subject: |
|
|
Master
Joined: 23 Jul 2009
Posts: 297
|
| Armageddon123 wrote: |
| I am rewriting the whole javacompute node into ESQL Compute |
Let me be a bit more specific:
I am not convinced that you've actually established that there is a leak.
Your flow may well be ill-designed, or contain some peculiar coding that consumes memory with each new message, but merely noticing an increase in the memory usage of the EG does not establish the presence of a leak, unless the memory usage *keeps* increasing to the point of collapse, after prolonged use.
Rewriting the code does not tell you whether there really is a problem, it does not identify the problem, nor does it help you to circumvent similar problems in the future.
It would be best if you had a more or less sufficient understanding of the situation before attempting a solution. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Feb 13, 2014 5:21 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Making a re-entrant call in the subflow is not a healthy proposition.
First design your flow so that subflow3 does not call subflow3 anymore. You will see that you get most bang for the buck on that one.
Did you inline the subflow3??
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Esa |
| Posted: Thu Feb 13, 2014 7:00 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| fjb_saper wrote: |
Making a re-entrant call in the subflow is not a healthy proposition.
First design your flow so that subflow3 does not call subflow3 anymore. You will see that you get most bang for the buck on that one.
Did you inline the subflow3??
Have fun |
The subflow is actually not directly calling itself - that would immediately crash the eg - but the output terminal of sublflow3 is wired to another instance of subflow3. That is a different thing. I suppose it's a typo in Armageddon123's post, but he hasn't confirmed or denied that.
Neither has he answered my questions:
| Esa wrote: |
Does the memory consumption continue to go up as aggressively as in the first time?
Do you have other flows running in the EG?
Message Broker version? |
|
|
| Back to top |
|
|
| Armageddon123 |
| Posted: Thu Feb 13, 2014 9:32 am Post subject: |
|
|
Acolyte
Joined: 11 Feb 2014
Posts: 61
|
Hi Guys,
Thanks for the comments!!, Please find the response below.
yes, the memory consumption goesup as agressively as first time.
there are two more flows in the same EG, but they are simple request/reply flows and is not memory intensive(i tested those flows by triggering around 100 times in test env, the CPU and memory doesnot shootup atall).
WMB version: 7.0.0.6
about the question of whether "output terminal of sublflow3 is wired to another instance of subflow3?".
i couldn't understand it completely. There is only one instance of the flow deployed.
flow uses propagate almostlike what is described in url
http://publib.boulder.ibm.com/infocenter/wmbhelp/v6r1m0/index.jsp?topic=%2Fcom.ibm.etools.mft.doc%2Fac12370_.htm
it even talks that memory is reclaimed when the propagate completes!!!!!!!!.)
"In the above example, the content of OutputRoot is reset before each PROPAGATE, because by default the node clears the output message buffer and reclaims the memory when the PROPAGATE statement completes" !
I searched on the manuals too for any clue. found this
http://publib.boulder.ibm.com/infocenter/wmbhelp/v6r1m0/index.jsp?topic=%2Fcom.ibm.etools.mft.doc%2Fau16534_.htm
under that this subject
"The execution group restarts before an MQGet node has retrieved all messages"
The solution mentioned over there is exactly what is done in the flow!!!!. So design wise, it appears ok to me but never know for sure!!!!!!,, it might be a small change somewhere which might resolve the issue!
The search continues!!!!!!!
Any comments or advise are most welcome !!!!!! |
|
| Back to top |
|
|
| Esa |
| Posted: Thu Feb 13, 2014 10:48 pm Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
Thank you for your answers, Armageddon123.
| Armageddon123 wrote: |
about the question of whether "output terminal of sublflow3 is wired to another instance of subflow3?".
i couldn't understand it completely. There is only one instance of the flow deployed.
|
This is how you describe the flow in your first post:
| Armageddon123 wrote: |
MQInput-Subflow1-Subflow2-Subflow3-Subflow3-MQputput
|
See?
MQInput-Subflow1-Subflow2-Subflow3-Subflow3-MQputput
I think you are right, it's obviously the JCN's that are causing the memory leak(s).
Causing memory leaks with JCN's is very easy, just forget to call .clearMessage() for one or all of the MbMessages you create.
When you create an MbMessage, the C code that is called via JNI allocates physical memory from the execution groups heap. (Edit: corrected JCN --> JNI)
There is no way the execution group can detect when the memory can be returned to the heap but your code calling clearMessage().
The best practice for avoiding this problem is to declare all MbMessage objects outside a try/catch block, instantiate the MbMessage
and do all work related to it within the try/catch block and call clearMessage() for each MbMessage in the finally block.
Last edited by Esa on Fri Feb 14, 2014 3:05 am; edited 2 times in total |
|
| Back to top |
|
|
| Armageddon123 |
| Posted: Fri Feb 14, 2014 1:58 am Post subject: |
|
|
Acolyte
Joined: 11 Feb 2014
Posts: 61
|
Hi Esa,
Oh, sorry , Subflow3-Subflow3 was a typo. Thankyou for pointing that!!!,
it is Subflow3-Subflow4 .
Now I am able to figure out all those questions from fjb_saper about the bad design!!,
sorry fjb_saper for getting your thoughts away from the real issue at hand!!!!!! ; Now that the typo is "corrected", would u like to provide any other advise for troubleshooting as i continue to rewrite the javacompute to esql?!!!!!,, |
|
| Back to top |
|
|
|
|
|
  |
Goto page 1, 2 Next |
Page 1 of 2 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
