| Author |
Message
|
| kash3338 |
 Posted: Fri Dec 21, 2012 9:48 am Post subject: Exg Grp Heap Size |
 |
|
Shaman
Joined: 08 Feb 2009
Posts: 709
Location: Chennai, India
|
Hi,
I just have a doubt on the Execution Group heap size.
There are scenarios wherein if the input payload is too large or if there is a complex recursive logic in the flow (and the flow does not have any nodes related to Java usage) and if the EG crashes or hangs, then one of the solution suggested is to increase the heap size of the execution group.
But AFAIK the heap size of a execution group refers to the JVM heap size, which has effect only if either of three scenarios is in the message flow,
1. A Java user-defined extension node
2. A JavaCompute node
3. A Java routine that is called from ESQL
Does the JVM heap size has something to do with the the processing of the flows (even without Java usage) in the EG? |
|
| Back to top |
|
 |
| lancelotlinc |
| Posted: Fri Dec 21, 2012 10:00 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
The heap sizing you are talking about is related only to the JVM where Java is used in your flow, as you have indicated in your original post which I read thoroughly.
The C-runtime memory usage is what you want to affect, as far as I know, is automatically driven. Someone else familiar with the source code could comment here, like mgk.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Dec 21, 2012 10:06 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Never assume that any given function in Broker is written in C++ or in Java.
Several of the built-in nodes are written in Java, in particular (and hopefully rather obviously) those nodes that require you to use a JDBC connection (databaseroute, databaseretrieve). |
|
| Back to top |
|
|
| kash3338 |
| Posted: Fri Dec 21, 2012 6:56 pm Post subject: |
|
|
Shaman
Joined: 08 Feb 2009
Posts: 709
Location: Chennai, India
|
| mqjeff wrote: |
Never assume that any given function in Broker is written in C++ or in Java.
Several of the built-in nodes are written in Java, in particular (and hopefully rather obviously) those nodes that require you to use a JDBC connection (databaseroute, databaseretrieve). |
Agreed. But that still does not answer my question  |
|
| Back to top |
|
|
| kimbert |
| Posted: Sat Dec 22, 2012 2:00 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Complex recurive logic sounds like it might be bad design. A loop with an MQGet node perhaps? |
|
| Back to top |
|
|
| mgk |
| Posted: Sat Dec 22, 2012 4:08 am Post subject: |
|
|
Padawan
Joined: 31 Jul 2003
Posts: 1642
|
If you could post a link to the advice you mention I could take a look. As suggested above, there is another category missing from your list, but otherwise you are correct:
4: Java usage by built in Broker nodes. Some nodes (e.g. SOAP, Collector) make use of Java internally.
Also as mentioned above, the C Runtime heap is dynamically driven, expanding as necessary based on demand.
I hope this helps.
_________________
MGK
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sat Dec 22, 2012 5:13 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| kash3338 wrote: |
| mqjeff wrote: |
Never assume that any given function in Broker is written in C++ or in Java.
Several of the built-in nodes are written in Java, in particular (and hopefully rather obviously) those nodes that require you to use a JDBC connection (databaseroute, databaseretrieve). |
Agreed. But that still does not answer my question |
I thought your question was
| Quote: |
| Does the JVM heap size has something to do with the the processing of the flows (even without Java usage) in the EG? |
And my answer was "You can't assume that there is no Java usage". So you can't say "even without Java usage".
So, yes, the JVM heap size has something to do with the processing of the flows.
But more importantly, if the EG hangs or crashes, you should debug the cause, rather than twiddle dials and hope you get it right.
And, again, complex recursive logic is, uhm, ... complex... and should be handled with care and designed carefully and implemented even more carefully. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Sat Dec 22, 2012 12:40 pm Post subject: Re: Exg Grp Heap Size |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| kash3338 wrote: |
| I just have a doubt on the Execution Group heap size. |
English tip: "question" is probably a better word here. "doubt" implies misgivings, mistrust, suspicion, alarm, fear, or apprehension.
"heap size" can either refer to the JVM heap, or to the main memory heap. Execution groups typically have a fixed capacity at runtime, (the details of which are OS-dependent), so increasing the JVM heap size would reduce the memory available for the main heap. If the EG crashes or hangs, then another suggestion might be to decrease the JVM heap size; it all depends on where the message flow might be running out of memory.
| mgk wrote: |
| 4: Java usage by built in Broker nodes. Some nodes (e.g. SOAP, Collector) make use of Java internally. |
4. ... not to be overlooked, the XSLTransform node can make heavy use of the JVM heap, particularly for large messages.
5. As of WMBv8, the global cache also uses the JVM heap. |
|
| Back to top |
|
|
| kash3338 |
| Posted: Sun Dec 23, 2012 8:00 am Post subject: Re: Exg Grp Heap Size |
|
|
Shaman
Joined: 08 Feb 2009
Posts: 709
Location: Chennai, India
|
| rekarm01 wrote: |
| English tip: "question" is probably a better word here. "doubt" implies misgivings, mistrust, suspicion, alarm, fear, or apprehension. |
Thanks reka for the tip, but here I ment doubt (which means "Feel uncertain about.") as I was uncertain about the concept of EG Heap Size, hence I guess I was right.
| rekarm01 wrote: |
"heap size" can either refer to the JVM heap, or to the main memory heap. Execution groups typically have a fixed capacity at runtime, (the details of which are OS-dependent), so increasing the JVM heap size would reduce the memory available for the main heap. If the EG crashes or hangs, then another suggestion might be to decrease the JVM heap size; it all depends on where the message flow might be running out of memory. |
Yes even I have gone through the link and have the same understanding but my question raises because in few projects I have seen one of the solution pointed by few architects is to increase the EG Heap Size in case when the EG hangs due to recursive logic or heavy payload. Hence I was in a doubt about the concept of heap size in EG's.
| rekarm01 wrote: |
4. ... not to be overlooked, the XSLTransform node can make heavy use of the JVM heap, particularly for large messages.
5. As of WMBv8, the global cache also uses the JVM heap. |
Agreed.
| kimbert wrote: |
Complex recurive logic sounds like it might be bad design. A loop with an MQGet node perhaps? |
Thanks for the suggestion kimbert but I do not face a problem with a recursive logic now, it was just a example i quoted for the reason to increase the EG heap size.
| mgk wrote: |
If you could post a link to the advice you mention I could take a look. |
Thanks for your response mgk, unfortunatly I do not have any links which states this, these options of increasing the EG heap size is what I come across from few architects as a solution in case of the listed issues.
Increasing the EG Heap Size is a solution that many have suggested even in cases when there is just a compute node used (but there might be a complex recursive logic or a huge loop). So just wanted to know what this heap size is since all I know is only about JVM heap size in EG (and as pointed by many here, is used only in few cases wherein a Java usage is involved either through the node internally or through the code). |
|
| Back to top |
|
|
| visasimbu |
| Posted: Sun Dec 23, 2012 10:01 am Post subject: Re: Exg Grp Heap Size |
|
|
Disciple
Joined: 06 Nov 2009
Posts: 171
|
| rekarm01 wrote: |
"heap size" can either refer to the JVM heap, or to the main memory heap. Execution groups typically have a fixed capacity at runtime, (the details of which are OS-dependent), so increasing the JVM heap size would reduce the memory available for the main heap.
|
I Just got the information in below URL which says, reduce the JVM heap size to process the larger message in message flow. By reducing the heap size, we will get the more heap size in the main memory heap.
Increase the heap memory, if we use the Java in message flow.
http://publib.boulder.ibm.com/infocenter/wmbhelp/v8r0m0/index.jsp?topic=%2Fcom.ibm.etools.mft.doc%2Fac55070_.htm
Will our main memory heap will get reduced, if we create the new EG ? |
|
| Back to top |
|
|
| kimbert |
| Posted: Mon Dec 24, 2012 3:24 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Most operating systems allow the maximum stack size to be adjusted. Maybe 'stack' and 'heap' have got confused somewhere along the line? |
|
| Back to top |
|
|
| kash3338 |
| Posted: Mon Dec 24, 2012 4:47 am Post subject: |
|
|
Shaman
Joined: 08 Feb 2009
Posts: 709
Location: Chennai, India
|
| kimbert wrote: |
| Most operating systems allow the maximum stack size to be adjusted. Maybe 'stack' and 'heap' have got confused somewhere along the line? |
Thanks for your point kimbert. I think thats exactly what I was looking for. I did a search from your point and got this link on Stack size,
http://publib.boulder.ibm.com/infocenter/wmbhelp/v7r0m0/index.jsp?topic=%2Fcom.ibm.etools.mft.doc%2Fac55020_.htm which says,
| Quote: |
In WebSphere® Message Broker, any processing that involves nested or recursive processing can cause extensive usage of the stack. For example, in the following situations you might need to increase the stack size:
•When a message flow is processing a message that contains a large number of repetitions or complex nesting.
•When a message flow is executing ESQL that calls the same procedure or function recursively, or when an operator (for example, the concatenation operator) is used repeatedly in an ESQL statement.
You can increase the stack size to improve performance.
|
But, where is this Stack Size set? As per the link, it is for a message flow, so does that mean there is nothing done on the EG level in senario's like recursive logic? |
|
| Back to top |
|
|
| kash3338 |
| Posted: Wed Jan 02, 2013 12:03 am Post subject: |
|
|
Shaman
Joined: 08 Feb 2009
Posts: 709
Location: Chennai, India
|
| I would like to retrigger this thread so that some of the experts here can clarify on this still pending discussion. |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Jan 02, 2013 1:07 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
If you are recursing to read a queue, this design will exhaust the heap size at some point depending on the queue depth.
Better to invoke the MQGET from a propagate statement in the compute node and make the loop inside the ESQL. |
|
| Back to top |
|
|
| kash3338 |
| Posted: Wed Jan 02, 2013 4:15 am Post subject: |
|
|
Shaman
Joined: 08 Feb 2009
Posts: 709
Location: Chennai, India
|
| zpat wrote: |
If you are recursing to read a queue, this design will exhaust the heap size at some point depending on the queue depth.
Better to invoke the MQGET from a propagate statement in the compute node and make the loop inside the ESQL. |
Thanks zpat. But I do not have any particular issue with looping through a MQ queue. I just took that as one of a scenario, if you can read through this complete thread. |
|
| Back to top |
|
|
|
|
