| Author |
Message
|
| gakana |
 Posted: Tue Jan 03, 2006 8:42 am Post subject: DataFlowEngine takes 70% CPU Utalization |
 |
|
Apprentice
Joined: 27 Jul 2005
Posts: 28
Location: Michigan
|
Hi,
DataFlowEngine takes more than 70% of CPU utalization on Windows XP and as well as on AIX5.2.2
We are on MB 5.0.4 and processing messages like around 170KB xml files.
Is there anyway to reduce the CPU utalization?
Please give me your valuable suggestions.
Thanks
Gakana |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Tue Jan 03, 2006 8:53 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
The most likely culprit is your message flow.
Are you doing lots of loops?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| gakana |
| Posted: Tue Jan 03, 2006 9:17 am Post subject: DataFlowEngine takes 70% of CPU Utalization |
|
|
Apprentice
Joined: 27 Jul 2005
Posts: 28
Location: Michigan
|
Thanks for quick reply
We have only one while loop in the esql code.
We are getting XML file as input and converting it into MRM and did some mappings and some esql coding.
We used one Input node, one ResetContentDescripter node, one mapping node, one compute node and then one output node and one fail node in my flow.
We are getting more than 35 messages/sec for the flow input queue. We have increased the additional instances for the flow also. But still the CPU utilization is the same.
Is there any best way to reduce the this?
I would be appreciate your help
Thanks
Gakana |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Jan 03, 2006 9:20 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Is it using lots of CPU when it's not processing any messages?
Do you have any tracing enabled?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| gakana |
| Posted: Tue Jan 03, 2006 9:41 am Post subject: DataFlowEngine takes 70% cCPU utalization |
|
|
Apprentice
Joined: 27 Jul 2005
Posts: 28
Location: Michigan
|
It is processing well, but the flow avg processing rate is 4 messages/sec. In earliar the input queue is queued up then we increased the additional instances. Now it is processing well and but CPU utalization is going to up like around 70 to 80%. Trace is not enabled.
Thanks
Gakana |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Jan 03, 2006 9:43 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Again, if you do something like temporarily get-disable the queue, so that your message flow won't process new messages, does the CPU still stay high? Or does it back down to reasonable numbers?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| gakana |
| Posted: Tue Jan 03, 2006 9:48 am Post subject: DataFlowEngine takes 70% CPU Utalization |
|
|
Apprentice
Joined: 27 Jul 2005
Posts: 28
Location: Michigan
|
When ever we are sending messages to flow inqueue, the CPU utalization going to up and then when ever it completing this process, it comes to down.
Thanks
Gakana |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Jan 03, 2006 9:59 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Okay, so it seems like it's your flow more than anything else.
What - generally - are you doing in your compute node? How complex is your message model - and how complex is the mapping that you're doing?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| gakana |
| Posted: Tue Jan 03, 2006 10:17 am Post subject: DataFlowEngine takes 70% CPU Utalization |
|
|
Apprentice
Joined: 27 Jul 2005
Posts: 28
Location: Michigan
|
Thank you very much for your help and quick replies.
Our message flows typically have the below pattern of nodes -
Input -> ResetContentDescriptor -> Map -> Compute -> Output
The esql in the compute node is fairly complex. There is only one loop that typically has few iterations, but there is much transformation happening for each message both within the mapping node and the compute node.
And to clarify - when not processing messages, the CPU utilization goes down to the usual 0-5% (95-99% idle).
One question - our message sets/BOs get sort of large, because we try to make them generic so that they're more maintainable. If we had a minimal message set specific to each message flow, would that make a considerable difference? The same segments/fields would still be used in the actual message, but we could strip out segments/fields from the BO/message set that don't ever get used.
Thanks
Gakana |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Jan 03, 2006 10:45 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
One of the easy spots for improving message flow performance is to use references wherever possible - particularly for accessing stuff in repeating data structures. Every time you use the bracket subscripts, the ESQL has to manually navigate through the message tree to get to the right subscript.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| gakana |
| Posted: Tue Jan 03, 2006 10:58 am Post subject: |
|
|
Apprentice
Joined: 27 Jul 2005
Posts: 28
Location: Michigan
|
I have read on documentation the default heap size of the execution group is 512MB. Could I increase this size on widow' s broker execution group
Thanks
Gakana |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Jan 03, 2006 11:14 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| gakana wrote: |
| I have read on documentation the default heap size of the execution group is 512MB. Could I increase this size on widow' s broker execution group |
I didn't think so, at first.
But then I saw an odd little note in the XSLTTranformation node documentation..
http://publib.boulder.ibm.com/infocenter/wbihelp/v6rxmx/index.jsp?topic=/com.ibm.etools.mft.doc/ac12490_.htm
| Quote: |
If you are dealing with large XML messages and receiving an out of memory error, you can use the mqsichangeproperties command to increase the Java Heap size for the XSLT engine. For example, using the following code:
mqsichangeproperties brokerName -e executionGroupLabel -o ComIbmJVMManager -n jvmMaxHeapSize -v newSize
replacing brokerName, executionGroupLabel and newSize with the appropriate values. |
This is clearly EG specific.
Do you see indications that memory paging is occurring excessively - causing your CPU utilization problem?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| sknrt1 |
| Posted: Tue Jan 03, 2006 12:46 pm Post subject: does any of the tag xml has data in form of entity reference |
|
|
Apprentice
Joined: 22 Jan 2003
Posts: 39
Location: USA
|
Does any of the filed/tag/element in your input xml messages has data in form of entity reference.
ie: xml tag containing large xml data (escaped) in form of entity reference
refer to the value of 'message' tag below
<message><siebel-xmlext-query-et><row>< ..............................................</message>
when an XML tag contains a large amount of entity references, broker process (takes very long time to parse) can cause CPU usage to be maxed out. Once processing is done, CPU will be back to normal. If all the input msg's are of same style, you will see high cpu usage for long time.
We had this scenario (in v2.1 & v5.0), got a patch from IBM and looks good from then.
If this is not the case, please ignore.
Thanks |
|
| Back to top |
|
|
| gakana |
| Posted: Tue Jan 03, 2006 12:51 pm Post subject: |
|
|
Apprentice
Joined: 27 Jul 2005
Posts: 28
Location: Michigan
|
I have already did this using this mqsichangeproperties command, even though no change.
Still it is taking the 70%
I did not see any indications that memory paging
Thanks
Gakana |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Jan 03, 2006 1:26 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
If in any way your messages match what sknrt1 describes - you should open a PMR with IBM and get the fix and try it.
Probably, you should open a PMR anyway.
I'm not surprised that adding to the heap didn't solve the problem. CPU doesn't usually get bad like this because of low memory - and when it does it's pretty apparent that it's a memory condition.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
|
|
