ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » IIB v10 dataflow engine taking more than 6GB memory.

Post new topic  Reply to topic
 IIB v10 dataflow engine taking more than 6GB memory. « View previous topic :: View next topic » 
Author Message
ramesh4800
PostPosted: Fri Mar 30, 2018 1:07 am    Post subject: IIB v10 dataflow engine taking more than 6GB memory. Reply with quote

Newbie

Joined: 05 Jan 2015
Posts: 5

We are processing large xml files by using soap request nodes.and EG level heap size is 1GB. There is no any additional instances,but still it is taking 6GB memory for dataflow engine.
Why this much 6GB memory is allocated?? it is not releasing although is it completed the process.
my question is why 6GB if we allocated only 1 GB on EG heap.What all the resources using memory???

thanks in advance.
Back to top
View user's profile Send private message
joebuckeye
PostPosted: Fri Mar 30, 2018 3:46 am    Post subject: Reply with quote

Partisan

Joined: 24 Aug 2007
Posts: 364
Location: Columbus, OH

My very unofficial answer that could be based on old information (since it's been awhile since we had to look at this) is that parts of IIB are written in Java and parts are in C/C++. I believe the EG heap size parameter is for the Java bits.

And once the DataFlowEngine allocates memory it never releases it back to the system because it assumes (not without reason) that if you needed that much memory once that you will need it again so it keeps it.

Again, we haven't had memory issues since v7/v8 so not sure if this is still true in v10 (we skipped v9).
Back to top
View user's profile Send private message
mpong
PostPosted: Fri Mar 30, 2018 2:06 pm    Post subject: Reply with quote

Disciple

Joined: 22 Jan 2010
Posts: 164

How does WebSphere Message Broker (WMB) handle the memory usage within a DataFlowEngine (DFE) Process?
When considering memory usage within a DataFlowEngine process there are two sources that the storage is allocated from, and these are :
1. The DataFlowEngine main memory heap
2. The operating system

When message flow processing requires some storage, then an attempt is first made to allocate the memory block required from the DataFlowEngine's heap. If there is not a large enough contiguous block of storage on the heap, then a request will be made to the operating system to allocate more storage to the DataFlowEngine for the message flow. Once this is done, then this would lead to the DataFlowEngine's heap growing with the additional storage, and the message flow will use this extra storage.
When the message flow has completed its processing, then it issues a "free" on all its storage and these blocks will be returned to the DataFlowEngine's heap ready for allocation to any other message flows of this DataFlowEngine. The storage is never released back to the operating system, because there is actually no programmatic mechanism to perform such an operation. The operating system will not retrieve storage from a process until the process is terminated. Therefore the user will never see the size of the DataFlowEngine process decrease, after it has increased.
When the next message flow runs, then it will make requests for storage, and these will then be allocated from the DataFlowEngine heap as before. Therefore there will be a re-use within the DataFlowEngine's internal storage where possible, minimizing the number of times that the operating system needs to allocate additional storage to the DataFlowEngine process. This would mean there may be some growth observed on DataFlowEngine's memory usage which is of the size of the subsequent allocations for message flow processing. Eventually we would expect the storage usage to plateau, and this situation would occur when the DataFlowEngine has a large enough heap such that any storage request can be satisfied without having to request more from the operating system.
Back to top
View user's profile Send private message
abhi_thri
PostPosted: Sat Mar 31, 2018 2:19 am    Post subject: Re: IIB v10 dataflow engine taking more than 6GB memory. Reply with quote

Knight

Joined: 17 Jul 2017
Posts: 516
Location: UK

hi...others have already explained on how broker allocates memory. The size of the message, how it is getting parsed and what else is esql doing with it - all contributes to the 6GB usage figure.

Have a look at below articles which may help in reducing the footprint,

https://www.ibm.com/support/knowledgecenter/SSMKHH_10.0.0/com.ibm.etools.mft.doc/ac67176_.htm

https://developer.ibm.com/integration/blog/2014/06/30/techniques-to-minimize-memory-usage-with-iib/

https://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=6&cad=rja&uact=8&ved=0ahUKEwiAjLy3qZbaAhUPMuwKHfT1C-kQFghyMAU&url=https3A%2F2Fshare.confex.com%2Fshare%2F124%2Fwebprogram%2FHandout%2FSession17043%2FThe%2520Dos%2520and%2520Donts%2520of%2520IBM%2520Integration%2520Bus%2520Performance%2520-%2520Wed%2520040315%2520315pm.pdf&usg=AOvVaw1cCl3vGkOWB1I58nVG3WS5
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » IIB v10 dataflow engine taking more than 6GB memory.
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.