ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker SupportIIB 9 taking huge memory for the product feed flows

Post new topicReply to topic
IIB 9 taking huge memory for the product feed flows View previous topic :: View next topic
Author Message
ramesh.patil
PostPosted: Fri Jul 06, 2018 11:11 pm Post subject: IIB 9 taking huge memory for the product feed flows Reply with quote

Newbie

Joined: 16 Dec 2015
Posts: 7

We have few product feeds flows which receives the data from data base. As soon as the data retrieved from the data base and store in Row variable the data flow engine consumes huge memory (consumes around 4GB of memory for 170 MB data). For understanding the problem we have created simple flow with MQInput node one compute node which returns the result set and stores it in Row variable. For this simple operation the memory consumption is 4.8GB and the actual data is 148 MB. Can someone help me to identify the root .
Back to top
View user's profile Send private message
abhi_thri
PostPosted: Mon Jul 09, 2018 12:32 am Post subject: Reply with quote

Disciple

Joined: 17 Jul 2017
Posts: 199
Location: UK

hi...how are you storing the data, is it by loading the data to a shared row only once or do you hit database every time?

If the flow logic is repopulating the shared row then you need to ensure that the memory is released, otherwise it will keep building up and will only be released when the flow is stopped/restarted. The below thread discusses how to do that,

http://mqseries.net/phpBB/viewtopic.php?t=68951&postdays=0&postorder=asc&start=0&sid=9fb341a87313ade8e0b6516f1d2c3909

Also worth looking at the below ibm article on minimizing memory usage,

https://developer.ibm.com/integration/blog/2014/06/30/techniques-to-minimize-memory-usage-with-iib/
Back to top
View user's profile Send private message
ramesh.patil
PostPosted: Tue Jul 10, 2018 11:06 pm Post subject: Reply with quote

Newbie

Joined: 16 Dec 2015
Posts: 7

We have confirmed that the ROW variable is consuming the huge data and it's not releasing the memory. we have tried multiple ways to delete the ROW variable fields (using Delete field and NULL) but no luck. We have raise PMR. Does anybody faced the same issue?if so can you please post solution.
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Jul 11, 2018 4:51 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25114
Location: Ohio, USA

ramesh.patil wrote:
We have confirmed that the ROW variable is consuming the huge data and it's not releasing the memory.


Why do you need it to release the memory? Won't the flow need it when it runs again?
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
ramesh.patil
PostPosted: Wed Jul 11, 2018 10:17 pm Post subject: Reply with quote

Newbie

Joined: 16 Dec 2015
Posts: 7

we have multiple feed file flows if each flow takes 4-5 GB memory then the whole RAM will be occupied with feeds flows. There is no RAM space left for other flows now. We want memory to be released so that we can have some space other flows.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Jul 12, 2018 4:48 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25114
Location: Ohio, USA

Why not simply add more RAM to the server? If it's a virtual, it's one parameter.

Or to put it another way, how did you determine (before observing this behavior) that the server would be large enough for the amount of processing you're attempting? What sizing methodology did you use when deciding how much RAM to give the server?
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
abhi_thri
PostPosted: Thu Jul 12, 2018 5:29 am Post subject: Reply with quote

Disciple

Joined: 17 Jul 2017
Posts: 199
Location: UK

I think OP's main concern/query is - why is retrieving 170MB of data from database to a ROW variable resulting in the Integration server using 4GB of memory (guess not all of 4GB are used for this operation - they need to run the flow with and without the database lookup/ROW scenario to see the difference).

I guess depending how the ROW variable is getting used by the flow (eg:- string operations) the flow will end up allocating more memory which it won't then release and that is expected...but if it is just the database lookup & assigning values to ROW variable that is taking such a high memory footprint then it needs a closer look (which IBM is investigating via PMR).

I couldn't find any explicit info about the recommended max size of the data one could store at ROW variables but below article discusses scenario where the performance peaks around 3000 rows (this could vary) when using SHARED ROW.

https://www.ibm.com/developerworks/websphere/library/techarticles/1405_dunn/1405_dunn.html
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Jul 12, 2018 5:48 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25114
Location: Ohio, USA

abhi_thri wrote:
I think OP's main concern/query is - why is retrieving 170MB of data from database to a ROW variable resulting in the Integration server using 4GB of memory (guess not all of 4GB are used for this operation - they need to run the flow with and without the database lookup/ROW scenario to see the difference).


That's kind of where I was going asking about their sizing methodology.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
ramesh.patil
PostPosted: Thu Jul 12, 2018 11:59 pm Post subject: Reply with quote

Newbie

Joined: 16 Dec 2015
Posts: 7

Here is my code.. We are just assigning the value to ROW variable.

CREATE FUNCTION Main() RETURNS BOOLEAN
BEGIN
DECLARE ResultsetRows ROW;
DECLARE CountResultSet INTEGER;
-- CALL CopyMessageHeaders();
-- CALL CopyEntireMessage();

set ResultsetRows.PRODUCT[] = PASSTHRU('select * from Product_catalog');
SET CountResultSet = CARDINALITY(ResultsetRows.PRODUCT[]);
SET Environment.Variables.CountResultSet = CountResultSet;

RETURN TRUE;
END;

And just to add the result set returns more than 300000 records and size of the file would be around 150-170 MB after writing
Back to top
View user's profile Send private message
timber
PostPosted: Fri Jul 13, 2018 1:36 am Post subject: Reply with quote

Sentinel

Joined: 25 Aug 2015
Posts: 817

I know it's only an experiment, and you probably know this already, but...
Code:
SET CountResultSet = CARDINALITY(ResultsetRows.PRODUCT[]);
SET Environment.Variables.CountResultSet = CountResultSet;

Using the CARDINALITY function is almost always a bad idea. It's definitely a bad idea if you have 300000 records to iterate over.

That does not explain the memory consumption though; I agree that it looks larger than expected. IBM's usual advice is that a message tree can consume up to10 times the size of the raw data that hit is representing. In your case, that would suggest that a memory usage of 1.7Gb would be reasonable.
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexWebSphere Message Broker SupportIIB 9 taking huge memory for the product feed flows
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.