| Author |
Message
|
| ramesh.patil |
 Posted: Fri Jul 06, 2018 11:11 pm Post subject: IIB 9 taking huge memory for the product feed flows |
 |
|
Newbie
Joined: 16 Dec 2015
Posts: 9
|
| We have few product feeds flows which receives the data from data base. As soon as the data retrieved from the data base and store in Row variable the data flow engine consumes huge memory (consumes around 4GB of memory for 170 MB data). For understanding the problem we have created simple flow with MQInput node one compute node which returns the result set and stores it in Row variable. For this simple operation the memory consumption is 4.8GB and the actual data is 148 MB. Can someone help me to identify the root . |
|
| Back to top |
|
 |
| abhi_thri |
| Posted: Mon Jul 09, 2018 12:32 am Post subject: |
|
|
Knight
Joined: 17 Jul 2017
Posts: 516
Location: UK
|
|
| Back to top |
|
|
| ramesh.patil |
| Posted: Tue Jul 10, 2018 11:06 pm Post subject: |
|
|
Newbie
Joined: 16 Dec 2015
Posts: 9
|
| We have confirmed that the ROW variable is consuming the huge data and it's not releasing the memory. we have tried multiple ways to delete the ROW variable fields (using Delete field and NULL) but no luck. We have raise PMR. Does anybody faced the same issue?if so can you please post solution. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jul 11, 2018 4:51 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ramesh.patil wrote: |
| We have confirmed that the ROW variable is consuming the huge data and it's not releasing the memory. |
Why do you need it to release the memory? Won't the flow need it when it runs again?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| ramesh.patil |
| Posted: Wed Jul 11, 2018 10:17 pm Post subject: |
|
|
Newbie
Joined: 16 Dec 2015
Posts: 9
|
| we have multiple feed file flows if each flow takes 4-5 GB memory then the whole RAM will be occupied with feeds flows. There is no RAM space left for other flows now. We want memory to be released so that we can have some space other flows. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jul 12, 2018 4:48 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Why not simply add more RAM to the server? If it's a virtual, it's one parameter.
Or to put it another way, how did you determine (before observing this behavior) that the server would be large enough for the amount of processing you're attempting? What sizing methodology did you use when deciding how much RAM to give the server?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| abhi_thri |
| Posted: Thu Jul 12, 2018 5:29 am Post subject: |
|
|
Knight
Joined: 17 Jul 2017
Posts: 516
Location: UK
|
I think OP's main concern/query is - why is retrieving 170MB of data from database to a ROW variable resulting in the Integration server using 4GB of memory (guess not all of 4GB are used for this operation - they need to run the flow with and without the database lookup/ROW scenario to see the difference).
I guess depending how the ROW variable is getting used by the flow (eg:- string operations) the flow will end up allocating more memory which it won't then release and that is expected...but if it is just the database lookup & assigning values to ROW variable that is taking such a high memory footprint then it needs a closer look (which IBM is investigating via PMR).
I couldn't find any explicit info about the recommended max size of the data one could store at ROW variables but below article discusses scenario where the performance peaks around 3000 rows (this could vary) when using SHARED ROW.
https://www.ibm.com/developerworks/websphere/library/techarticles/1405_dunn/1405_dunn.html |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jul 12, 2018 5:48 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| abhi_thri wrote: |
| I think OP's main concern/query is - why is retrieving 170MB of data from database to a ROW variable resulting in the Integration server using 4GB of memory (guess not all of 4GB are used for this operation - they need to run the flow with and without the database lookup/ROW scenario to see the difference). |
That's kind of where I was going asking about their sizing methodology.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| ramesh.patil |
| Posted: Thu Jul 12, 2018 11:59 pm Post subject: |
|
|
Newbie
Joined: 16 Dec 2015
Posts: 9
|
Here is my code.. We are just assigning the value to ROW variable.
CREATE FUNCTION Main() RETURNS BOOLEAN
BEGIN
DECLARE ResultsetRows ROW;
DECLARE CountResultSet INTEGER;
-- CALL CopyMessageHeaders();
-- CALL CopyEntireMessage();
set ResultsetRows.PRODUCT[] = PASSTHRU('select * from Product_catalog');
SET CountResultSet = CARDINALITY(ResultsetRows.PRODUCT[]);
SET Environment.Variables.CountResultSet = CountResultSet;
RETURN TRUE;
END;
And just to add the result set returns more than 300000 records and size of the file would be around 150-170 MB after writing |
|
| Back to top |
|
|
| timber |
| Posted: Fri Jul 13, 2018 1:36 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
I know it's only an experiment, and you probably know this already, but...
| Code: |
SET CountResultSet = CARDINALITY(ResultsetRows.PRODUCT[]);
SET Environment.Variables.CountResultSet = CountResultSet; |
Using the CARDINALITY function is almost always a bad idea. It's definitely a bad idea if you have 300000 records to iterate over.
That does not explain the memory consumption though; I agree that it looks larger than expected. IBM's usual advice is that a message tree can consume up to10 times the size of the raw data that hit is representing. In your case, that would suggest that a memory usage of 1.7Gb would be reasonable. |
|
| Back to top |
|
|
|
|
