| Author |
Message
|
| mqsiuser |
 Posted: Wed Nov 19, 2014 8:18 am Post subject: Delete *and free memory* from a SHARED ROW |
 |
|
Yatiri
Joined: 15 Apr 2008
Posts: 637
Location: Germany
|
Dear experts,
we use Broker 7.0.0.6 and we use a SHARED ROW to cache data from a db (a couple of rows, ~few 100 to 1000).
A TimeoutNotification node triggers the cache-refresh/reload every 10 seconds.
During every cache-reload I try to release (and reuse) the memory from the SHARED ROW, but without success.
Our memory grows over time to hundreds of MB, then GBs on MQROOT (that must be where the SHARED ROW(s) reside). We can see it very well in the Message Broker Explorer.
Here is an APAR: http://www-01.ibm.com/support/docview.wss?uid=swg1IC87560
But should be fixed in 7.0.0.6
What should work (regardless of whether it's a bug in WMB):
A)
| Code: |
| SET mySharedRow = NULL; |
B)
| Code: |
| DELETE FIELD mySharedRow; |
C)
| Code: |
| SET mySharedRow.Rows = NULL; |
D)
| Code: |
| DELETE FIELD mySharedRow.Rows; |
Or combinations of the above ?
Thank you,
mqsiuser
_________________
Just use REFERENCEs |
|
| Back to top |
|
 |
| McueMart |
| Posted: Wed Nov 19, 2014 8:51 am Post subject: |
|
|
Chevalier
Joined: 29 Nov 2011
Posts: 490
Location: UK...somewhere
|
Is all your data under mySharedRow.Rows? i.e.
mySharedRow.Rows.Data1 = 'foo';
mySharedRow.Rows.Data2 = 'bar';
If so, then I would say that A , C and D would be fine. If you have
mySharedRow.Rows[0].Data = 'foo';
mySharedRow.Rows[1].Data = 'bad';
then I would say only A is going to clean it all up. |
|
| Back to top |
|
|
| maurito |
| Posted: Wed Nov 19, 2014 9:01 am Post subject: |
|
|
Partisan
Joined: 17 Apr 2014
Posts: 358
|
DELETE FIELD and SET xxx = NULL are not the same.
one of releases the memory, the other does not. if I remember well , it has already been discussed in this forum |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Nov 19, 2014 9:02 am Post subject: Re: Delete *and free memory* from a SHARED ROW |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqsiuser wrote: |
| During every cache-reload I try to release (and reuse) the memory from the SHARED ROW, but without success. |
All of those statements delete shared row elements from the tree to one extent or another.
Memory management is under the control of the broker runtime, so there's no guarantee that just because you've removed the elements the runtime is reclaim the memory. The only thing guaranteed to reclaim the memory is a reload of the EG.
More details on error handling (and possible tricks to make the runtime reclaim memory) will need a PMR.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Armageddon123 |
| Posted: Wed Nov 19, 2014 9:48 am Post subject: |
|
|
Acolyte
Joined: 11 Feb 2014
Posts: 61
|
Please see the post.
http://www.mqseries.net/phpBB/viewtopic.php?p=374834&highlight=#374834
Here Esa had mentioned that
| Quote: |
SET NULL doesn't free the memory immediately, it just detaches the element. The memory is released later when the flow instance terminates.
This command releases the memory immediately:
Code:
DELETE FIELD Environment.Data; |
We ,in our project, had tested the same then too. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Nov 19, 2014 9:58 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Armageddon123 wrote: |
Please see the post.
http://www.mqseries.net/phpBB/viewtopic.php?p=374834&highlight=#374834
Here Esa had mentioned that
| Quote: |
SET NULL doesn't free the memory immediately, it just detaches the element. The memory is released later when the flow instance terminates.
This command releases the memory immediately:
Code:
DELETE FIELD Environment.Data; |
We ,in our project, had tested the same then too. |
I don't think that's the case.
If it is, then you can use that to solve your problem.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqsiuser |
| Posted: Wed Nov 19, 2014 11:06 am Post subject: |
|
|
Yatiri
Joined: 15 Apr 2008
Posts: 637
Location: Germany
|
| Armageddon123 wrote: |
This command releases the memory immediately:
Code:
DELETE FIELD Environment.Data;
We ,in our project, had tested the same then too. |
I use a SHARED ROW (not the Environment) and I used / tested that.
I was almost sure it would work.
Especially since we are on 7.0.0.6 (which includes an APAR that has fixed (basically my) issue).
I changed the code from "SET mySharedRow = NULL;" to "DELETE FIELD mySharedRow;" (didn't help!)
I can move everything "down" by one (enclosing) element.
Will this help? "DELETE FIELD mySharedRow.Rows"
I will try.
_________________
Just use REFERENCEs |
|
| Back to top |
|
|
| mqsiuser |
| Posted: Fri Nov 21, 2014 12:25 am Post subject: Re: Delete *and free memory* from a SHARED ROW |
|
|
Yatiri
Joined: 15 Apr 2008
Posts: 637
Location: Germany
|
Thank you all for aiding the discussion, ... holding my hand... and nearing the cause (and solution) !
I have tested it ("DELETE FIELD mySharedRow.Rows;") and it doesn't work.
The memory is still growing (beyond 1 GB) though "slower" than before.
I think Vitor is right (experince wins (this time)):
| Vitor wrote: |
Memory management is under the control of the broker runtime, so there's no guarantee that just because you've removed the elements the runtime is reclaim the memory. The only thing guaranteed to reclaim the memory is a reload of the EG.
More details on error handling (and possible tricks to make the runtime reclaim memory) will need a PMR. |
| Vitor wrote: |
| I don't think that's the case. |
Vitor doesn't think that "DELETE FIELD" on a SHARED ROW recylces  the Memory.
Though I think this is a bug (in WMB). As described in the ARPA (link above) also SHARED ROWs should release Memory (to be reused later) *imho*.
We will open a PMR.
_________________
Just use REFERENCEs |
|
| Back to top |
|
|
| maurito |
| Posted: Fri Nov 21, 2014 1:20 am Post subject: |
|
|
Partisan
Joined: 17 Apr 2014
Posts: 358
|
| You also need to consider that the memory growth maybe be due to another factor. How do you build your shared ? |
|
| Back to top |
|
|
| Craig B |
| Posted: Fri Nov 21, 2014 2:53 am Post subject: |
|
|
Partisan
Joined: 18 Jun 2003
Posts: 316
Location: UK
|
Hi,
As previous replies have mentioned DELETE FIELD is the correct approach here. SETting any message tree field to NULL will detach the elements from the message tree. It does not free them for re-use.
The other observations are also correct in that the default broker memory management in this area is to keep objects and re-use them where it can.
When a SET to NULL is used, then this will detach the message tree fields.
This means that when you reload the SHARED ROW variable with new content a new set of message tree fields would be created. At this point the old set would be in memory and the new set would be in memory. As each content reload is processed the number of message tree fields would keep growing.
DELETE FIELD frees the existing message tree fields for re-use. It does not delete the memory and free it to the heap. This has memory advantages such that the new content will re-use the freed message tree fields that were previously deleted. Obviously if the next content reload has more content then more fields will be used and so more memory will be used (this might come from other free blocks on the DataFlowEngine heap or the heap may be extended by the OS to satisfy the request).
From what you have described it sounds like you are SETting fields in the SHARED ROW variable. This population by tree copy does not create new parsers in the target ROW variable and as such I think we are only talking about the single Root parser that represents the ROW variable. This should simplify the scenario as we have one parser that does not need to be re-used and the fields within in do need to be re-used.
It is possible to create nested parsers within a SHARED ROW using the CREATE statement with the DOMAIN clause. If this was being used then the scope of the nested parser would be important. Prior to V7.0.0.4, if a nested parser was in a shared ROW variable and the root element of the parser was deleted then this did not free the nested parser for re-use. As such we would "grow" the amount of cached parsers. This was fixed in 7.0.0.4 such that if the root element of a parser was deleted and no elements were in use, then the parser would be freed for re-use.
There was also an issue whereby if a message tree was copied into a SHARED ROW variable where there were existing fields already, then these would be detached instead of deleted. I believe this was fixed in 7.0.0.6 but I am not sure this applies to your scenario as you are already exploring the options of clearing the existing content first.
In V7.0.0.7 the logic from shared ROW variables is expanded to all message trees and ROW variables. This means that for any parser in any message tree, if the Root element is deleted and all other elements then the parser will be freed for re-use.
At this point I will re-iterate that the objects remain in scope and the space is not freed in the heap. The saving here is that new parsers/fields are not created if they are freed for re-use.
Other discussions will talk about MQSI_FREE_MASTER_PARSERS and this being the answer to this type of growth. When this environment variable is set it causes parsers/fields to be deleted when the message flow finishes processing the current input message. This actually deletes the objects and as such frees blocks on the heap. However, the broker does not implement any functionality that would return memory back to the operating system. So if the heap grows to a certain point it is unlikely to shrink back down again. For those cases where the heap does reduce this will be down to the OS. Or it could be that a user is monitoring the wrong memory metric and real memory is swapped out and as such numbers go down.
In summary here if you are using DELETE FIELD and you process your peak number of rows from the database, then you would expect a plateau in memory usage. Currently you are not getting this and as such you suspect an issue with the SHARED ROW variable. You could raise PMR for this so that it is investigated.
Before doing this you could examine the parser statistics that can be viewed in MBX. While these stats do not show the number of parser instances for each type it does show the total number of fields. If you monitor the parser stats you should be able to see an entry for your Flow name. As each shared row variable reload takes place you should not see any significant growth in the "Fields" column then the problem is growth is unlikely to be in the Parser area.
_________________
Regards
Craig |
|
| Back to top |
|
|
| Craig B |
| Posted: Fri Nov 21, 2014 2:55 am Post subject: |
|
|
Partisan
Joined: 18 Jun 2003
Posts: 316
Location: UK
|
One additional point I forget to mention in my previous post is that MQSI_FREE_MASTER_PARSERS does NOT apply to parsers/fields in SHARED ROW variables. This environment variable only applies to parsers/fields that were specifically related to the processing of the last input message on one thread. SHARED ROW variables have wider scope that a message flow thread instance. This is why in 7.0.0.4 functionality was implemented so that DELETE FIELD could free parsers etc.
_________________
Regards
Craig |
|
| Back to top |
|
|
| mqsiuser |
| Posted: Fri Nov 21, 2014 4:09 am Post subject: |
|
|
Yatiri
Joined: 15 Apr 2008
Posts: 637
Location: Germany
|
Thank you very much, Craig!
| Craig B wrote: |
| As each shared row variable reload takes place you should not see any significant growth in the "Fields" column then the problem is growth is unlikely to be in the Parser area. |
We do see significant growth in the "Fields" column! The count was ~ 5million and growing at a rate of ~12tsd each minute.
It is roughly the amount that get's fetched from the DB (actually it is 2 times the number of rows, but that still fits, I guess). So every 10 seconds ~900 rows are fetched and "Fields" grows at a rate of ~2000 each 10 seconds.
As from what you write we could upgrade to 7.0.0.7 or explicitly use parsers (within the SHARED ROW) and manage their creation and destruction.
I prefer the second option. I know that I will find the solution on Google. If anyone (with more than zero experience with exactly this -like me-) would share some (lines of) code, that would be great (also for future readers of this thread  )
_________________
Just use REFERENCEs |
|
| Back to top |
|
|
| Craig B |
| Posted: Fri Nov 21, 2014 4:25 am Post subject: |
|
|
Partisan
Joined: 18 Jun 2003
Posts: 316
Location: UK
|
Before exploring other coding options I would be interested to find out why your Fields column is growing. Ie why the Fields are not be allocated for re-use. Would it be possible to provide the snippet of ESQL that populates the SHARED ROW variable and how it is cleared?
_________________
Regards
Craig |
|
| Back to top |
|
|
| mqsiuser |
| Posted: Fri Nov 21, 2014 4:40 am Post subject: |
|
|
Yatiri
Joined: 15 Apr 2008
Posts: 637
Location: Germany
|
Yes, good point. Here is the logic (that I came up with )
| Code: |
BROKER SCHEMA com.myCompany.mySchema
PATH <...>
DECLARE mySharedRow SHARED ROW;
DECLARE mySharedRow2 SHARED ROW;
<...>
CREATE PROCEDURE reloadCache( IN refEnvironment REFERENCE )
BEGIN
DELETE FIELD mySharedRow.Calls;
CACHING: BEGIN ATOMIC
SET mySharedRow.Calls.Call[] = ( SELECT <...> );
------ *Preprocess the result into mySharedRow2 ------
DELETE FIELD mySharedRow2.Calls;
<Loop over mySharedRow and put into mySharedRow2 with curly brace magic "{criteria}" (for grouping)>
END CACHING;
END; |
_________________
Just use REFERENCEs |
|
| Back to top |
|
|
| Craig B |
| Posted: Fri Nov 21, 2014 5:07 am Post subject: |
|
|
Partisan
Joined: 18 Jun 2003
Posts: 316
Location: UK
|
Thanks for the additional information. So it looks like you are using two SHARED ROW variables and mapping between them in some way. Can you give some examples of the kind of "preprocessing" and "staging" you do between the two shared row variables. Specifically I am interested in the kind of ESQL you use to copy values between them.
_________________
Regards
Craig |
|
| Back to top |
|
|
|
|
