| Author |
Message
|
| rk9999 |
 Posted: Thu Mar 04, 2010 3:13 pm Post subject: Clearing Shared Variables from memory |
 |
|
Newbie
Joined: 04 Mar 2010
Posts: 2
|
Hi,
I am trying to save a process a file with multiple records and i read one record at a time rather then the whole file.
As my first record goes thru i save it in a SHARED variable and then return to the file get the next one. If the ID of the second message is the same as the previous one(saved in the shared variable) i append it to that and get the next record and if that ID differs from the previous message i propage the previous message to the Q.
I use a few shared variables to save some values at flow level because the Environment tree is wiped out after processing one record.
This is using up all the memory on the server/local machine and even after the file and the flow is done it does not release the memory.
Is there a way to clear the memory after finishing the file?
Do I have any alternate methods to do what I am doing?
Thanks |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu Mar 04, 2010 4:43 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Moved to more relevent section
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Mar 04, 2010 4:52 pm Post subject: Re: Clearing Shared Variables from memory |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| rk9999 wrote: |
| I am trying to save a process a file with multiple records and i read one record at a time rather then the whole file. |
You have 2 issues here - you're using a file rather than messages, and the records in the file have affinity; i.e. the processing of one record depends on the previous record.
| rk9999 wrote: |
| As my first record goes thru i save it in a SHARED variable and then return to the file get the next one. If the ID of the second message is the same as the previous one(saved in the shared variable) i append it to that and get the next record and if that ID differs from the previous message i propage the previous message to the Q |
I'm assuming for the rest of the post that by ID you mean some business value in the file record. I'm also assuming that all of the records with the same ID occur sequentially in the file (or your current solution is toast).
| rk9999 wrote: |
Do I have any alternate methods to do what I am doing?
|
Well the simple solution is to split the file by ID. As I indicate above, you must be sorting the file by ID so spliting it is a fairly simple task.
Failing that you could use a Collector node to group up the various IDs and send them forward.
Other design solutions, possibly better ones, are undoubably possible.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| rk9999 |
| Posted: Fri Mar 05, 2010 9:36 am Post subject: Re: Clearing Shared Variables from memory |
|
|
Newbie
Joined: 04 Mar 2010
Posts: 2
|
| Vitor wrote: |
| rk9999 wrote: |
| I am trying to save a process a file with multiple records and i read one record at a time rather then the whole file. |
You have 2 issues here - you're using a file rather than messages, and the records in the file have affinity; i.e. the processing of one record depends on the previous record.
| rk9999 wrote: |
| As my first record goes thru i save it in a SHARED variable and then return to the file get the next one. If the ID of the second message is the same as the previous one(saved in the shared variable) i append it to that and get the next record and if that ID differs from the previous message i propage the previous message to the Q |
I'm assuming for the rest of the post that by ID you mean some business value in the file record. I'm also assuming that all of the records with the same ID occur sequentially in the file (or your current solution is toast).
| rk9999 wrote: |
Do I have any alternate methods to do what I am doing?
|
Well the simple solution is to split the file by ID. As I indicate above, you must be sorting the file by ID so spliting it is a fairly simple task.
Failing that you could use a Collector node to group up the various IDs and send them forward.
Other design solutions, possibly better ones, are undoubably possible. |
Thanks for the quick reply ... yes we are looking into a different solution as well.
So there no way to clear the SHARED variables in the memory even after the flow is done processing a message? or even the file? |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Mar 05, 2010 9:48 am Post subject: Re: Clearing Shared Variables from memory |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| rk9999 wrote: |
| So there no way to clear the SHARED variables in the memory even after the flow is done processing a message? or even the file? |
Not that I'm aware of. I'll leave it to better qualified people than me to discuss the details of WMB memory recovery, and also suggest other alternatives to you.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| joebuckeye |
| Posted: Fri Mar 05, 2010 11:18 am Post subject: |
|
|
Partisan
Joined: 24 Aug 2007
Posts: 365
Location: Columbus, OH
|
| Quote: |
| This is using up all the memory on the server/local machine and even after the file and the flow is done it does not release the memory. |
We are running WMB 6.0 (going to 7 later this year) on AIX and have been told by IBM that once an execution group (EG) grabs memory it does not release it back to the OS. Basically that the EG saves that memory because it needed that much in the past and might need it in the future so it holds onto it.
The only way to get that memory back is to restart the EG. This would only be feasible if this was a special situation (large data load?) and not a daily occurrence. |
|
| Back to top |
|
|
| Mandeep |
| Posted: Fri Mar 26, 2010 11:46 pm Post subject: |
|
|
Apprentice
Joined: 03 May 2004
Posts: 33
|
We have faced this problem with Shared Variables:
If you try to change the shared Variable value (once populated/initialized by the first instance of the flow) by every message being processed - which you are doing here - it eats memeory exponentially. And then EG doesn't releases it unless restarted.
So check and change the design in case:
You are trying to change the shared variable value every time you process a message?
Once initialized you should change its value very rarely. Good design would be to check if its popualted you should not re-assign it in the same flow, unless the condition becomes true very rarely like change of the date/year etc.. |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Sat Mar 27, 2010 2:31 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
| Mandeep wrote: |
| it eats memeory exponentially. |
This sounds very dubious and if true worthy of futher investigation / a PMR. How did you prove this? Did you ask IBM for advice? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Mar 27, 2010 8:42 am Post subject: Re: Clearing Shared Variables from memory |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Vitor wrote: |
| rk9999 wrote: |
| So there no way to clear the SHARED variables in the memory even after the flow is done processing a message? or even the file? |
Not that I'm aware of. I'll leave it to better qualified people than me to discuss the details of WMB memory recovery, and also suggest other alternatives to you. |
@rk9999 You are using the shared memory for the wrong purpose...
What you need to do is determine whether or not the record you receive is the last record and create your output. (Detach from shared and attach to output tree). Not sure whether this will release any memory...
Anyways as stated above there are other and better solutions available like splitting the file, using a collector node etc...
You could potentially use this for automatic splitting of the file.
Use file input and file output nodes and change the name when the ID changes.
All you have in shared memory is then the ID and that should not be such a burden...
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Mandeep |
| Posted: Mon Mar 29, 2010 1:32 am Post subject: |
|
|
Apprentice
Joined: 03 May 2004
Posts: 33
|
A simple test would be:
Check memory first time when Shared Variable is initialized i.e. at the first hit, you would see it taking considerable memory. Similarily, if you are re assigning a value to it every time, it would add/consume that amount of memeory each time.
Yes we worked with IBM for this. We were facing issues with Shared Variable value not visible in the nested subflow ( we had 3 levels of subflows). To overcome this the developer was creating a new Shared Variable with same name in the 3rd level subflow and assigning it the value every time, which was a mistake.
So a check if a shared variable is populated, then do not re-assign it, worked fine for us. Unless the condition (at which value should be refreshed) becomes true - which should not be very often |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Mon Mar 29, 2010 1:37 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
| Mandeep wrote: |
Yes we worked with IBM for this. |
So did they confirm it as an issue? Working as designed? Fix pending? Or was your "work around" of not updating it too often where it all ended? |
|
| Back to top |
|
|
| Mandeep |
| Posted: Mon Mar 29, 2010 3:52 am Post subject: |
|
|
Apprentice
Joined: 03 May 2004
Posts: 33
|
Hi, yes they confirmed there is no ready made solution available for this probelm at this point in time.
We were suggested to remove the 3rd level of subflow and move the subflow code one level up, in a compute node may be.
We instead declared the same Shared Variable at level 3, as this subflow was being reused in other flows as well.
We have informed the onsite team about the work around, who is dealing with IBM.
Would you suggest as an independent (without involving the client who is busy with Go Live at this stage) I can raise it with IBM? If yes, then how? |
|
| Back to top |
|
|
|
|
