| Author |
Message
|
| jonasb |
 Posted: Thu May 15, 2008 12:17 am Post subject: memory leak? |
 |
|
Apprentice
Joined: 20 Dec 2006
Posts: 49
Location: Sweden
|
Hi,
We're running broker 6.1.0.1 broker on AIX with the 6.1.0.103 Toolkit (win).
We have a serious problem with memory consumption on the server.
At startup, broker servers and execution groups takes up about 3GB.
For people not used to the broker this might seem high in itself, but
we are running with 5GB physical memory, so its ok.
After 2 months without stopping the server, it had grown to 13GB...
Needless to say, it was hard to update the execution groups with new flows.
(I waited 35 minutes for one execution group to deploy, without any response. We then tried to take down the server, which succeeded after about 25 minutes.)
We have about the same amount of load every day (little lower on week ends)
The memory usage seems to grow linear, which would indicate memory leaks?
My question is:
Can I do anything about this, except for restarting the server once or twice a week?
(trying to uphold 24/7 availablity, we do not want to have to restart servers too often)
I mean: is there anything special we should avoid doing in developing the flows, which is memory leak prone?
Any and all suggestions are welcome!
_________________
contact admin |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu May 15, 2008 12:25 am Post subject: Re: memory leak? |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| contact admin wrote: |
| Can I do anything about this, except for restarting the server once or twice a week? |
Raise a PMR. While the odds of any IBM code containing any kind of error are enormous, it's a possibility which can't be ruled out without investigation.
As a minimum, a PMR will enable a more detailed look at the broker's resource usage and could help isolate a problem in the flows or elsewhere.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| jonasb |
| Posted: Thu May 15, 2008 12:53 am Post subject: |
|
|
Apprentice
Joined: 20 Dec 2006
Posts: 49
Location: Sweden
|
>While the odds of any IBM code containing any kind of error are enormous/.../
very funny :-)
I will raise a PMR, I just wanted to see what the community thoughts were.
_________________
contact admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu May 15, 2008 2:16 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| contact admin wrote: |
>While the odds of any IBM code containing any kind of error are enormous/.../
very funny 
|
| contact admin wrote: |
I will raise a PMR, I just wanted to see what the community thoughts were. |
Good thinking. There's a sporting chance another community member will have a more positive idea than mine. I hope someone has, since this will speed resolution of your problem. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqmatt |
| Posted: Thu May 15, 2008 4:08 am Post subject: |
|
|
Grand Master
Joined: 04 Aug 2004
Posts: 1213
Location: Hursley, UK
|
Agreed - you should raise a PMR.
On the off-chance, if you're using JavaCompute nodes make sure you are calling clearMessage() after you've finished with each message. |
|
| Back to top |
|
|
| Gaya3 |
| Posted: Thu May 15, 2008 4:13 am Post subject: |
|
|
Jedi
Joined: 12 Sep 2006
Posts: 2493
Location: Boston, US
|
 and good point too
| mqmatt wrote: |
Agreed - you should raise a PMR.
On the off-chance, if you're using JavaCompute nodes make sure you are calling clearMessage() after you've finished with each message. |
indeed JCN is showing some problems on V6. Some times it wont get deleted from the broker.
Regards
Gayathri
_________________
Regards
Gayathri
-----------------------------------------------
Do Something Before you Die |
|
| Back to top |
|
|
| Esa |
| Posted: Thu May 22, 2008 10:29 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
Hej contact admin,
you propably know that execution groups do not free any memomy they have allocated?
I have never seen any official explanation of this, but I assume it has to do with performance optimization: eg's not wanting to end up allocating and freeing memory like jo-jos, perhaps?
Anyway, there is a command called mqsireload that you can use to free all memory allocated by an execution group. Many users have this command scheduled for each execution group every morning.
Esa |
|
| Back to top |
|
|
| ADV |
| Posted: Thu May 22, 2008 11:20 am Post subject: |
|
|
Apprentice
Joined: 24 Apr 2007
Posts: 44
Location: Boston, MA
|
MQSIRELOAD is nothing but bouncing the execution group. Means, it stops all the flows running under eg, before it goes down.
MORE INFO from IBM info center......
Use the mqsireload command to request the broker to stop and restart execution groups.
When you issue this command, a message is sent to the broker, which stops and restarts all its execution groups. You can specify a single execution group to be reloaded, but you are recommended to use the default form of this command to reload all execution groups.
Because an execution group does not stop until all message flows within it terminates, the ability of the broker to reload quickly depends on the processing time for the longest running message flow. This affects the performance of this command, and you are recommended to review any long-running message flows.
If you have included a user-defined node or parser within a message flow on the broker, these are deleted by this command, and the relevant termination functions called. When message flows are restarted, the resources used by user-defined nodes and parsers are re-accessed and reacquired. However, you are recommended to ensure that user-defined nodes and parsers provide their own mechanism to reload persistent state and data dynamically, and do not rely on the use of this command. |
|
| Back to top |
|
|
| jonasb |
| Posted: Tue May 27, 2008 4:59 am Post subject: |
|
|
Apprentice
Joined: 20 Dec 2006
Posts: 49
Location: Sweden
|
Hi Esa & ADV,
Thanks for the input, much appreciated!
I have noticed that memory is never returned. Still, if there were no memory leaks, the excution groups would stop growing eventually (unless bigger and bigger messages would arrive all the time, which is not the case).
It was comforting to here that others restart their execution groups on a regular basis :-)
We might consider doing this in the future.
_________________
contact admin |
|
| Back to top |
|
|
| bobbee |
| Posted: Tue Jul 08, 2008 10:47 am Post subject: |
|
|
Knight
Joined: 20 Sep 2001
Posts: 545
Location: Tampa
|
I have tried to use Profiling and Loggin perspective in the Toolkit. It seems to have some suggestion of memeory monitoring. I updated the RAC on the Broker v6.0 install to RAC 7.0.1 (Windows) . I see the agent directory and the serviceconfig.xml now mentions the agents and the other elements discussed. My issue is the stuff seems to be only supported on Windows and Linux (x86).
I have two questions. Has ANYBODY used Profiling and Logging....in UNIX (AIX)....in Windows? Does it have benefits.
What else can one use on AIX to debug possible memory issues with Broker.
There are EG's that grow to about 1/2 gig each. When the flows are not processing, for hours, the memory footprint does not go down. |
|
| Back to top |
|
|
|
|
