| Author |
Message
|
| er_pankajgupta84 |
 Posted: Fri Dec 02, 2011 6:34 am Post subject: Message Memory in Compute Node - question |
 |
|
Master
Joined: 14 Nov 2008
Posts: 203
Location: charlotte,NC, USA
|
I have used java most of the time for coding in message broker. I have a question regarding "Compute" node.
I know PROPAGATE statement will clear any output message if used in compute node (Without any DELETE NONE option).
Suppose I have written in my compute node Main function:
1.
SET OutputRoot = InputRoot;
but never propagate anything then would will be a potential place for memory leak where the out message is not getting cleared.
2. for the above case if I add one more line as well:
SET OutputRoot = InputRoot;
PROPAGATE TO TERMINAL 'out2' DELETE NONE;
3. If the code is like this:
SET OutputRoot = InputRoot;
PROPAGATE TO TERMINAL 'out2' DELETE NONE;
RETURN FALSE;
4. If the code is like:
SET OutputRoot = InputRoot;
RETURN FALSE;
does any of the above code represents the memory leak situation.
Also,
If I have to send a message to two terminals then does the following code works:
SET OutputRoot = InputRoot;
PROPAGATE TO TERMINAL 'out2' DELETE NONE;
RETURN true;
or I need to add SET OutputRoot = InputRoot;
before the RETURN true statement -> if I have to then what's the use of DELETE NONE option in line 2.
I have joined at new place recently and have no working environment setup yet otherwise I could have done some POCs myself. I tried reading manuals but got confused.
If someone knows about it please respond. |
|
| Back to top |
|
 |
| mgk |
| Posted: Fri Dec 02, 2011 6:49 am Post subject: |
|
|
Padawan
Joined: 31 Jul 2003
Posts: 1642
|
Hello,
1: No - Code you write in ESQL should not leak - it is a heigher level language than Java and has no need of the clearMessage() functionality for example.
2, 3, 4: No. See 1:
Your last example (DELETE NONE) should also work fine, assuming the two terminals are Out and Out2.
Hope this helps.
Kind regards,
_________________
MGK
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Dec 02, 2011 6:49 am Post subject: Re: Message Memory in Compute Node - question |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| er_pankajgupta84 wrote: |
1.
SET OutputRoot = InputRoot;
but never propagate anything then would will be a potential place for memory leak where the out message is not getting cleared. |
If that's all you have then how do you get it to build? A Compute node is a function that returns a boolean. You must have a RETURN TRUE or a RETURN FALSE.
| er_pankajgupta84 wrote: |
2. for the above case if I add one more line as well:
SET OutputRoot = InputRoot;
PROPAGATE TO TERMINAL 'out2' DELETE NONE; |
See my comments above.
| er_pankajgupta84 wrote: |
3. If the code is like this:
SET OutputRoot = InputRoot;
PROPAGATE TO TERMINAL 'out2' DELETE NONE;
RETURN FALSE;
4. If the code is like:
SET OutputRoot = InputRoot;
RETURN FALSE;
does any of the above code represents the memory leak situation. |
Again, the Compute node is a function. Once you've returned it (and it's storage) goes out of scope.
| er_pankajgupta84 wrote: |
If I have to send a message to two terminals then does the following code works:
SET OutputRoot = InputRoot;
PROPAGATE TO TERMINAL 'out2' DELETE NONE;
RETURN true;
|
Yes. If one of the terminals you want to send to is Out.
| er_pankajgupta84 wrote: |
if I need to add SET OutputRoot = InputRoot;
before the RETURN true statement -> if I have to then what's the use of DELETE NONE option in line 2. |
Why would you think you need to add the line, if you've coded DELETE NONE?
| er_pankajgupta84 wrote: |
| I have joined at new place recently and have no working environment setup yet otherwise I could have done some POCs myself. I tried reading manuals but got confused. |
Confused how? This is fairly basic coding. Or do you have Java poisoning?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| er_pankajgupta84 |
| Posted: Fri Dec 02, 2011 7:30 am Post subject: This is an abstract from infocenter |
|
|
Master
Joined: 14 Nov 2008
Posts: 203
Location: charlotte,NC, USA
|
This is an abstract from infocenter
| Quote: |
An alternative method is to instruct the node not to clear the output message on the first two PROPAGATE statements, so that the message is available for routing to the next destination. The code to do this is:
SET OutputRoot = InputRoot; PROPAGATE DELETE NONE; SET OutputRoot = InputRoot; PROPAGATE TO TERMINAL ’Out1’ DELETE NONE; SET OutputRoot = InputRoot; PROPAGATE TO LABEL ’ThirdCopy’;
If you do not initialize the output buffer, an empty message is generated, and the message flow detects an error and throws an exception.
|
That's why I got confused. When we are using DELETE NONE then why I have to reset outputroot.
So compute node will never result in memory leaks (for out message) as soon as we leave the compute node output message space will be cleared (this is unlike java where we have to use clearMessage). |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Dec 02, 2011 7:58 am Post subject: Re: This is an abstract from infocenter |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| er_pankajgupta84 wrote: |
| That's why I got confused. When we are using DELETE NONE then why I have to reset outputroot. |
But if you use DELETE NONE the buffer is initialized.
| er_pankajgupta84 wrote: |
| So compute node will never result in memory leaks (for out message) as soon as we leave the compute node output message space will be cleared (this is unlike java where we have to use clearMessage). |
The Compute node doesn't leak for any item defined within it not just the output message and for the reason you give.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| er_pankajgupta84 |
| Posted: Fri Dec 02, 2011 8:11 am Post subject: |
|
|
Master
Joined: 14 Nov 2008
Posts: 203
Location: charlotte,NC, USA
|
Thanks victor.
That's how i reacted when I see the SET OutputRoot = InputRoot; after the PROPAGATE TO TERMINAL ’Out1’ DELETE.
There is no need for that.
I don't have a runtime environment otherwise I could have checked it by running a small POC. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Dec 02, 2011 8:20 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| er_pankajgupta84 wrote: |
| Thanks victor. |
So not just Java poisoning but C poisoning eh? 
| er_pankajgupta84 wrote: |
That's how i reacted when I see the SET OutputRoot = InputRoot; after the PROPAGATE TO TERMINAL ’Out1’ DELETE.
There is no need for that. |
There is if you code what you've posted. Not if you code DELETE NONE.
| er_pankajgupta84 wrote: |
| I don't have a runtime environment otherwise I could have checked it by running a small POC. |
Interested as to how. What's your preferred method of checking given than you can't use JVM monitoring tools on a Compute node?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Esa |
| Posted: Thu Feb 16, 2012 7:13 am Post subject: Re: Message Memory in Compute Node - question |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| er_pankajgupta84 wrote: |
does any of the above code represents the memory leak situation.
|
To my surprise I have just found out that PROPAGATE DELETE NONE seems to leak memory. Message Broker V 6.1.0.5. I did an APAR search and did not find anything related to this.
My aim was to speed up splitting a large file containing over 50 000 records into messages that contain only one record. The current implementation took over 15 minutes to run.
So I created a mutable InputRoot the way you do with large messages. Then I copied the headers to OutputRoot to avoid having to copy them for each record. I reused OutputRoot by creating NEXTSIBLING of the MRM message, moving the reference to it and deleting it's previous sibling. I propagated with DELETE NONE.
In the tests the flow started to spit out messages fery fast, about 18000 messages during the first minute, but then it continuously slowed down so that the last 10 000 messages took over 10 minutes and the overall processing time was 17 minutes.
Then I replaced MQOutput with a Passthrough node to test if the problem was with MQ. Same results, also it must be related to propagate.
Commented out propagate and the test run very fast.
Then I changed the code so that I propagated with DELETE DEFAULT and copied the headers for each output message. The test run in 50 seconds.
To me that looks like a memory leak... |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Feb 16, 2012 7:39 am Post subject: Re: Message Memory in Compute Node - question |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Esa wrote: |
| To me that looks like a memory leak... |
To me it looks like you should try it again on a more recent version of v6.1. Many fixes are not in APARs, and many fixes are not intended to fix an errant behaviour but are a happy accident.
Better still, try it again on the most recent version of v7. If it still happens, raise a PMR & get an APAR all your own.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Esa |
| Posted: Fri Feb 17, 2012 12:44 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
I have tested the scenario on 7.0.3 with the same results.
On a second thought, I don't think this is a memory leak. It's just that when you propagate with DELETE NONE, the execution group does not release all the resources it should. But it releases them when the thread terminates, which means it's not a leak. |
|
| Back to top |
|
|
| Esa |
| Posted: Fri Feb 17, 2012 3:54 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| No, the AIX server was running out of memory after my tests and the broker had to be restarted. Seems it's a memory leak after all. We will open a PMR. |
|
| Back to top |
|
|
| marko.pitkanen |
| Posted: Fri Feb 17, 2012 5:16 am Post subject: |
|
|
Chevalier
Joined: 23 Jul 2008
Posts: 440
Location: Jamsa, Finland
|
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Feb 17, 2012 5:48 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Esa wrote: |
| We will open a PMR. |
I for one await the outcome with interest.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Esa |
| Posted: Fri Feb 17, 2012 8:57 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
Hi Marko,
I hope you have had time to do some skiing. I haven't 
There may be a mutual cause, you never know. My case is clearly related to the way you call PROPAGATE. Alastair, in his post, did not mention anything about propagation.
My PRM, unfortunately, has to wait until Monday. Today I was busy trying to prove my other PMR unnecessary - with success, as far as I can tell. |
|
| Back to top |
|
|
| marko.pitkanen |
| Posted: Sun Feb 19, 2012 11:12 pm Post subject: |
|
|
Chevalier
Joined: 23 Jul 2008
Posts: 440
Location: Jamsa, Finland
|
Yes Esa,
I have done some cross country skiing. 
--
Marko |
|
| Back to top |
|
|
|
|
