| Author |
Message
|
| francoisvdm |
 Posted: Fri Oct 24, 2008 1:10 am Post subject: MQGet Node eats memory |
 |
|
Partisan
Joined: 09 Aug 2001
Posts: 332
|
MB V6.0.3 on Windows, MQ V6.0.2.2
I have the following code in a compute node directly before the MQGet node:
| Code: |
SET Environment.Loop = 1;
WHILE Environment.Loop = 1 DO
PROPAGATE TO TERMINAL 'out';
END WHILE;
RETURN FALSE; |
The terminal "out" goes to the MQGet node.
So, in effect there is a LOOP created by this.
When I play this against a queue with a few thousand messages in I see that more memory is used and never released (expected behavior to not release memory).
so, it is as if the broker is building some kind of "chain".
Transaction mode is set to "no".
Any ideas? Thank you
_________________
If you do not know the answer or you get the urge to answer with "RTFM" or "Search better in this forum", please refrain from doing so, just move on to the next question. Much appreciated.
Francois van der Merwe |
|
| Back to top |
|
 |
| Gaya3 |
| Posted: Fri Oct 24, 2008 1:17 am Post subject: Re: MQGet Node eats memory |
|
|
Jedi
Joined: 12 Sep 2006
Posts: 2493
Location: Boston, US
|
| francoisvdm wrote: |
| Code: |
SET Environment.Loop = 1;
WHILE Environment.Loop = 1 DO
PROPAGATE TO TERMINAL 'out';
END WHILE;
RETURN FALSE; |
|
Does the while loop looks like an infinite loop? 
_________________
Regards
Gayathri
-----------------------------------------------
Do Something Before you Die |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Oct 24, 2008 1:20 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Search the forum, there was something about having an MQGET node in a loop and memory consumption... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Gaya3 |
| Posted: Fri Oct 24, 2008 1:22 am Post subject: |
|
|
Jedi
Joined: 12 Sep 2006
Posts: 2493
Location: Boston, US
|
i suggest you dont use environment variables as its a raw data types for loop counters and all,
Use proper data types for the same
_________________
Regards
Gayathri
-----------------------------------------------
Do Something Before you Die |
|
| Back to top |
|
|
| francoisvdm |
| Posted: Fri Oct 24, 2008 1:33 am Post subject: |
|
|
Partisan
Joined: 09 Aug 2001
Posts: 332
|
No, it is not an infinite loop, that var gets set in compute node after MQGet when no more messages, and it does stop the flow.
By "proper data types" do you mean a "shared" variable declared as "int"?
Thank you, and I'll try another search.
_________________
If you do not know the answer or you get the urge to answer with "RTFM" or "Search better in this forum", please refrain from doing so, just move on to the next question. Much appreciated.
Francois van der Merwe |
|
| Back to top |
|
|
| mgk |
| Posted: Fri Oct 24, 2008 3:33 am Post subject: |
|
|
Padawan
Joined: 31 Jul 2003
Posts: 1642
|
| Quote: |
| i suggest you dont use environment variables as its a raw data types for loop counters and all, |
Why ever not? The code posted:
| Code: |
| SET Environment.Loop = 1; |
will simple create a syntax element called Loop and assign its value to be the Integer 1. This will then be no different from an Integer declared as:
DECLARE Loop INTEGER 1;
The only time that I can see this causing a problem is if someone accidently used '1' instead of 1 as this will give Loop a character value rather than an integer value. And with the code posted this does not happen.
Also using a normal variable (SHARED or not) will make no differerence to the problem raised in this thread
Regards,
_________________
MGK
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. |
|
| Back to top |
|
|
| Gaya3 |
| Posted: Fri Oct 24, 2008 3:48 am Post subject: |
|
|
Jedi
Joined: 12 Sep 2006
Posts: 2493
Location: Boston, US
|
| mgk wrote: |
The only time that I can see this causing a problem is if someone accidently used '1' instead of 1 as this will give Loop a character value rather than an integer value.
|
I am cared about this part, thats the reason, i mentioned dont use Environment variables too much.
| mgk wrote: |
Also using a normal variable (SHARED or not) will make no differerence to the problem raised in this thread
|
I agree.
_________________
Regards
Gayathri
-----------------------------------------------
Do Something Before you Die |
|
| Back to top |
|
|
| Esa |
| Posted: Sun Oct 26, 2008 1:52 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
Dear Francois,
the cause of increasing memory consumption is not the MQGET node but your application architecture. If you read thousands of messages from a queue within one single message flow instance, a memory buffer is allocated for each message. The allocated memory is released only when the flow thread terminates.
You should consider using the good old boring MQInput nodes instead. That way each message would get processed in its own thread and memory consumption would not grow. If you need to maintain state, as you obviously do, you can use ESQL SHARED variables, a database or status messages you put in another queue and get with MQGet. |
|
| Back to top |
|
|
| francoisvdm |
| Posted: Sun Oct 26, 2008 8:36 pm Post subject: |
|
|
Partisan
Joined: 09 Aug 2001
Posts: 332
|
| mgk wrote: |
| Quote: |
| i suggest you dont use environment variables as its a raw data types for loop counters and all, |
Why ever not? The code posted:
| Code: |
| SET Environment.Loop = 1; |
will simple create a syntax element called Loop and assign its value to be the Integer 1. This will then be no different from an Integer declared as:
DECLARE Loop INTEGER 1;
The only time that I can see this causing a problem is if someone accidently used '1' instead of 1 as this will give Loop a character value rather than an integer value. And with the code posted this does not happen.
Also using a normal variable (SHARED or not) will make no differerence to the problem raised in this thread
Regards, |
I thought
| Code: |
| DECLARE Loop INTEGER 1; |
will only be visible to code in that bit of ESQL, and I need to assign a different variable later on in the flow to that variable in order to end the loop. Any comments?[/code]
_________________
If you do not know the answer or you get the urge to answer with "RTFM" or "Search better in this forum", please refrain from doing so, just move on to the next question. Much appreciated.
Francois van der Merwe |
|
| Back to top |
|
|
| francoisvdm |
| Posted: Sun Oct 26, 2008 8:39 pm Post subject: |
|
|
Partisan
Joined: 09 Aug 2001
Posts: 332
|
| Esa wrote: |
Dear Francois,
the cause of increasing memory consumption is not the MQGET node but your application architecture. If you read thousands of messages from a queue within one single message flow instance, a memory buffer is allocated for each message. The allocated memory is released only when the flow thread terminates.
You should consider using the good old boring MQInput nodes instead. That way each message would get processed in its own thread and memory consumption would not grow. If you need to maintain state, as you obviously do, you can use ESQL SHARED variables, a database or status messages you put in another queue and get with MQGet. |
Ahh, thanks Esa, now this makes more sense to me on why it is working like this. I thought it would return the memory when it goes back past the MQGet node, and that every "propagate" statement will start a new thread. Clearly that is not the way it works. And I thought this was a very elegant way of getting done what I tried. Ag ja, next version... 
Thank for the help, much appreciated.
_________________
If you do not know the answer or you get the urge to answer with "RTFM" or "Search better in this forum", please refrain from doing so, just move on to the next question. Much appreciated.
Francois van der Merwe |
|
| Back to top |
|
|
|
|
