| Author |
Message
|
| shalabh1976 |
 Posted: Mon Feb 02, 2009 5:14 am Post subject: Using MQGet to empty the queue - STACK OVERFLOW |
 |
|
Partisan
Joined: 18 Jul 2002
Posts: 381
Location: Gurgaon, India
|
Hi,
I am using the MQGET node to empty a queue because that is a requirement for one of the services that we are developing.
The flow works fine if the source queue has a depth of 4000-5000 messages. However if the queue depth goes upto 9000-10000, we get a Broker abend with a Stack overflow. I have changed the MQGet properties to set transaction mode to No, Generate mode to None, Parse timing to On demand.
Other than increasing the MQSI_THREAD_STACK_SIZE, which has hardware limitations what other methods do I have to make this work using WMB?
_________________
Shalabh
IBM Cert. WMB V6.0
IBM Cert. MQ V5.3 App. Prog.
IBM Cert. DB2 9 DB Associate |
|
| Back to top |
|
 |
| Vitor |
| Posted: Mon Feb 02, 2009 5:24 am Post subject: Re: Using MQGet to empty the queue - STACK OVERFLOW |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| shalabh1976 wrote: |
| I am using the MQGET node to empty a queue because that is a requirement for one of the services that we are developing. |
I question the requirement. The MQGet is designed to read a single message, typically a response. Why use it to empty a queue in preference to the MQInput node which is a more obvious choice for this task?
| shalabh1976 wrote: |
| Other than increasing the MQSI_THREAD_STACK_SIZE, which has hardware limitations what other methods do I have to make this work using WMB? |
Clearly a question for a passing IBMer with access to the source code...
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| shalabh1976 |
| Posted: Mon Feb 02, 2009 5:45 am Post subject: |
|
|
Partisan
Joined: 18 Jul 2002
Posts: 381
Location: Gurgaon, India
|
MQInput cannot be used in the middle of a flow. It has no input terminal.
Also MQInput does not allow me to set the name of the Source Queue dynamically.
My version is 6.1.0.3 on Windows.
_________________
Shalabh
IBM Cert. WMB V6.0
IBM Cert. MQ V5.3 App. Prog.
IBM Cert. DB2 9 DB Associate |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Feb 02, 2009 5:52 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| shalabh1976 wrote: |
| MQInput cannot be used in the middle of a flow. It has no input terminal. |
But why, in the middle of a flow, do you need to read a queue until it's empty? As opposed to this functionallity being a separate flow?
| shalabh1976 wrote: |
Also MQInput does not allow me to set the name of the Source Queue dynamically. |
Well not at run time certainly. Again, how is it you're halfway down a flow before you know what queue you're reading? There has to be another design solution to that.
Taking all that to one side, the broker certainly shouldn't just blow. You're entitled to a PMR on that if all else fails.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Feb 02, 2009 6:44 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Apart from the fact that this is something that broker is not really designed for, I would suggest one of the following.
1)
Start Flow
Delete N Messages in a loop using MQget
Output a control Message to a Queue.
End Flow
In the original flow have an alternate MQInput Node that reads messages from the Queue that the message was sent to above.
The process starts again until N messages are read or the MQGet returns a 2033.
'N' being a counter with a reasonable small value, say 500.
This keeps the resources that broker uses down to a reasonable limit.
2)
Use PCF to clear the flow. However this may not be possible if an application has the Queue open for output.
3) Write a simple C or Java app to do the job outside of Broker. Start it off by sending it a message with the name of the queue you want emptied.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Feb 02, 2009 6:50 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| smdavies99 wrote: |
| Apart from the fact that this is something that broker is not really designed for, I would suggest one of the following. |
I took "clear the queue" to mean "read and process the messages until empty", with the MQGet being used for some reason - apparently to set the queue name at run time.
If all you want to do is flush the queue contents, I agree a flow is a bad way of doing it.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Feb 02, 2009 8:13 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Vitor
If the purpose is just to clear the queue then really broker is not the tool to do it. My option 1 will work for the case where something has to be done with the messages. By keeping the loop count down to a sensible number then you wont have to do it all in one UOW. I can't think of any application where you have to process thousands of messages in a single UOW.
Personally, I am slightly saddened by the number of posts in this forum where the poster is plainly trying to use broker for something it is not really designed for. I suppose the number of Java related queries (when plainly it can be done simply in ESQL) may also have something to do with it. Perhaps some of these apps should be better off done inside WAS rather than Broker. Just my of 2p worth.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Feb 02, 2009 8:21 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
I stand by my original question - why MQGet instead of MQInput?
The question "why broker " is increasing valid though.
I suspect the answer to "why broker" will be "because that's the requirement". Effectively "we're using broker because we're using broker". Circular management logic.
Should have laid off the coffee earlier..... 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mgk |
| Posted: Mon Feb 02, 2009 8:47 am Post subject: |
|
|
Padawan
Joined: 31 Jul 2003
Posts: 1642
|
Whilst I agree with the others above, do you have a loop in your message flow?
Regards,
_________________
MGK
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Feb 02, 2009 1:35 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| smdavies99 wrote: |
| Personally, I am slightly saddened by the number of posts in this forum where the poster is plainly trying to use broker for something it is not really designed for. I suppose the number of Java related queries (when plainly it can be done simply in ESQL) may also have something to do with it. Perhaps some of these apps should be better off done inside WAS rather than Broker. Just my of 2p worth. |
 Java just because you can do anything in java is not appropriate for the broker. Use a Java compute node to transform and manipulate the message like you would in an ESQL node, maybe with some more complex logic and mainly because of training and skill set. Do not use Java for the sake of Java... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| shalabh1976 |
| Posted: Mon Feb 02, 2009 10:32 pm Post subject: |
|
|
Partisan
Joined: 18 Jul 2002
Posts: 381
Location: Gurgaon, India
|
To answer mgk's query,
Yes there is a loop, that is how the queue is being emptied.
The out terminal is wired to the in terminal. Flow continues from no message found terminal.
As mentioned in my original mail, flow is fine if the source queue has 4000-5000 messages. Problem appears if there are 9000-10000 messages.
Based on the request different queues have to be cleared. By dynamically setting the queue name mqget works fine as explained above.
MQInput cannot be used in the above scanario.
My query is regarding the STACK OVERFLOW and how to avoid it.
_________________
Shalabh
IBM Cert. WMB V6.0
IBM Cert. MQ V5.3 App. Prog.
IBM Cert. DB2 9 DB Associate |
|
| Back to top |
|
|
| paranoid221 |
| Posted: Mon Feb 02, 2009 10:42 pm Post subject: |
|
|
Centurion
Joined: 03 Apr 2006
Posts: 101
Location: USA
|
Agree with the other peers here. Broker a bad choice for your requirement.
You could perhaps leverage some sort of triggering mechanism to kick-off an external Java class that can empty your queue. You can atleast keep broker out of it.
It is worth mentioning here that by designing a solution for this via broker, you are also jeopardizing the performance of other more critical flows that may deployed on the same broker. Your Stack OverFlow issue is a testimony to that which is not going to stand the test of time before you will eventually see serious resource starvation issues on the broker server.
_________________
LIFE is a series of complex calculations, somewhere multiplied by ZERO. |
|
| Back to top |
|
|
| elvis_gn |
| Posted: Mon Feb 02, 2009 10:54 pm Post subject: |
|
|
Padawan
Joined: 08 Oct 2004
Posts: 1905
Location: Dubai
|
Hi shalabh1976,
Java has this bad behavior of increasing in memory consumption even when you are not doing anything.
For example, when you call a Db query through MB, the flow is actually not doing anything, but if the result takes too long, you would find the memory shoot...
I would suppose the same is happening in your case too, 4-5K is handled but 9-10K just gets too much.
And as above, I would say this is not a good design....why didn't you think of PCF, or triggering 5000th msg on queue, or even expiry.
If you continue to go this way, you might increase the memory for the execution group, but you'll find things failing on 15-20K...where will you draw the line...
Regards. |
|
| Back to top |
|
|
| mgk |
| Posted: Tue Feb 03, 2009 1:32 am Post subject: |
|
|
Padawan
Joined: 31 Jul 2003
Posts: 1642
|
| Quote: |
| Yes there is a loop, that is how the queue is being emptied. |
This is the cause of your problem. A loop in the flow will always cause a problem if you loop enough times. If you must do this, then change the flow to NOT have a loop, using ESQL PROPAGATE from a compute node.
Regards,
_________________
MGK
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. |
|
| Back to top |
|
|
| Esa |
| Posted: Tue Feb 03, 2009 1:41 pm Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
Thanks, mgk. This has been bothering me for some months but I never had time to run a test if my theories were correct. It is nice to hear this from Hursley, now I do not have to test it myself.
With "a loop in the flow" you mean a visual loop where you connect a nodes output terminal to its own input terminal either directly or indirecly. This is a bad thing because when the flow execution moves from a node to another, the previous node actually calls the next node. And the next node calls some other nodet etc. This means that any resources (memory) allocated by the second node are not released until the call chain returns to the first node.
If you have an MQGet node that calls itself, memory allocated for the messages will accumulate and will not be released until all messages are read and the whole call chain returns.
But if you loop in a compute node or a java compute node and call the MQGet node from within that loop by propagating some message, the call will return back to the compute node after each message read from the MQGet queue. And, as you can read from between the lines of mgks post, the memory allocated for each message is released immediately after it has been handled.
Well, actually the memory is not released to the operating system but some MB internal memory buffer where it is - hopefully - immedially availabe for other message flow threads. What I was not sure about was if any memory allocated by nodes could be released while the thread is still running or only after it has terminated. |
|
| Back to top |
|
|
|
|
