ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker SupportStack space exceeded while using MQGET node

Post new topicReply to topic
Stack space exceeded while using MQGET node View previous topic :: View next topic
Author Message
madrox
PostPosted: Sun Jul 07, 2019 2:47 pm Post subject: Stack space exceeded while using MQGET node Reply with quote

Acolyte

Joined: 11 Mar 2015
Posts: 61

I know this topic has been extensively discussed here, but unfortunately for me even after reading multiple of these post i am unable to get this working.
Here is my scenario.
I have about 10000+ messages on a queue which i need to get after other processes i run during a flow. I need to read the messages from the queue and make a WS call, depending on success or failure i write status to a file.
Here is what i have implemented:
after my other process is complete, i call a subflow. The subflow contains
inputnode>MQGET>compute(buildURL)>WSRequest>Compute(to write to file and loop back to MQGET

The above implementation works fine while i test my code with about 10-15message. But when i load the queue up with 10000 messages, it processes about 30 messages and i see the following in the logs
Quote:
BIP2187E: The stack space limit('2097152') was exceeded during flow processing

I am pretty positive its the loop while getting the messages of the queue. Is there a better way to achieve this?
Back to top
View user's profile Send private message
bruce2359
PostPosted: Sun Jul 07, 2019 3:16 pm Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8418
Location: US: west coast, almost. Otherwise, enroute.

Moved to Broker forum.
_________________
There are two types of people in this world:
1) Those that can extrapolate from incomplete data
Back to top
View user's profile Send private message
abhi_thri
PostPosted: Mon Jul 08, 2019 12:36 am Post subject: Reply with quote

Master

Joined: 17 Jul 2017
Posts: 276
Location: UK

hi...yes, it is the wired loop within the flow that is causing the stack overflow error as you suspected and it is generally discouraged to use any sort of wired loop within a flow due to this reason as it will exhaust the memory stack (as the flow need to store the current flow state before starting a new loop) depending on the no. of the loops.

As these 10000+ messages are independent ones you need to process it independently by using separate flow transactions. i.e once the first message is processed successfully instead of doing a hard wired loop back to the MQGet just emit a dummy message to 'inputnode' so that it will start a new transaction and process the next message.

To do that you could define the current subflow as a separate flow which starts with an MQInput node which can then receive first trigger to start processing the messages via the MQGet node and the subsequent dummy triggers.
Back to top
View user's profile Send private message
timber
PostPosted: Mon Jul 08, 2019 2:16 am Post subject: Reply with quote

Grand Master

Joined: 25 Aug 2015
Posts: 1002

The suggestion from abhi_thri is the ideal because it splits the processing into separate, short-running transactions.

If you absolutely *need* to process all of the messages in single message flow execution then you can also do this:
- Write a loop in a Compute node with a PROPAGATE TO TERMINAL 'Out1' (or some other free terminal) statement.
- On terminal Out1, attach the message flow nodes to read exactly one message from the queue and process it. But do not wire any loop into the message flow.

This will result in a message flow which takes a very long time to complete, but it will not blow up the stack.
Back to top
View user's profile Send private message
madrox
PostPosted: Mon Jul 08, 2019 7:04 am Post subject: Reply with quote

Acolyte

Joined: 11 Mar 2015
Posts: 61

abhi_thri wrote:
hi...yes, it is the wired loop within the flow that is causing the stack overflow error as you suspected and it is generally discouraged to use any sort of wired loop within a flow due to this reason as it will exhaust the memory stack (as the flow need to store the current flow state before starting a new loop) depending on the no. of the loops.

As these 10000+ messages are independent ones you need to process it independently by using separate flow transactions. i.e once the first message is processed successfully instead of doing a hard wired loop back to the MQGet just emit a dummy message to 'inputnode' so that it will start a new transaction and process the next message.

To do that you could define the current subflow as a separate flow which starts with an MQInput node which can then receive first trigger to start processing the messages via the MQGet node and the subsequent dummy triggers.


Thank you abhi_thri for your response. I agree with the part that this should be a part of another flow, but since i need to process all the message after the processes of the main flow itself that would not work.
Could you elaborate on this, i did not follow what you meant. How would I trigger this flow if it has a MQinput node?
Quote:
To do that you could define the current subflow as a separate flow which starts with an MQInput node which can then receive first trigger to start processing the messages via the MQGet node and the subsequent dummy triggers.
Back to top
View user's profile Send private message
madrox
PostPosted: Mon Jul 08, 2019 7:30 am Post subject: Reply with quote

Acolyte

Joined: 11 Mar 2015
Posts: 61

timber wrote:
The suggestion from abhi_thri is the ideal because it splits the processing into separate, short-running transactions.

If you absolutely *need* to process all of the messages in single message flow execution then you can also do this:
- Write a loop in a Compute node with a PROPAGATE TO TERMINAL 'Out1' (or some other free terminal) statement.
- On terminal Out1, attach the message flow nodes to read exactly one message from the queue and process it. But do not wire any loop into the message flow.

This will result in a message flow which takes a very long time to complete, but it will not blow up the stack.


Thanks Timber,
If i wrote a loop in a compute node, i would still need to propagate to MQGet terminal, would i not?
Right now here is what i am doing, after my other processes complete, I come to the subflow with the MQGet. So there is the
Quote:
inputTerminal>MQGet>Compute>WSReq>Compute>FileOutput

If i wrote a loop in the second compute i would still need to wire back to the MQGet In terminal
Back to top
View user's profile Send private message
Vitor
PostPosted: Mon Jul 08, 2019 7:36 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25732
Location: Ohio, USA

madrox wrote:
If i wrote a loop in a compute node, i would still need to propagate to MQGet terminal, would i not?


Yes, but you'd be propogating 10,000 separate records in 10,000 threads to 10,000 copies of the MQGet node.

This is very different in terms of memory management to looping through a single copy of the MQGet node.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
timber
PostPosted: Mon Jul 08, 2019 1:18 pm Post subject: Reply with quote

Grand Master

Joined: 25 Aug 2015
Posts: 1002

I think you are misunderstanding this pattern.
The ESQL loop goes into the first Compute node. It repeatedly PROPAGATEs to a terminal until it is told to stop (more on that later).

The nodes attached to the terminal process exactly one message. There is no wiring path from this 'leg' of the message flow back to the first Compute node. This leg just ends when the processing of one message is done, and then the stack unwinds back to the Compute node loop. The maximum stack depth is therefore (approximately) 1/10000th of what it would be in your current solution.

So how does the loop terminate? Well, one way is to check a value in the Environment tree every time around the loop. When MQGet returns with a 'no message found' error, set the Environment field and terminate the loop. And...make sure that you don't use LocalEnvironment (it will not get propagated back into the first Compute node when the stack unwinds).
Back to top
View user's profile Send private message
abhi_thri
PostPosted: Tue Jul 09, 2019 12:19 am Post subject: Reply with quote

Master

Joined: 17 Jul 2017
Posts: 276
Location: UK

madrox wrote:
Thank you abhi_thri for your response. I agree with the part that this should be a part of another flow, but since i need to process all the message after the processes of the main flow itself that would not work.
Could you elaborate on this, i did not follow what you meant. How would I trigger this flow if it has a MQinput node?


By splitting the subflow into a separate flow which starts with an MQInput node before the MQGet node you can achieve you are trying to do,

- the first flow after finishing all it's processing can trigger the second flow by sending an trigger (any message) to it.

- the second flow (MQInput --> MQGet --> etc) will start processing after receiving the first trigger from the main flow. After processing a message from the MQGet queue it will send send another trigger message to itself (i.e self triggering - by sending a trigger message to the queue MQInput listens to)
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Tue Jul 09, 2019 4:07 am Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20025
Location: LI,NY

abhi_thri wrote:

By splitting the subflow into a separate flow which starts with an MQInput node before the MQGet node you can achieve you are trying to do,

- the first flow after finishing all it's processing can trigger the second flow by sending an trigger (any message) to it.

- the second flow (MQInput --> MQGet --> etc) will start processing after receiving the first trigger from the main flow. After processing a message from the MQGet queue it will send send another trigger message to itself (i.e self triggering - by sending a trigger message to the queue MQInput listens to)

A little bit convoluted. I liked Timber's suggestion better.
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
abhi_thri
PostPosted: Tue Jul 09, 2019 4:49 am Post subject: Reply with quote

Master

Joined: 17 Jul 2017
Posts: 276
Location: UK

fjb_saper wrote:

A little bit convoluted. I liked Timber's suggestion better.


One thing to keep in mind when using Propagate within a loop is that the whole Propagate loop is still part of the same transaction which means if any of the those 10000+ messages fails for some reason the whole lot will get backed out, if you were doing it in separate flows that won't be the case. Alternatively for the Propagate scenario you could make the part of the processing non-transnational if that holds up
Back to top
View user's profile Send private message
timber
PostPosted: Tue Jul 09, 2019 12:53 pm Post subject: Reply with quote

Grand Master

Joined: 25 Aug 2015
Posts: 1002

Agreed. And if this processing is not intended to be a single 10,000-message transaction then your solution may well be a better choice.
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexWebSphere Message Broker SupportStack space exceeded while using MQGET node
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.