| Author |
Message
|
| madrox |
 Posted: Sun Jul 07, 2019 2:47 pm Post subject: Stack space exceeded while using MQGET node |
 |
|
Acolyte
Joined: 11 Mar 2015
Posts: 71
|
I know this topic has been extensively discussed here, but unfortunately for me even after reading multiple of these post i am unable to get this working.
Here is my scenario.
I have about 10000+ messages on a queue which i need to get after other processes i run during a flow. I need to read the messages from the queue and make a WS call, depending on success or failure i write status to a file.
Here is what i have implemented:
after my other process is complete, i call a subflow. The subflow contains
inputnode>MQGET>compute(buildURL)>WSRequest>Compute(to write to file and loop back to MQGET
The above implementation works fine while i test my code with about 10-15message. But when i load the queue up with 10000 messages, it processes about 30 messages and i see the following in the logs
| Quote: |
| BIP2187E: The stack space limit('2097152') was exceeded during flow processing |
I am pretty positive its the loop while getting the messages of the queue. Is there a better way to achieve this? |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Sun Jul 07, 2019 3:16 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9472
Location: US: west coast, almost. Otherwise, enroute.
|
Moved to Broker forum.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| abhi_thri |
| Posted: Mon Jul 08, 2019 12:36 am Post subject: |
|
|
Knight
Joined: 17 Jul 2017
Posts: 516
Location: UK
|
hi...yes, it is the wired loop within the flow that is causing the stack overflow error as you suspected and it is generally discouraged to use any sort of wired loop within a flow due to this reason as it will exhaust the memory stack (as the flow need to store the current flow state before starting a new loop) depending on the no. of the loops.
As these 10000+ messages are independent ones you need to process it independently by using separate flow transactions. i.e once the first message is processed successfully instead of doing a hard wired loop back to the MQGet just emit a dummy message to 'inputnode' so that it will start a new transaction and process the next message.
To do that you could define the current subflow as a separate flow which starts with an MQInput node which can then receive first trigger to start processing the messages via the MQGet node and the subsequent dummy triggers. |
|
| Back to top |
|
|
| timber |
| Posted: Mon Jul 08, 2019 2:16 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
The suggestion from abhi_thri is the ideal because it splits the processing into separate, short-running transactions.
If you absolutely *need* to process all of the messages in single message flow execution then you can also do this:
- Write a loop in a Compute node with a PROPAGATE TO TERMINAL 'Out1' (or some other free terminal) statement.
- On terminal Out1, attach the message flow nodes to read exactly one message from the queue and process it. But do not wire any loop into the message flow.
This will result in a message flow which takes a very long time to complete, but it will not blow up the stack. |
|
| Back to top |
|
|
| madrox |
| Posted: Mon Jul 08, 2019 7:04 am Post subject: |
|
|
Acolyte
Joined: 11 Mar 2015
Posts: 71
|
| abhi_thri wrote: |
hi...yes, it is the wired loop within the flow that is causing the stack overflow error as you suspected and it is generally discouraged to use any sort of wired loop within a flow due to this reason as it will exhaust the memory stack (as the flow need to store the current flow state before starting a new loop) depending on the no. of the loops.
As these 10000+ messages are independent ones you need to process it independently by using separate flow transactions. i.e once the first message is processed successfully instead of doing a hard wired loop back to the MQGet just emit a dummy message to 'inputnode' so that it will start a new transaction and process the next message.
To do that you could define the current subflow as a separate flow which starts with an MQInput node which can then receive first trigger to start processing the messages via the MQGet node and the subsequent dummy triggers. |
Thank you abhi_thri for your response. I agree with the part that this should be a part of another flow, but since i need to process all the message after the processes of the main flow itself that would not work.
Could you elaborate on this, i did not follow what you meant. How would I trigger this flow if it has a MQinput node?
| Quote: |
| To do that you could define the current subflow as a separate flow which starts with an MQInput node which can then receive first trigger to start processing the messages via the MQGet node and the subsequent dummy triggers. |
|
|
| Back to top |
|
|
| madrox |
| Posted: Mon Jul 08, 2019 7:30 am Post subject: |
|
|
Acolyte
Joined: 11 Mar 2015
Posts: 71
|
| timber wrote: |
The suggestion from abhi_thri is the ideal because it splits the processing into separate, short-running transactions.
If you absolutely *need* to process all of the messages in single message flow execution then you can also do this:
- Write a loop in a Compute node with a PROPAGATE TO TERMINAL 'Out1' (or some other free terminal) statement.
- On terminal Out1, attach the message flow nodes to read exactly one message from the queue and process it. But do not wire any loop into the message flow.
This will result in a message flow which takes a very long time to complete, but it will not blow up the stack. |
Thanks Timber,
If i wrote a loop in a compute node, i would still need to propagate to MQGet terminal, would i not?
Right now here is what i am doing, after my other processes complete, I come to the subflow with the MQGet. So there is the
| Quote: |
| inputTerminal>MQGet>Compute>WSReq>Compute>FileOutput |
If i wrote a loop in the second compute i would still need to wire back to the MQGet In terminal |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jul 08, 2019 7:36 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| madrox wrote: |
| If i wrote a loop in a compute node, i would still need to propagate to MQGet terminal, would i not? |
Yes, but you'd be propogating 10,000 separate records in 10,000 threads to 10,000 copies of the MQGet node.
This is very different in terms of memory management to looping through a single copy of the MQGet node.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| timber |
| Posted: Mon Jul 08, 2019 1:18 pm Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
I think you are misunderstanding this pattern.
The ESQL loop goes into the first Compute node. It repeatedly PROPAGATEs to a terminal until it is told to stop (more on that later).
The nodes attached to the terminal process exactly one message. There is no wiring path from this 'leg' of the message flow back to the first Compute node. This leg just ends when the processing of one message is done, and then the stack unwinds back to the Compute node loop. The maximum stack depth is therefore (approximately) 1/10000th of what it would be in your current solution.
So how does the loop terminate? Well, one way is to check a value in the Environment tree every time around the loop. When MQGet returns with a 'no message found' error, set the Environment field and terminate the loop. And...make sure that you don't use LocalEnvironment (it will not get propagated back into the first Compute node when the stack unwinds). |
|
| Back to top |
|
|
| abhi_thri |
| Posted: Tue Jul 09, 2019 12:19 am Post subject: |
|
|
Knight
Joined: 17 Jul 2017
Posts: 516
Location: UK
|
| madrox wrote: |
Thank you abhi_thri for your response. I agree with the part that this should be a part of another flow, but since i need to process all the message after the processes of the main flow itself that would not work.
Could you elaborate on this, i did not follow what you meant. How would I trigger this flow if it has a MQinput node? |
By splitting the subflow into a separate flow which starts with an MQInput node before the MQGet node you can achieve you are trying to do,
- the first flow after finishing all it's processing can trigger the second flow by sending an trigger (any message) to it.
- the second flow (MQInput --> MQGet --> etc) will start processing after receiving the first trigger from the main flow. After processing a message from the MQGet queue it will send send another trigger message to itself (i.e self triggering - by sending a trigger message to the queue MQInput listens to) |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Jul 09, 2019 4:07 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| abhi_thri wrote: |
By splitting the subflow into a separate flow which starts with an MQInput node before the MQGet node you can achieve you are trying to do,
- the first flow after finishing all it's processing can trigger the second flow by sending an trigger (any message) to it.
- the second flow (MQInput --> MQGet --> etc) will start processing after receiving the first trigger from the main flow. After processing a message from the MQGet queue it will send send another trigger message to itself (i.e self triggering - by sending a trigger message to the queue MQInput listens to) |
A little bit convoluted. I liked Timber's suggestion better. 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| abhi_thri |
| Posted: Tue Jul 09, 2019 4:49 am Post subject: |
|
|
Knight
Joined: 17 Jul 2017
Posts: 516
Location: UK
|
| fjb_saper wrote: |
A little bit convoluted. I liked Timber's suggestion better. |
One thing to keep in mind when using Propagate within a loop is that the whole Propagate loop is still part of the same transaction which means if any of the those 10000+ messages fails for some reason the whole lot will get backed out, if you were doing it in separate flows that won't be the case. Alternatively for the Propagate scenario you could make the part of the processing non-transnational if that holds up |
|
| Back to top |
|
|
| timber |
| Posted: Tue Jul 09, 2019 12:53 pm Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
| Agreed. And if this processing is not intended to be a single 10,000-message transaction then your solution may well be a better choice. |
|
| Back to top |
|
|
|
|
