| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Exponential processing time when saving data in Environment |
« View previous topic :: View next topic » |
| Author |
Message
|
| francoisvdm |
 Posted: Wed Jul 21, 2010 1:45 am Post subject: Exponential processing time when saving data in Environment |
 |
|
Partisan
Joined: 09 Aug 2001
Posts: 332
|
MB V7, no fixpacks on Windows 7
Playing with some performance tests I came across the following:
Description:
BTW, I know the business logic is not clear, I reduced this code from a bigger flow.
Some setup nodes, then a compute node (LoopControl) that acts as a loop controller with the following code:
| Code: |
CALL CopyEntireMessage();
DECLARE iCounter INTEGER 4000;
WHILE iCounter > 0 DO
SET iCounter = iCounter - 1;
PROPAGATE TO TERMINAL 'out1' DELETE NONE;
END WHILE;
RETURN TRUE; |
So, the idea of the code above is to do the downstream nodes from 'out1' 4000 times. The first compute node after LoopControl 'Do1' looks like this:
| Code: |
SET OutputRoot = InputRoot;
SET Environment.DATASTORE.Data = InputRoot;
PROPAGATE TO TERMINAL 'out1';
RETURN TRUE; |
The compute node following 'out1' does almost nothing, here is the code:
| Code: |
SET OutputRoot = InputRoot;
SET OutputRoot.MQMD.PutTime = CAST(CURRENT_TIME AS TIME);
RETURN TRUE; |
Here is my problem:
In the second node, with the following line of code
| Code: |
| SET Environment.DATASTORE.Data = InputRoot; |
the following performance figures as I adjust the loop counter in the first node:
4000 repititions = 4 seconds
6000 repititions = 11 seconds
8000 repititions = 23 seconds
10000 repititions = 41 seconds
The above figures looks like an exponential increase in response time!!!
When I remove the copy to Environment the response times are linear and A LOT faster.
40 000 repitions = 6 seconds (yes, that is 40 thousand)
80 000 repititions = 11 seconds
160 000 repititions = 23 seconds etc.
Anybody got any idea why this is happening? Is there something I must clean up between the repititions? I think I've tried the obvious things like cleaning the environment etc.
The above is biting us in processing big files...and yes, I did read the docs on big file processing, but then again..... I'm sure I've missed something after spending many hours on this now.
_________________
If you do not know the answer or you get the urge to answer with "RTFM" or "Search better in this forum", please refrain from doing so, just move on to the next question. Much appreciated.
Francois van der Merwe |
|
| Back to top |
|
 |
| mgk |
| Posted: Wed Jul 21, 2010 2:17 am Post subject: |
|
|
Padawan
Joined: 31 Jul 2003
Posts: 1642
|
For performance, the SET statement simply performs the equivalent of a DETACH before assigning the source into the target, so your statement:
| Code: |
| SET Environment.DATASTORE.Data = InputRoot; |
will keep around an increasing number of elements. Normally this is fine because the trees are cleared out when the flow finishes and control returns to the Input node before the next message is processed. However, in your case you are not returning to the input node for a long time so the element count keeps growing and slows you down as new elements have to be created for each assign. Therefore, I suggest you try DELETEing the Data before you assign to it each time. This will make the elements available for reuse by the subsequent assignment. So your code would look like:
| Code: |
DELETE Environment.DATASTORE.Data;
SET Environment.DATASTORE.Data = InputRoot; |
Kind Regards,
_________________
MGK
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Wed Jul 21, 2010 10:13 am Post subject: Re: Exponential processing time when saving data in Environm |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
If the LoopControl Compute node is propagating the same message to the Do1 Compute node 4000 times, there's no reason to write it to the Environment each and every time. Wouldn't this work just as well?
| Code: |
IF Environment.DATASTORE.Data IS NULL
THEN
SET Environment.DATASTORE.Data = InputRoot;
END IF; |
If it's not the same message each time, then mgk's DELETE/SET should work well.
Another idea is to use OutputLocalEnvironment instead of Environment; unlike the Environment tree, don't a Compute node's Output trees get reclaimed each time the Compute node returns? |
|
| Back to top |
|
|
| francoisvdm |
| Posted: Thu Jul 22, 2010 4:01 am Post subject: |
|
|
Partisan
Joined: 09 Aug 2001
Posts: 332
|
From bad to worse.... when I add the line
| Code: |
| DELETE FIELD Environment.DATASTORE.Data; |
or
| Code: |
| DELETE FIELD Environment.DATASTORE; |
or
| Code: |
| DELETE LASTCHILD of Environment.DATASTORE; |
as suggested the broker dump around the 3950th loop. The symptoms are typical "stack overflow" ... but I must admit I can't see where I'm creating the loop.
Afaik when a leg after a PROPAGATE finishes and the control returns to the code just after the PROPAGATE the stack entries for that leg should be freed.... or at least, that was the way it worked in V6.1.
Any more ideas?
Thank you[/code]
_________________
If you do not know the answer or you get the urge to answer with "RTFM" or "Search better in this forum", please refrain from doing so, just move on to the next question. Much appreciated.
Francois van der Merwe |
|
| Back to top |
|
|
| mgk |
| Posted: Thu Jul 22, 2010 4:15 am Post subject: |
|
|
Padawan
Joined: 31 Jul 2003
Posts: 1642
|
| Quote: |
| Afaik when a leg after a PROPAGATE finishes and the control returns to the code just after the PROPAGATE the stack entries for that leg should be freed.... or at least, that was the way it worked in V6.1. |
This is true for the Output* trees in 6.1 and V7 (unless you use the DELETE NONE clauses), but it was never the case for the Environment tree, so using the OutputLocalEnvironment maybe an alternative as suggested above.
However, the broker should not dump from running this code, so I suggest you raise a PMR to get the dump looked at.
Kind Regards,
_________________
MGK
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. |
|
| Back to top |
|
|
| francoisvdm |
| Posted: Thu Jul 22, 2010 4:15 am Post subject: |
|
|
Partisan
Joined: 09 Aug 2001
Posts: 332
|
OK, I can prevent the error if in the very first node I change the line
| Code: |
| PROPAGATE TO TERMINAL 'out1' DELETE NONE; |
to
| Code: |
| PROPAGATE TO TERMINAL 'out1'; |
Now, is there a logical reason for this? Is this expected behaviour?
What if I want to preserve my output? Must I save it in Environment?
_________________
If you do not know the answer or you get the urge to answer with "RTFM" or "Search better in this forum", please refrain from doing so, just move on to the next question. Much appreciated.
Francois van der Merwe |
|
| Back to top |
|
|
| francoisvdm |
| Posted: Thu Jul 22, 2010 4:22 am Post subject: |
|
|
Partisan
Joined: 09 Aug 2001
Posts: 332
|
| mgk wrote: |
| Quote: |
| Afaik when a leg after a PROPAGATE finishes and the control returns to the code just after the PROPAGATE the stack entries for that leg should be freed.... or at least, that was the way it worked in V6.1. |
This is true for the Output* trees in 6.1 and V7 (unless you use the DELETE NONE clauses), but it was never the case for the Environment tree, so using the OutputLocalEnvironment maybe an alternative as suggested above.
However, the broker should not dump from running this code, so I suggest you raise a PMR to get the dump looked at.
Kind Regards, |
Thanks MGK .... I understand better now.... that pesky DELETE NONE. Also, the dump is an old error from many versions ago, as soon as you ran out of stack space the broker will dump. I opened a PMR a few years ago on it... got some silly answer back (avoid loops) lol
_________________
If you do not know the answer or you get the urge to answer with "RTFM" or "Search better in this forum", please refrain from doing so, just move on to the next question. Much appreciated.
Francois van der Merwe |
|
| Back to top |
|
|
| mgk |
| Posted: Thu Jul 22, 2010 5:37 am Post subject: |
|
|
Padawan
Joined: 31 Jul 2003
Posts: 1642
|
| Quote: |
| got some silly answer back (avoid loops) |
It is true that the advice is to avoid loops in your flow wiring logic between nodes, but in this case I see no loop from what you posted, and your evidence that this works without the DELETE statement backs this up. Therefore, I again suggest you raise a PMR, but also if you can post the call stack from the abend here I can take a quick look to see if there is anything obvious happening...
Kind Regards,
_________________
MGK
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Jul 22, 2010 6:31 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
It's not clear from this.
Are you *currently* getting dumps, using the above?
Or are merely discussing that in the past you got dumps using a very old version of Broker that may not have handled this particular case nearly as well and probably didn't support the DELETE clause at all...?
In general the advice is "If you get an abend, open a PMR".
In general, if you get back a response to a PMR that does not seem useful, you can continue to work the PMR until you either resolve the communications difficulty or get a better answer. Usually an answer that seems silly is the result of a miscommunication. |
|
| Back to top |
|
|
| francoisvdm |
| Posted: Thu Jul 22, 2010 11:30 pm Post subject: |
|
|
Partisan
Joined: 09 Aug 2001
Posts: 332
|
I agree, no obvious loop....just that "DELETE NONE" is not cleaning up the stack (by design or by accident). I'll open a PMR.
And yes, it is still dumping
_________________
If you do not know the answer or you get the urge to answer with "RTFM" or "Search better in this forum", please refrain from doing so, just move on to the next question. Much appreciated.
Francois van der Merwe |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
