| Author |
Message
|
| Vitor |
 Posted: Tue Jul 03, 2012 7:30 am Post subject: WMB Speed Control |
 |
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
WMB 7.0.0.3 running on Solaris.
Me running on caffine.
A flow is deployed to a single execution group with no additional instances. About twice a day, a file of around 90k records is placed onto the broker's server & processed via a FileInput node. There's some dancing and the input COBOL-esq data comes out looking like XML and is put onto a queue for processing by a downstream package.
The flow processes records about 10 times faster than the package, leading to a build up of messages on the queue. I've been told that:
| Quote: |
| It's unacceptable to have messages waiting in a queue like that |
After confirming that they were serious and trying to explain (unsuccessfully) why the object is called a "queue" by analogy with a busy McDonnalds, I'm now tasked with finding a way to make WMB process 10 times slower, as it's "impossible" to make the package run faster.
Suggestions from the assembled readership welcomed. I have some ideas, but they mostly involve pain, death, suffering and are not the kind of technical implementation being sought here.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
 |
| lancelotlinc |
| Posted: Tue Jul 03, 2012 7:35 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
Add a JCN with a Thread.sleep. I've tried ESQL sleep function, but cannot compile in 7.0.0.3. Can't find the ESQL function. JCN sleep works.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jul 03, 2012 7:41 am Post subject: Re: WMB Speed Control |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| trying to explain (unsuccessfully) why the object is called a "queue" by analogy with a busy McDonnalds, |
See, that's you trying to use British words to an American audience.
Thread sleep is fine. ESQL sleep is probably more up your alley.
Timeout nodes is probably better, as they don't keep your flow invocation hanging around.
But you either need the records sitting in some queue somewhere or you need them sitting in memory waiting to be processed. I know which I'd rather have.
Tell 'em you'll need to create a dedicated channel and they'll have to have write CICS code to create a PCF message to start the channel, if you can't have the records piling up on their side. That should change their opinions on the value of a queue. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jul 03, 2012 7:52 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| lancelotlinc wrote: |
| Add a JCN with a Thread.sleep. I've tried ESQL sleep function, but cannot compile in 7.0.0.3. Can't find the ESQL function. JCN sleep works. |
This was 1 option I came up with that didn't involve a large, blood stained axe. What's your view on calculating / providing the sleep period? Clearly you could come up with a number which yields a 10x speed reduction through performance testing and/or trial & error; is it worth trying to parameterise it? I'm looking to the day when the impossible happens & this thing goes from 10 TPS to 20 TPS & they want WMB to speed up a little.
Also, if the requirement is for WMB to run at "10 TPS" rather than "10x slower", do you see that as realistically achiveable? Or should I just sleep until I hit 12 - 13 TPS & say that's good enough?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jul 03, 2012 7:59 am Post subject: Re: WMB Speed Control |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| Vitor wrote: |
| trying to explain (unsuccessfully) why the object is called a "queue" by analogy with a busy McDonnalds, |
See, that's you trying to use British words to an American audience. |
Still trying to master the language.
| mqjeff wrote: |
| Thread sleep is fine. ESQL sleep is probably more up your alley. |
Quite.
| mqjeff wrote: |
| Timeout nodes is probably better, as they don't keep your flow invocation hanging around. |
Hanging round is potentially an advantage here. 
But you either need the records sitting in some queue somewhere or you need them sitting in memory waiting to be processed. I know which I'd rather have.
| mqjeff wrote: |
| Tell 'em you'll need to create a dedicated channel and they'll have to have write CICS code to create a PCF message to start the channel, if you can't have the records piling up on their side. |
The package in question is distributed not CICS & they're fairly cool with me running at full speed.
| mqjeff wrote: |
| That should change their opinions on the value of a queue. |
The opinion is coming from our WMQ team (yes, 2 teams, 1 for WMQ, 1 for WMB. Pity me.) who don't believe messages should pile up on a queue.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Tue Jul 03, 2012 8:28 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| Vitor wrote: |
| lancelotlinc wrote: |
| Add a JCN with a Thread.sleep. I've tried ESQL sleep function, but cannot compile in 7.0.0.3. Can't find the ESQL function. JCN sleep works. |
This was 1 option I came up with that didn't involve a large, blood stained axe. What's your view on calculating / providing the sleep period? Clearly you could come up with a number which yields a 10x speed reduction through performance testing and/or trial & error; is it worth trying to parameterise it? I'm looking to the day when the impossible happens & this thing goes from 10 TPS to 20 TPS & they want WMB to speed up a little.
Also, if the requirement is for WMB to run at "10 TPS" rather than "10x slower", do you see that as realistically achiveable? Or should I just sleep until I hit 12 - 13 TPS & say that's good enough? |
>> What's your view on calculating / providing the sleep period?
I would express it as a UDP integer. That way, if your Test environment is slower (or faster) than your prod environment, you can configure it as a Bar File Override.
>> Or should I just sleep until I hit 12 - 13 TPS & say that's good enough?
I would use the Bar File Override to tune. Over time, if the requirements change, just change the BFO.
By the way, if you get the ESQL sleep function to work, please post a code example. I was not able to get it to compile.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jul 03, 2012 8:38 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| lancelotlinc wrote: |
| By the way, if you get the ESQL sleep function to work, please post a code example. I was not able to get it to compile. |
Will do. Not sure where the 7.0.0.4 upgrade fits in my immediate future but I'll check it out.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jul 03, 2012 8:43 am Post subject: Re: WMB Speed Control |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| mqjeff wrote: |
| Timeout nodes is probably better, as they don't keep your flow invocation hanging around. |
Hanging round is potentially an advantage here. |
Until the EG crashes because you've used up it's available memory? |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Jul 03, 2012 8:49 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| ESQL sleep works for me in 7.0.0.3 - have you done -f all on your broker? |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jul 03, 2012 9:01 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| zpat wrote: |
| have you done -f all on your broker? |
No I've done some flows already.... 
The effective level is at 7.0.0.3.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jul 03, 2012 9:27 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| In v7 compiling is done at toolkit, not broker. So whether or not he's done -f all, he still needs to make sure his toolkit is patched. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jul 03, 2012 9:35 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| In v7 compiling is done at toolkit, not broker. So whether or not he's done -f all, he still needs to make sure his toolkit is patched. |
Which is likewise at 7.0.0.3
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mgk |
| Posted: Tue Jul 03, 2012 10:22 am Post subject: |
|
|
Padawan
Joined: 31 Jul 2003
Posts: 1642
|
| Code: |
| DECLARE result BOOLEAN SLEEP(10000); |
You probably forgot that in ESQL you cannot ignore return values when you call functions... Certainly SLEEP went in before 7.0.0.3 IIRC...
Kind Regards,
_________________
MGK
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jul 03, 2012 10:40 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mgk wrote: |
| Code: |
| DECLARE result BOOLEAN SLEEP(10000); |
You probably forgot that in ESQL you cannot ignore return values when you call functions... Certainly SLEEP went in before 7.0.0.3 IIRC... |
It's the most obvious things which are the hardest to see. 
The code snippet posted here works fine under 7.0.0.3.
I've filled out the Feedback on the InfoCenter page to suggest revising the example code shown there, which don't show the return value being handled even thought the entry clearly (and correctly) refers to SLEEP as a function.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jul 03, 2012 10:42 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Does anyone have any different thoughts on the original question? With thanks obviously to all who have already responded. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
