| Author |
Message
|
| aravind703 |
 Posted: Tue Jul 03, 2012 10:52 am Post subject: @ Vitor |
 |
|
Apprentice
Joined: 24 Sep 2011
Posts: 29
|
@ Vitor
i have a small confusion. If we use sleep function does the broker go to sleep for that specified time period. Stopping all the other applications running on that broker. |
|
| Back to top |
|
 |
| zpat |
| Posted: Tue Jul 03, 2012 10:55 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| No, just your message flow. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Tue Jul 03, 2012 12:06 pm Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| Vitor wrote: |
| different thoughts on the original question? |
I don't know about your POWER7 machine, but mine has a family of squirrels living inside the CPU case. If I don't put so much food out, they turn their exercise wheel more slowly. This is when I usually notice the mouse not responding quite as readily. Of course, I'm not sure who'se feeding the mouse... 
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jul 03, 2012 12:29 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| lancelotlinc wrote: |
| Vitor wrote: |
| different thoughts on the original question? |
I don't know about your POWER7 machine, but mine has a family of squirrels living inside the CPU case. If I don't put so much food out, they turn their exercise wheel more slowly. This is when I usually notice the mouse not responding quite as readily. Of course, I'm not sure who'se feeding the mouse... |
It's a valid point and I wondered about it; the broker is question lives in a virtual world so the amount of resource it get is configurable. Thr problem is that while this bunch appear to run at the speed of an snail dragging an anvil, other consumers want their data at 1000 TPS or better. So I can't retard the entire broker just for these retards......
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Jul 03, 2012 1:12 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
I've seen the same sort of issue your PHB's have over using Queues before.
If they have been drinking the 'Pure and unadulterated WebServices' cool aid then queues don't have a role to play anywhere.
It would be interesting to know what their reasons are for not wanting data to be held on the queue prior to processing.
I wonder if you can enact some for of request/reply scenario with the application that is running slow.
If you can, you can write the data (to a Q naturally) where it is held until the response from a previous request is received and a seconf flow reads the next message from the hidden q and sends it to the app that runs like a one legged dog.
you would satisfy the demands of the PHB's and not have to artificially slow down broker which is IMHO fundamentally the wring thing to do.
One other possibility is to run you broker instance on a 1PVU box. That will be just about fast enough... 
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue Jul 03, 2012 4:09 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| smdavies99 wrote: |
It would be interesting to know what their reasons are for not wanting data to be held on the queue prior to processing.
|
I agree, what is the reason for their angst with a queue that is queuing? I guess you and your trout addressed any valid concerns they had, but what were they?
Maybe they're basing this fear of a queue the one time their QM was brought down for maintenance and the lousy queue lost all their messages, all those priceless non persistent messages.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Tue Jul 03, 2012 11:07 pm Post subject: Re: WMB Speed Control |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| Vitor wrote: |
| WMB 7.0.0.3 running on Solaris ... I'm now tasked with finding a way to make WMB process 10 times slower, as it's "impossible" to make the package run faster. |
Modify the receiving application to send a reply when it gets a message, and wait for the reply before sending the next message. (smdavies99 suggested something similar, but with a hidden queue; though it's not clear how to hide a queue from the WMQ team.)
Or add a JCN to monitor the output queue depth, and sleep when the queue depth gets too large.
Or modify the message flow to put one big message on the queue, rather than 90,000 little messages. Modify the receiving application to break up the big message.
Or move the broker to Windows. (That might be even slower than smdavies99's 1PVU box, or lancelotlinc's rodents.) Use a modem to transfer the file from Solaris to Windows, if necessary. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Jul 03, 2012 11:51 pm Post subject: Re: WMB Speed Control |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| rekarm01 wrote: |
[
Modify the receiving application to send a reply when it gets a message, and wait for the reply before sending the next message. (smdavies99 suggested something similar, but with a hidden queue; though it's not clear how to hide a queue from the WMQ team.)
|
Sadly you may have to revert to a little magic/slight of hand here.
There is a US TV Show that gets aired very late at night here in Blighty. It is called 'In Plain Sight'. with that in mind...
| Code: |
SYSTEM.BROKER.DELAY.QUEUE
or
SYSTEM.BROKER.DELAY.STREAM
|
I know the purists/IBM'ers at Hursley/and others will be shuddering at the thought of doing this but sometimes, needs must.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| McueMart |
| Posted: Wed Jul 04, 2012 12:21 am Post subject: |
|
|
Chevalier
Joined: 29 Nov 2011
Posts: 490
Location: UK...somewhere
|
| Horrible solution to a horrible problem: How about setting the max queue depth of the output queue to 1 (or something acceptably small)? Your flow would then need to be able to handle an MQPut error by retrying it a moment later. I guess this is just a different approach to the idea of using a JCN to monitor the queue depth which rekarm posted. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jul 04, 2012 6:20 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Vitor wrote: |
After confirming that they were serious and trying to explain (unsuccessfully) why the object is called a "queue" by analogy with a busy McDonnalds, I'm now tasked with finding a way to make WMB process 10 times slower, as it's "impossible" to make the package run faster.
Suggestions from the assembled readership welcomed. I have some ideas, but they mostly involve pain, death, suffering and are not the kind of technical implementation being sought here. |
Aside from the pain of non persistent messages disappearing on them, could the comment of queue depth being unacceptable just mean that they use a mix of online/batch and they don't want the online stuff to have to wait behind the batch? i.e. persistent messages + priority for the online stuff, (possibly setting up aliases with different priorities, default persistence) could fix the problem?
Methinks you need more dialogue with the offending party... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Esa |
| Posted: Thu Jul 05, 2012 12:53 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
Sleeping a flow that reads records from a file is quite challenging transaction point of view. Depending on how log the whole process will take. If you need to make the output messages available one by one, you must put them outside the transaction. What if the eg crashes while the flow is processing a file, what if there is an exception? The file is backed out, the whole file or just the records that haven't been processed yet? I have not tested it, but maybe somebody else can tell it.
Reading the whole file in, putting it in a work queue and having another flow read the file message, process only the first record and write the rest of the message back to the work queue would certainly slow down the process. And make it safer from transaction point of view.
The problem is that while the file message gets smaller, the flow will get faster. So you will have to add a sleep that will increase with the amount of records processed. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jul 05, 2012 5:27 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| PeterPotkay wrote: |
| smdavies99 wrote: |
It would be interesting to know what their reasons are for not wanting data to be held on the queue prior to processing.
|
I agree, what is the reason for their angst with a queue that is queuing? I guess you and your trout addressed any valid concerns they had, but what were they? |
After using the Trout Of Enquiry I received the following (over a series of meetings & conversations):
- all those messages will take up disc space and we don't have budget to buy more.
- If you buy more disc space we'll have to get the Solaris sys admins to install it and they'll make us fill out forms and attend meetings
- If we let you store messages in a queue everyone will want to store messages in a queue like it's a database. We've already had problems where people send presistent response message with no expiry & the messages just pile up. (I actually pushed back on that & asked how that goof got through the design review. I was told "we don't review designs, we're not the message police"....)
- If you have a large queue depth it takes the queue manager too long to fail over & we have a tight recovery SLA (I checked the local RA Wiki - they have a 60 minute window)
- If you have a deep queue, the queue manager starts to run really really slowly & impacts all the other users
| PeterPotkay wrote: |
| Maybe they're basing this fear of a queue the one time their QM was brought down for maintenance and the lousy queue lost all their messages, all those priceless non persistent messages. |
Not that bad, they do understand the difference between persistent & non-persistent but another reason I trouted out of them was "While the messages are on the queue it's on us if anything happens to them. Keeping the queues empty manages that risk"
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jul 05, 2012 5:31 am Post subject: Re: WMB Speed Control |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| smdavies99 wrote: |
There is a US TV Show that gets aired very late at night here in Blighty. It is called 'In Plain Sight'. with that in mind...
| Code: |
SYSTEM.BROKER.DELAY.QUEUE
or
SYSTEM.BROKER.DELAY.STREAM
|
|
It's a good idea & if I'd known up front this was going to happen I'd have snuck those queues in at broker creation time when I had mqm access & could plausibly have said "well, they're system queues I can't influence those." As it is I'd have to have them add them & after the discussions they might smell a rat.
It's hard enough getting a new alias set up when I want a new execution group.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Jul 05, 2012 9:56 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| Vitor wrote: |
After using the Trout Of Enquiry I received the following (over a series of meetings & conversations):
- all those messages will take up disc space and we don't have budget to buy more.
- If you buy more disc space we'll have to get the Solaris sys admins to install it and they'll make us fill out forms and attend meetings
- If we let you store messages in a queue everyone will want to store messages in a queue like it's a database. We've already had problems where people send presistent response message with no expiry & the messages just pile up. (I actually pushed back on that & asked how that goof got through the design review. I was told "we don't review designs, we're not the message police"....)
- If you have a large queue depth it takes the queue manager too long to fail over & we have a tight recovery SLA (I checked the local RA Wiki - they have a 60 minute window)
- If you have a deep queue, the queue manager starts to run really really slowly & impacts all the other users
|
Hold on, let me check....yup, they got 'em all covered.
Lazy reasons
Cheap reasons
Fearful reason
Reasons based on misunderstanding how MQ works
You're anything but meek and uninformed, so there's no reason to provide counterpoints to each of the above. You know them and would have shared them. If you're here for alternate solutions, they must have pictures of you in compromising positions or something.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Jul 05, 2012 10:11 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| I was told "we don't review designs, we're not the message police"....) |
Right. So they're not the message police, so they don't have any say in how many messages pile up in queues. |
|
| Back to top |
|
|
|
|
