| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Have you faced anything like this? |
« View previous topic :: View next topic » |
| Author |
Message
|
| mqonnet |
 Posted: Wed Sep 24, 2008 10:27 am Post subject: Have you faced anything like this? |
 |
|
Grand Master
Joined: 18 Feb 2002
Posts: 1114
Location: Boston, Ma, Usa.
|
All,
I have a good "food for thought" question and looking to see what other companies did in a similar scenario.
Scenario:
2 FR and 2 PR. Let us assume that there are 10 applications with 10 different queues (cluster Pairs) hosted on 2 PR's. Let us say to balance the load, 2 apps/q's have extremely traffic and the rest 8 are very light. Load balancing does not matter for the 8 applications/q's.
All is well.
After certain period of time, as the business grows, the load on 2PR's grows exponentially for ONE of the 2 heavy traffic applications/q's. It is decided that ONE more PR will be added for JUST this application/q. Which means, now for ONE application there will be 3 q's hosted on 3 PR's and for the rest of the 9 applications there will be ONLY 2 q's hosted 2 PR's.
Sounds perfectly valid business case.
Guess What??? MQ Balks on it. Why? Because it tries to "act" smart. 
It takes the lowest busiest route for this ONE application and instead of load balancing across 3 PR's, it now sends all traffic to NEW PR/q. (as designed and documented)
Question:
1) Has anyone faced something like this, which sounds like a perfectly simple business case.
2) What has been done in such cases?
Considerations:
1) To keep "business as usual", the solution needs to have minimal impact without any major changes.
2) Adding more clusters to the picture (suggested by IBM) cannot be considered as it is not scalable.
We have a PMR with IBM open and none of their answers really did convince us with an apt solution. We are working for a change request.
The aim of this question and thread here is to learn from other's experiences and brain storm with some of the largest and the greatest minds (real MQ users  ) in this space.
Cheers
DD
_________________
IBM Certified WebSphere MQ V5.3 Developer
IBM Certified WebSphere MQ V5.3 Solution Designer
IBM Certified WebSphere MQ V5.3 System Administrator |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Wed Sep 24, 2008 10:35 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| I'll take "What is CLWLPRTY?" for 200, Alex. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Sep 24, 2008 10:37 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
MQ 6.0.2.5, due out any day now, addresses this clustering annoyance.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Sep 24, 2008 10:43 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Sep 24, 2008 10:45 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
CLWLWGHT?
 |
|
| Back to top |
|
|
| mqonnet |
| Posted: Wed Sep 24, 2008 10:47 am Post subject: |
|
|
Grand Master
Joined: 18 Feb 2002
Posts: 1114
Location: Boston, Ma, Usa.
|
Peter,
Thanks for chipping in. Incidentally, IBM also proposed the same fix and we applied it. But it turns out that the wording in the APAR description is misleading and Hursley is working on it to correct the same so that it does not lead us to believe what it does.
Bottom line is, we are back to square one after the fix.
Cheers
DD
_________________
IBM Certified WebSphere MQ V5.3 Developer
IBM Certified WebSphere MQ V5.3 Solution Designer
IBM Certified WebSphere MQ V5.3 System Administrator |
|
| Back to top |
|
|
| mqonnet |
| Posted: Wed Sep 24, 2008 10:53 am Post subject: |
|
|
Grand Master
Joined: 18 Feb 2002
Posts: 1114
Location: Boston, Ma, Usa.
|
By the way, can someone from IBM (Jeff or others) please fix the spelling mistake on this url?
Local fix
Recycling all QM's in the cluster will reset counters which are
at the root cause of thsi problem.
| PeterPotkay wrote: |
| http://www-01.ibm.com/support/docview.wss?rs=171&context=SSFKSJ&context=SSEP7X&q1=6.0.2.5+cluster&uid=swg1IZ23058&loc=en_US&cs=utf-8&lang=en |
_________________
IBM Certified WebSphere MQ V5.3 Developer
IBM Certified WebSphere MQ V5.3 Solution Designer
IBM Certified WebSphere MQ V5.3 System Administrator |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Sep 24, 2008 10:54 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
 That was one of the main reasons I was going to upgrade to 6.0.2.5. Oh well.
I did bring this point up at one of the MQ Conferance's Feedback sessions that was dedicated to MQ clustering. It was acknowledged.
I don't there is any way around this for now, other then introducing a second MQ cluster stacked on top of your current MQ cluster. The new cluster includes a new set of CLUSRCVR / CLUSSNDR channels, and the only queues and QMs in it are the 3 for this application. I *think* this would get around the problem. You would have to test, but since the only things using the new channels would be the messges realted to this one app, the workload should be even, no?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mqonnet |
| Posted: Wed Sep 24, 2008 11:00 am Post subject: |
|
|
Grand Master
Joined: 18 Feb 2002
Posts: 1114
Location: Boston, Ma, Usa.
|
Peter,
Appreciate your comments.
What you suggested is what we also thought of and was in fact suggested by IBM as well. But the problem is, this approach is NOT practical. We design and architect MQ environment with the information we have on hand and with a buffer of say 2 years, with some visibility it is even better.
But business drive the designs at the end of the day. As business grow, we need to have the underlying infrastructure scale appropriately without making drastic changes.
However, adding clusters, though will resolve this problem, we cannot continue to do this on a regular basis. Moreover, there needs to be a justification for any design changes. In this case, the only thing that i can think of is saying, "Because of limitations within MQ" we had to take this measure. Vertical Scalability is again another question. Though this approach solves the problem, as the number of apps grow, we need to redo the design/clustering for all such q's/applications that are being impacted.
Any product should provide scalability. Applications sitting on top of it shouldn't compromise on their design to address such issues. IMHO.
Cheers
DD
_________________
IBM Certified WebSphere MQ V5.3 Developer
IBM Certified WebSphere MQ V5.3 Solution Designer
IBM Certified WebSphere MQ V5.3 System Administrator |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Sep 24, 2008 11:33 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Agree 100%. Do it once for this app's queues, and then next week you have to define another set of queues in another stacked cluster. Ugh.
This is a flaw in the current MQ cluster design in my opinion. With no easy work around.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mq_developer |
| Posted: Thu Sep 25, 2008 7:39 pm Post subject: |
|
|
Voyager
Joined: 18 Feb 2002
Posts: 82
|
Well , yes its a known issue . I don't have any bright ideas other than what is discussed , except an argument to make.
This is not a "BUG" in any means , lets take step back and look at this. Clustering is grouping of "similar" nodes to collectively "share" the workload. Right now this problem occurs only when one of the node in "Cluster" is unique & unlike.
Right now what your are asking for IBM to do is to allow "granularity/control" at queue level , as opposed to current "cluster" level.
If you look at it from architect/design perspective , current behavior makes most sense.Should i need to "granularly" address performance constraints of an application (queues), "isolating" them to a dedicated cluster gives more control - what is desired.
Look at it in "real" life (as you quote - "real" mq users ) perspective , if 3 windows are open , two of them are already servicing 'n' type of requests and service you need is "dedicatedly" offered in window-3 (in addition to other 2 windows) with out any wait - where will you go? 
All in all , i don't see this flaw in any which way nor a "change" is required in this behavior. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Sep 26, 2008 6:56 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
If you asked 1000 MQ Admins / Architects / Developers should one application's messages effect another's distribution, what do you think the % would be that would say "Sure! Why not?"
Note that I did not call it a bug. Its working as designed. I think the design should be changed. Either load balance at the q level, or allow an MQ Admin to choose queue or channel level load balancing, because you are right, for a small % of users, they may want to load balance by cluster channel.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mq_developer |
| Posted: Fri Sep 26, 2008 9:12 am Post subject: |
|
|
Voyager
Joined: 18 Feb 2002
Posts: 82
|
| Quote: |
| If you asked 1000 MQ Admins / Architects / Developers should one application's messages effect another's distribution, what do you think the % would be that would say "Sure! Why not?" |
See point i want to highlight is "imbalance" is created by the architecture , so is the behavior. Clustering is grouping of "similar" nodes, now we introduce one node like unlike into the mix , shouldn't there be an "imbalance" ?
Bottomline is don't "group" them all into one in the first place , which is the solution proposed by IBM & used by others to create new cluster.
| Quote: |
| Note that I did not call it a bug. Its working as designed. I think the design should be changed. Either load balance at the q level, or allow an MQ Admin to choose queue or channel level load balancing, because you are right, for a small % of users, they may want to load balance by cluster channel. |
All i am saying current design makes sense , design shouldn't be changed , it should be used correctly.
[/quote] |
|
| Back to top |
|
|
| mqonnet |
| Posted: Fri Sep 26, 2008 10:02 am Post subject: |
|
|
Grand Master
Joined: 18 Feb 2002
Posts: 1114
Location: Boston, Ma, Usa.
|
Good points.
Let me try and answer your thoughts.
This is not a "BUG" in any means , lets take step back and look at this.
- You develop and sell a product. Your product does 10 things and in this world where technology and business needs changes everyday, if your product cannot address all issues, then being the developer of the product you cannot say that this is how it is designed to work and get away. Put yourself in customer's shoes and you will realize if this is a bug or not.
Clustering is grouping of "similar" nodes to collectively "share" the workload. Right now this problem occurs only when one of the node in "Cluster" is unique & unlike.
--Agreed.
Right now what your are asking for IBM to do is to allow "granularity/control" at queue level , as opposed to current "cluster" level.
--Exactly.
If you look at it from architect/design perspective , current behavior makes most sense.Should i need to "granularly" address performance constraints of an application (queues), "isolating" them to a dedicated cluster gives more control - what is desired.
Look at it in "real" life (as you quote - "real" mq users ) perspective , if 3 windows are open , two of them are already servicing 'n' type of requests and service you need is "dedicatedly" offered in window-3 (in addition to other 2 windows) with out any wait - where will you go?
--Comparing apples to apples make more sense. Your example does not portray what is in production in any environment using MQ. There will NEVER be an instance wherein you would have 3 windows dedicated to just one person. This way you will have only one-on-one mapping and is undesirable from many aspects, not just one.
All in all , i don't see this flaw in any which way nor a "change" is required in this behavior.
--If you think from IBM's point of view, you are 100% right. Otherwise, your stance will definitely change.
Moreover, what you stated is what is already known. Just wanted to get thoughts on how to resolve it.
Cheers
DD[/quote]
_________________
IBM Certified WebSphere MQ V5.3 Developer
IBM Certified WebSphere MQ V5.3 Solution Designer
IBM Certified WebSphere MQ V5.3 System Administrator |
|
| Back to top |
|
|
| mqonnet |
| Posted: Fri Sep 26, 2008 10:07 am Post subject: |
|
|
Grand Master
Joined: 18 Feb 2002
Posts: 1114
Location: Boston, Ma, Usa.
|
See point i want to highlight is "imbalance" is created by the architecture , so is the behavior. Clustering is grouping of "similar" nodes, now we introduce one node like unlike into the mix , shouldn't there be an "imbalance" ?
Bottomline is don't "group" them all into one in the first place , which is the solution proposed by IBM & used by others to create new cluster.
-- This is completely IMPRACTICAL. As i mentioned earlier, you can forsee only as much. Which means your design and architecture is based on the facts that you have today. Tomorrow a sales guy succeeds in getting walmart as our customer and guess what, you get to see numbers that are 100 times the combined numbers you have seen in the past say 5 odd years.
| Quote: |
| Note that I did not call it a bug. Its working as designed. I think the design should be changed. Either load balance at the q level, or allow an MQ Admin to choose queue or channel level load balancing, because you are right, for a small % of users, they may want to load balance by cluster channel. |
All i am saying current design makes sense , design shouldn't be changed , it should be used correctly.
[/quote][/quote]
However, i will stick to my words and call it a BUG or at least a limitation that MUST be addressed. You cannot just say "that is how it is designed" and get away with it. This is a real life example and NOT an MQ cert question.
Moreover, please do understand that i do NOT want MQ to change it's behavior at all. What works now is how it ought to be. But in circumstances such as this one, there needs to be provisions made available so that the existing system can still leverage MQ features seamlessly without impacting the business.
Cheers
DD
_________________
IBM Certified WebSphere MQ V5.3 Developer
IBM Certified WebSphere MQ V5.3 Solution Designer
IBM Certified WebSphere MQ V5.3 System Administrator |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
