| Author |
Message
|
| WBI_User21 |
 Posted: Sun Jul 25, 2010 11:52 am Post subject: Cluster load balancing problems -Help required |
 |
|
Voyager
Joined: 12 Jun 2007
Posts: 98
|
Hi MQ experts,
Env :
MQ 6.0.2.1
Solaris 5.10
We have application X with 6 queuemanager clustered with serval other queue managers (upstream/downstream) . Due to increase in message rate we have added 6 more queue manager ( different hosts) as partial repository . On newly added queue manager , once queues are brought into cluster , we observed that newly added queues are receiving high amount of messages. To sort out problems ,we have tried following options ..
1. Checked all sender channels from Sender to application X queue managers . All are OK . No problems at all .
2. Identified some work around provided by IBM but no luck
http://www-01.ibm.com/support/docview.wss?rs=171&uid=swg1IZ51783
3. On further investigation we found that , newly added PR , we have brought only one queue into cluster . As per my understanding CLuster work load balancing works at channel level ( note : we have not tuned any cluster work load balancing param at queue level or channel level . Every thing was set to default as we are not interested in weighted load balancing ). Since only one queue added into cluster in newly added PR, cluster receiver channel is happy to pump/receive more messages to new target i.e newly added PR Queue managers (PR) .
Does any one has faced similar problems .. Can you share your experiences ?
Thanks in advance |
|
| Back to top |
|
 |
| exerk |
| Posted: Sun Jul 25, 2010 12:05 pm Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
Check the bind options on the newly introduced queues are BIND(NOTFIXED).
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| WBI_User21 |
| Posted: Sun Jul 25, 2010 12:33 pm Post subject: |
|
|
Voyager
Joined: 12 Jun 2007
Posts: 98
|
| Yes BIND is NOT FIXED .. This is one of the properties in our enterprise to qualify AIS... |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Jul 25, 2010 12:48 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Apply the corresponding fix pack as in the link you provided.
We have also found that stopping the corresponding cluster sender channels with mode inactive (so the interruption is minimal) will reset this parameter.
To find out which those are, just check the queue usage of the new destination.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| WBI_User21 |
| Posted: Sun Jul 25, 2010 1:04 pm Post subject: |
|
|
Voyager
Joined: 12 Jun 2007
Posts: 98
|
Thanks fjb_saper. ..
Applying 6.0.2.8 Fixpack will really solve problem ?
Does number of queues added into cluster in PR influence load balancing ..
Let me describe scenario again
There are 6 receiving qmgrs (one of them is FR and other FR is on sender qmgrs ) . 6 more receiving qmgrs are added as PR . On newly added qmgrs , only few queues ( infact one queue ) are brought into cluster instead all queues present in cluster. Does this give cluster receiver channel opportunity receive more messages from upstreams when compared to other queue managers ? ..
Thanks in advance .. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Jul 25, 2010 6:34 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Go to 6.0.2.9 it's been out there for a while. And yes that should fix it according to the APAR you pointed to.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| WBI_User21 |
| Posted: Fri Jul 30, 2010 3:11 am Post subject: |
|
|
Voyager
Joined: 12 Jun 2007
Posts: 98
|
We have upgraded to 6.0.2.9 .. But still have problems with uneven load balancing .
We are stuck with same question again ?
Does nr of queues added in cluster influence load balancing ?
I have 8 receiving qmgrs out of which 6 have 10 queues in cluster and 1 had only 1 queue receiving messages . This seems to be creating problem . We are trying to simulate this in test environment .. Any thoughts ? |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Jul 30, 2010 4:15 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| WBI_User21 wrote: |
| Does nr of queues added in cluster influence load balancing ? |
Only in so far as all queues in the cluster are considered as part of the workload balancing algorythm.
| WBI_User21 wrote: |
| I have 8 receiving qmgrs out of which 6 have 10 queues in cluster and 1 had only 1 queue receiving messages. |
It's old & simple advice but with clusters you have to dot the i's and cross the t's:
- make sure all the cluster sender/receiver channels are running
- make sure you have only 2 FRs and that all the cluster senders point to one or the other
- make sure all the queues have their cluster attribute set (I've been told a number of times over the years that adding a queue manager to a cluster automatically adds all the queues to the cluster. Seriously.)
- make sure all the cluster sender / receiver channels have their cluster attribute set
- make sure you can display all the instances of the cluster queues (and indeed all the cluster queue managers) from an FR with QCLUSTER & CLUSQMGR
Finally & most importantly with load balancing problems check that all the applications are using BIND_NOT_FIXED. As has been said many, many, many times on this forum DEFBIND is just a default. The application may or may not be picking it up and using it. As an admin you can't enforce this any more than you can enforce persistence.
<plug>Not out of the box anyway</plug>
As I said all simple points & I'm sure you've already checked some if not all of them.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| WBI_User21 |
| Posted: Sat Jul 31, 2010 7:13 am Post subject: |
|
|
Voyager
Joined: 12 Jun 2007
Posts: 98
|
Hi Vitor
| Quote: |
| make sure all the cluster sender/receiver channels are running |
.. .Some of them are inactive state .However , as per my understanding , we need not start channels explicity in clustering as round robin algorithm kick starts them
| Quote: |
| make sure you have only 2 FRs and that all the cluster senders point to one or the other : |
Yes Only 2 FRs & both are healthy ..
| Quote: |
| - make sure all the queues have their cluster attribute set (I've been told a number of times over the years that adding a queue manager to a cluster automatically adds all the queues to the cluster. Seriously.) |
OK .. some break through here . Do you mean the queues , I don't really want to receive messages in are also need to be defined with CLUSTER property set ( i.e cluster property set to BLANK CLUSTER(' ')). I have missed this .
| Quote: |
- make sure all the cluster sender / receiver channels have their cluster attribute set
- make sure you can display all the instances of the cluster queues (and indeed all the cluster queue managers) from an FR with QCLUSTER & CLUSQMGR |
Yes above checks are as expected . No issues with channels or queue added into cluster never disappeared when displayed using DIS QC(QUEUE*)
Also application uses DEFPSIST and DEFBIND as defined on Queue . We have set DEFBIND as NOT FIXED.. in all receiving qmgrs |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sat Jul 31, 2010 11:23 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| OK .. some break through here . Do you mean the queues , I don't really want to receive messages in are also need to be defined with CLUSTER property set ( i.e cluster property set to BLANK CLUSTER(' ')). I have missed this . |
When a qmgr joins a cluster, all the definitions known to the FRs will become available to other qmgrs in the cluster. Likewise, all object definitions known to the new qmgr will be shared with the FR, and available to all other qmgrs in the cluster.
If the new qmgr does not have an instance of the destination queue OR if the destination queue does not specify the cluster attribute (with the name of the cluster), then this qmgr will not be a cluster-queue destination for messages from other qmgrs in the cluster.
CLUSTER property set to CLUSTER('') indicates that this object (queue, channel,...) will not be known to other qmgrs in the cluster.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Sat Aug 07, 2010 11:40 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| WBI_User21 wrote: |
There are 6 receiving qmgrs (one of them is FR and other FR is on sender qmgrs ) . 6 more receiving qmgrs are added as PR . On newly added qmgrs , only few queues ( infact one queue ) are brought into cluster instead all queues present in cluster. Does this give cluster receiver channel opportunity receive more messages from upstreams when compared to other queue managers ? ..
. |
Yes, working as designed. Your original queue managers have a lot more traffic going to them because they have a lot more queues participating and getting traffic. The cluster is trying to be slick and load balance for you, but since its load balancing on the channel level and not the queue level, it sees the new QMs as not busy at all (overall) when compared to the original QMs.
Imagine you set up 10 Partial Repository QMs all with QueueA clustered. AppA send one message a minute to QueueA. You will basically get an even distribution. Then one day QueueB is added to QM1 and QM2 only. AppB sends 1000 thousand messages a minute. Guess what - you may never see messages go to QueueA on QM1 and QM2 again if QueueA is available on QM3 thru QM10. 
You may be able to fake out this behavior by playing with the various cluster attributes but it will be tricky and not fool proof. The fact of the matter is that when you have multiple apps sharing a cluster, they can influence each others load balancing characteristics.
A good rule of thumb for clusters is make each cluster queue manager as similar as every other one in that cluster. Since the load balancing is happening at the cluster channel level, using my example above, you may consider making a cluster for Queue A and an overlapping cluster for Queue B, each cluster with its own cluster channels. That will work, but is only feasible in simple scenarios. You wouldn't want to have Cluster A thru Cluster Z stacked on top of each other.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| WBI_User21 |
| Posted: Sat Aug 07, 2010 2:11 pm Post subject: |
|
|
Voyager
Joined: 12 Jun 2007
Posts: 98
|
Thanks PeterPotkay . This is exactly what we observed in test environment and something very interesting which IBM never documented in MQ product documentation or i misunderstood cluster load balancing algorithm .
Apparently we have 6 additional queue managers to be added in production with only single queue in cluster in addition to existing 9 qmgrs with 9+ queues in cluster.Now we have decided to all 6 queue manager at a time instead adding them one by one so that messages get load balanced across these 6qmgrs and application happy to process them using its 6 different instances. |
|
| Back to top |
|
|
| mvic |
| Posted: Tue Sep 14, 2010 3:13 pm Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Sep 14, 2010 4:49 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| (I've been told a number of times over the years that adding a queue manager to a cluster automatically adds all the queues to the cluster. Seriously.) |
You've been misled.
When you add a qmgr to a cluster, only the cluster objects of the new qmgr become available to the cluster - not all objects.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Sep 14, 2010 7:43 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Vitor wrote: |
| I've been told a number of times over the years that adding a queue manager to a cluster automatically adds all the queues to the cluster. Seriously. |
Nice touch of sarcasm and irony Vitor... However for the newbies you might have added a sarcasm tag...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
