| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Load Balancing cluster queue/QM inhibit question |
« View previous topic :: View next topic » |
| Author |
Message
|
| NealM |
 Posted: Thu Dec 18, 2014 9:43 am Post subject: Load Balancing cluster queue/QM inhibit question |
 |
|
Master
Joined: 22 Feb 2011
Posts: 230
Location: NC or Utah (depends)
|
We currently use a DataPower appliance for load balancing SOA web service requests between 4 WMB's, and it works great for that.
Now though we are looking at load balancing MQ message requests coming from a legacy system to these same 4 brokers. We don't see a way to involve DataPower in this, so are looking at placing the 4 broker QMs plus 1 (or 2 for HA) gateway QMs into a cluster with the broker flow input queues involved (plus matching alias queues on the gateways) becoming cluster queues and their properties set up for load balancing.
A POC looks good.
However, we have one concern to work out. At times a broker may need to be taken out of service for various reasons while its QM needs to remain up, or, rarely, a flow may be having issues on one broker. For DataPower, the former situation is quckly handled by operations and the latter is an automatic with its algorithms. Is there a Best Practice for handling these situations in a load balancing MQ cluster?
We don't want to find a buildup of MQ request messages on the QM of a down broker or broker flow. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu Dec 18, 2014 9:55 am Post subject: Re: Load Balancing cluster queue/QM inhibit question |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| NealM wrote: |
| At times a broker may need to be taken out of service for various reasons while its QM needs to remain up |
Why? If the broker's down, what's the rational for leaving the queue manager running?
| NealM wrote: |
| Is there a Best Practice for handling these situations in a load balancing MQ cluster? |
It's the solution you alude to in the title of this post - you put inhibit the instance of the clustered queue(s) you want not to receive requests, and enable it again when service is restored.
Clearly this is something it's highly desirable to automate through script or PCF-enabled application.
Or you can suspend the QM from the cluster. As long as there is at least 1 other valid target for the messages, the cluster workload mechanism will honour the suspension.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| NealM |
| Posted: Thu Dec 18, 2014 10:49 am Post subject: |
|
|
Master
Joined: 22 Feb 2011
Posts: 230
Location: NC or Utah (depends)
|
Thanks for the quick response Vitor.
Regarding the why of the broker QM remaining up, well, yes, there is little reason for the QM to remain up. However, we do find ourselves with the broker bringup problem, in that we heavily depend on the new embedded global cache, which can take several minutes to re-initialize. Looking at an expected rate of incoming legacy system MQ message requests of 40/sec, just relying on stopping the broker QM isn't acceptable.
But your mention of the cluster suspend (MQCMD_SUSPEND_Q_MGR_CLUSTER) and the corresponding resume appears to be ideal for our most common situation, and looks to be easily scriptable.
So that leaves the rare situation of an individual flow going haywire on one broker. Time for me to read up on events/triggers/Omegamon and see what might make sense with a buildup on a flow input queue. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Dec 18, 2014 11:05 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| NealM wrote: |
| Regarding the why of the broker QM remaining up, well, yes, there is little reason for the QM to remain up. However, we do find ourselves with the broker bringup problem, in that we heavily depend on the new embedded global cache, which can take several minutes to re-initialize. |
Ok, I don't see what the global cache has to do with the queue manager.
| NealM wrote: |
| Looking at an expected rate of incoming legacy system MQ message requests of 40/sec, just relying on stopping the broker QM isn't acceptable. |
Nor do I see the sense in that sentance. If you just stopped the broker's QM (which would after a brief delay cause the broker to stop) then you can rely on the fact that any messages sent to the stopped queue manager would not arrive due to the cluster channel going bye-bye. You'll get a backlog in the SCTQ and that would take a while to clear (or not if this is the last target) but I don't really understand what you're getting at here.
| NealM wrote: |
| But your mention of the cluster suspend (MQCMD_SUSPEND_Q_MGR_CLUSTER) and the corresponding resume appears to be ideal for our most common situation, and looks to be easily scriptable. |
It is.
| NealM wrote: |
| So that leaves the rare situation of an individual flow going haywire on one broker. Time for me to read up on events/triggers/Omegamon and see what might make sense with a buildup on a flow input queue. |
Doing the put inhibit I mentioned and you aluded to.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| NealM |
| Posted: Thu Dec 18, 2014 11:57 am Post subject: |
|
|
Master
Joined: 22 Feb 2011
Posts: 230
Location: NC or Utah (depends)
|
Regarding global cache, no, no effect on the queue manager. But, think of this situation:
When you start up a broker you must also start up its queue manger (pre-IIBv10 anyway). So the QM comes up and, without a QM cluster suspend in place, its cluster queues become available for the legacy system to again put request messages on. An important point that I left out above is that the legacy system needs a reply returned within 7 seconds.
In the meantime, the broker is starting up, with global cache rebuilding containers, catalog servers, etc, and refreshing content from the other brokers (common GC policy file). On a good day that takes close to 120 seconds on our zLinux brokers.
So, 1/4 x 40 x (120 -7) = 1130 failed (expired) messsages.
Since our Operations already know how to check for global cache being operational before adding the broker back to DataPower, adding a QCMD_RESUME_Q_MGR_CLUSTER at that point will be simple enough.
Thanks again |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Dec 18, 2014 4:19 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| NealM wrote: |
So that leaves the rare situation of an individual flow going haywire on one broker. Time for me to read up on events/triggers/Omegamon and see what might make sense with a buildup on a flow input queue. |
NealM,
Take a look at amqsclm.
http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_7.5.0/com.ibm.mq.dev.doc/q024620_.htm
Also, when it comes to suspending a QM from the cluster, the other QMs are only advised to avoid sending to this QM is possible. If the suspended QM hosts the only reachable instance of the queue, it will get the message. Likewise, if the sending app specifically addresses an instance of the queue on the suspended QM, the message will still go to the suspended QM.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| NealM |
| Posted: Thu Dec 18, 2014 5:26 pm Post subject: |
|
|
Master
Joined: 22 Feb 2011
Posts: 230
Location: NC or Utah (depends)
|
Hey Peter, long time! (4 years since our last clustering discussion)
And thanks! At first blush the AMQSCLM sample program looks like it is exactly what we need for the case of individual message flow problems.
Regarding our cluster setup:
Each broker will have exactly the same flows deployed.
Each cluster input queue involved will have the same properties for load balancing
The gateway QM(s) will have only alias queues, and the full repository.
The legacy app will only be sending messages to the gateway QM
...so the only (extremely unlikely) problem that I forsee would be if only 1 broker is up and one of its cluster-involved flows burps. But of course we will test the heck out of this setup before it ever heads to production. |
|
| Back to top |
|
|
| JosephGramig |
| Posted: Mon Dec 29, 2014 6:44 am Post subject: |
|
|
Grand Master
Joined: 09 Feb 2006
Posts: 1244
Location: Gold Coast of Florida, USA
|
NealM,
I would not use inhibit of the cluster queue to control the flow of msgs to it. If, like you say, there is a stream of msgs being sent to it, then most likely between the time you inhibit the queue and the channel sends a batch of msgs that batch will go to the DLQ because the queue is inhibited.
I hit that before, so it will happen. Suspending the Qmgr or removing the object from the cluster is the way to go. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
