| Author |
Message
|
| ivanachukapawn |
 Posted: Thu Feb 02, 2012 9:05 am Post subject: cluster xmit queue backup |
 |
|
Knight
Joined: 27 Oct 2003
Posts: 561
|
| I have 4 AIX servers with 1 QM each - MQ 7.0.1.2. These 4 QMs are in a MQ cluster. 2 full and 2 partial. There are approximately 100 applications connected to each QM - there is a lot of distributed queueing going on and traffic is heavy on the cluster xmit queues. Although we have not as yet experienced any problems with this cluster, I am concerned that we may encounter problems in the future with regards to the applications which are starting transactions - problems related to cluster xmit queue backup in the scenario in which a cluster sender gets into an IN-DOUBT status and a cluster xmit queue gets stuck because of a single application's transaction thereby hanging up 99 other applications' messages on the xmit queue. Is this a reasonable concern? If so, should we consider slotting the problematic transaction-queueing applications into their own cluster? |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu Feb 02, 2012 9:18 am Post subject: Re: cluster xmit queue backup |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ivanachukapawn wrote: |
| Is this a reasonable concern? |
How would an application cause a channel to go into an in-doubt status?
If the channel is in doubt, then all traffic to that queue manager is stuck but all the other cluster traffic will continue to move & as with point-to-point you should have procedures to detect & resolve channel problems in the cluster.
I really don't see the question you're asking. Unless you have applications directly queueing transactions in the xmitq and that's a receipe for disaster in any WMQ environment.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| exerk |
| Posted: Thu Feb 02, 2012 9:23 am Post subject: Re: cluster xmit queue backup |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| ivanachukapawn wrote: |
| ...Is this a reasonable concern? If so, should we consider slotting the problematic transaction-queueing applications into their own cluster? |
What good will that do if you use the same queue managers? There'll only be one cluster transmission queue!
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Feb 02, 2012 9:38 am Post subject: Re: cluster xmit queue backup |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| ivanachukapawn wrote: |
| ... should we consider slotting the problematic transaction-queueing applications into their own cluster? |
Yes.
Problematic applications of any type should isolated from applications proven (demonstrated) to be well-behaved and meeting business requirements. This is why we isolate TEST and QA from PROD.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| ivanachukapawn |
| Posted: Thu Feb 02, 2012 9:50 am Post subject: |
|
|
Knight
Joined: 27 Oct 2003
Posts: 561
|
vitor,
Please excuse my badly phrased/conceived question. You stated that
| Quote: |
| If the channel is in doubt, then all traffic to that queue manager is stuck but all the other cluster traffic will continue to move |
- I am fuzzy on this concept. If, as Exerk points out (touche Exerk), there is only one cluster xmit queue then how does all the other cluster traffic continue to move (from this QM)?
Bruce,
Only some applications Begin and commit transactions. I arbitrarily typedef's these apps as "problematic" because I supposed that if the cluster sender channel were to get into difficulties if might be because of these apps - I'm probably wrong about that. I meant the term "problematic" to denote apps which used Begin and CMIT |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Feb 02, 2012 9:59 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Feb 02, 2012 10:11 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ivanachukapawn wrote: |
| there is only one cluster xmit queue then how does all the other cluster traffic continue to move (from this QM)? |
Because WMQ is written by very clever people.
More technically (and obviously) there's 1 MCA for each cluster sender channel. If one is reporting in-doubt, the others will still be running and moving traffic to the relevant queue managers.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Feb 02, 2012 10:24 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Vitor wrote: |
| ... there's 1 MCA for each cluster sender channel. If one is reporting in-doubt, the others will still be running and moving traffic to the relevant queue managers. |
And each/every MCA for cluster sender channels monitors the one-and-only cluster transmission queue.
Each MCA will scan the xmit queue for messages destined for cluster queues for its partner cluster receiver channel. A message destined for a queue that exists in multiple qmgrs will not be stranded if at least one qmgr and cluster sender-receiver channel remains.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| ivanachukapawn |
| Posted: Thu Feb 02, 2012 10:37 am Post subject: |
|
|
Knight
Joined: 27 Oct 2003
Posts: 561
|
vitor and Bruce! Thats a spicy meatball! Thank you very much for this help! I believe that I am beginning to comprehend what is going on.
For any partial repository QM in this architecture with say
1000 messages in last 5 minutes destined to the other 3 cluster QMs, how many cluster senders (auto defined I assume) might I expect to find on the QM at one time? I imagine at least 3 cluster senders (one for each of the other QMs) - would there be more than one sender to any one QM? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Feb 02, 2012 10:43 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| There's one cluster sender for each cluster receiver on a given queue manager. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Feb 02, 2012 11:05 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Cluster channels come in pairs - just like (non-cluster) sender-receiver channels.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Feb 02, 2012 5:27 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Your concern is not so out there.
App A has its queue on all 4 QMs. Queue.A. It reads from this queue.
App B has its queue on all 4 QMs Queue.B. It reads from the queue.
App C sends requests to both A and B and expects a reply.
Right or wrong, App A records every message it reads from the queue in a common DB. That DB goes down. App A goes on strike. App C just keeps sending millions of requests cuz its like a batch app or something. Queue.A starts filling up. Queue.A fills up everywhere.
Depending on your CLUSRCVRs channels Message Retry values, you may find any and all other traffic going thru the common channels severely impacted.
At this point it is not so much the common Cluster Transmit Q that is the issue, its the one common channel from any one QM to the other in the cluster.
You could mitigate by making multiple overlapped or stacked clusters so you have multiple channels and each queue is in its own cluster (Ack!!!). Now one channel message retrying only impacts that cluster and the queues in this cluster. BUT, if that continues for to long, the transmit queue gets deeper, and deeper, and deeper. Eventually the other channels find their select gets by Correl ID out of the transmit queue get slower and slower.
Yeah, if you get two apps, one sending missile launch codes (Critical) and the other one sending horoscopes (eh, waste of electrons), you probably don't want to share infrastructure including an MQ cluster.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Feb 03, 2012 5:50 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| PeterPotkay wrote: |
| Yeah, if you get two apps, one sending missile launch codes (Critical) and the other one sending horoscopes (eh, waste of electrons), you probably don't want to share infrastructure including an MQ cluster. |
I don't disagree with anything in this post, including the "Ack!!!" of multiple overlapped & stacked clusters. You need to weigh the likelyhood & impact of a failure against the administrative overhead of this setup. Where is overhead includes sticking it all back together at some future point after a less experienced member of staff makes an unfortunate change.
Launch codes and horoscopes? Absolutely. $5m stock trades and month end earning statements? Maybe. Invoices & warehouse orders? Less so.
It's a risk/reward judgement call. The bottom line is that it doesn't immediately jam up as you feared it would.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Feb 03, 2012 9:14 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
You need to realize that a cluster queue full is a VERY BAD situation in any cluster.
With standard values for retry and retry interval it slows down the communications between the qmgrs in the cluster considerably and yes it will be very noticeable.
Quick fix is to increase the max qdepth of the queue filling up.
This is a VERY temporary fix. Get the application consuming back up, scale it to load.... IMMEDIATELY!
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
