ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » Clustering » cluster xmit queue backup

Post new topic  Reply to topic
 cluster xmit queue backup « View previous topic :: View next topic » 
Author Message
ivanachukapawn
PostPosted: Thu Feb 02, 2012 9:05 am    Post subject: cluster xmit queue backup Reply with quote

Chevalier

Joined: 27 Oct 2003
Posts: 446

I have 4 AIX servers with 1 QM each - MQ 7.0.1.2. These 4 QMs are in a MQ cluster. 2 full and 2 partial. There are approximately 100 applications connected to each QM - there is a lot of distributed queueing going on and traffic is heavy on the cluster xmit queues. Although we have not as yet experienced any problems with this cluster, I am concerned that we may encounter problems in the future with regards to the applications which are starting transactions - problems related to cluster xmit queue backup in the scenario in which a cluster sender gets into an IN-DOUBT status and a cluster xmit queue gets stuck because of a single application's transaction thereby hanging up 99 other applications' messages on the xmit queue. Is this a reasonable concern? If so, should we consider slotting the problematic transaction-queueing applications into their own cluster?
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Feb 02, 2012 9:18 am    Post subject: Re: cluster xmit queue backup Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 21151
Location: Ohio, USA

ivanachukapawn wrote:
Is this a reasonable concern?


How would an application cause a channel to go into an in-doubt status?

If the channel is in doubt, then all traffic to that queue manager is stuck but all the other cluster traffic will continue to move & as with point-to-point you should have procedures to detect & resolve channel problems in the cluster.

I really don't see the question you're asking. Unless you have applications directly queueing transactions in the xmitq and that's a receipe for disaster in any WMQ environment.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
exerk
PostPosted: Thu Feb 02, 2012 9:23 am    Post subject: Re: cluster xmit queue backup Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 5068

ivanachukapawn wrote:
...Is this a reasonable concern? If so, should we consider slotting the problematic transaction-queueing applications into their own cluster?

What good will that do if you use the same queue managers? There'll only be one cluster transmission queue!
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.

Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu Feb 02, 2012 9:38 am    Post subject: Re: cluster xmit queue backup Reply with quote

Jedi Council

Joined: 05 Jan 2008
Posts: 6707
Location: US: west coast, almost.

ivanachukapawn wrote:
... should we consider slotting the problematic transaction-queueing applications into their own cluster?

Yes.

Problematic applications of any type should isolated from applications proven (demonstrated) to be well-behaved and meeting business requirements. This is why we isolate TEST and QA from PROD.
_________________
I'm not paranoid; but the fellow following me is.
I have nothing to hide from the people I trust.

Energizer Bunny arrested! Charged with battery.
Back to top
View user's profile Send private message
ivanachukapawn
PostPosted: Thu Feb 02, 2012 9:50 am    Post subject: Reply with quote

Chevalier

Joined: 27 Oct 2003
Posts: 446

vitor,
Please excuse my badly phrased/conceived question. You stated that
Quote:
If the channel is in doubt, then all traffic to that queue manager is stuck but all the other cluster traffic will continue to move
- I am fuzzy on this concept. If, as Exerk points out (touche Exerk), there is only one cluster xmit queue then how does all the other cluster traffic continue to move (from this QM)?
Bruce,
Only some applications Begin and commit transactions. I arbitrarily typedef's these apps as "problematic" because I supposed that if the cluster sender channel were to get into difficulties if might be because of these apps - I'm probably wrong about that. I meant the term "problematic" to denote apps which used Begin and CMIT
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu Feb 02, 2012 9:59 am    Post subject: Reply with quote

Jedi Council

Joined: 05 Jan 2008
Posts: 6707
Location: US: west coast, almost.

This http://publib.boulder.ibm.com/infocenter/wmqv6/v6r0/index.jsp?topic=%2Fcom.ibm.mq.csqzae.doc%2Fic11350_.htm explains what in-doubt means for a channel. Applications don't cause in-doubt channels.

MQBEGIN and MQCMIT/MQBACK pose no risk to channels - unless a transaction fills the xmit queue or logs. I suppose you should consider this a problematic application.
_________________
I'm not paranoid; but the fellow following me is.
I have nothing to hide from the people I trust.

Energizer Bunny arrested! Charged with battery.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Feb 02, 2012 10:11 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 21151
Location: Ohio, USA

ivanachukapawn wrote:
there is only one cluster xmit queue then how does all the other cluster traffic continue to move (from this QM)?


Because WMQ is written by very clever people.

More technically (and obviously) there's 1 MCA for each cluster sender channel. If one is reporting in-doubt, the others will still be running and moving traffic to the relevant queue managers.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu Feb 02, 2012 10:24 am    Post subject: Reply with quote

Jedi Council

Joined: 05 Jan 2008
Posts: 6707
Location: US: west coast, almost.

Vitor wrote:
... there's 1 MCA for each cluster sender channel. If one is reporting in-doubt, the others will still be running and moving traffic to the relevant queue managers.

And each/every MCA for cluster sender channels monitors the one-and-only cluster transmission queue.

Each MCA will scan the xmit queue for messages destined for cluster queues for its partner cluster receiver channel. A message destined for a queue that exists in multiple qmgrs will not be stranded if at least one qmgr and cluster sender-receiver channel remains.
_________________
I'm not paranoid; but the fellow following me is.
I have nothing to hide from the people I trust.

Energizer Bunny arrested! Charged with battery.
Back to top
View user's profile Send private message
ivanachukapawn
PostPosted: Thu Feb 02, 2012 10:37 am    Post subject: Reply with quote

Chevalier

Joined: 27 Oct 2003
Posts: 446

vitor and Bruce! Thats a spicy meatball! Thank you very much for this help! I believe that I am beginning to comprehend what is going on.
For any partial repository QM in this architecture with say
1000 messages in last 5 minutes destined to the other 3 cluster QMs, how many cluster senders (auto defined I assume) might I expect to find on the QM at one time? I imagine at least 3 cluster senders (one for each of the other QMs) - would there be more than one sender to any one QM?
Back to top
View user's profile Send private message
mqjeff
PostPosted: Thu Feb 02, 2012 10:43 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 13194

There's one cluster sender for each cluster receiver on a given queue manager.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu Feb 02, 2012 11:05 am    Post subject: Reply with quote

Jedi Council

Joined: 05 Jan 2008
Posts: 6707
Location: US: west coast, almost.

Cluster channels come in pairs - just like (non-cluster) sender-receiver channels.
_________________
I'm not paranoid; but the fellow following me is.
I have nothing to hide from the people I trust.

Energizer Bunny arrested! Charged with battery.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Thu Feb 02, 2012 5:27 pm    Post subject: Reply with quote

Jedi Council

Joined: 15 May 2001
Posts: 6857
Location: Hartford CT

Your concern is not so out there.

App A has its queue on all 4 QMs. Queue.A. It reads from this queue.
App B has its queue on all 4 QMs Queue.B. It reads from the queue.
App C sends requests to both A and B and expects a reply.

Right or wrong, App A records every message it reads from the queue in a common DB. That DB goes down. App A goes on strike. App C just keeps sending millions of requests cuz its like a batch app or something. Queue.A starts filling up. Queue.A fills up everywhere.

Depending on your CLUSRCVRs channels Message Retry values, you may find any and all other traffic going thru the common channels severely impacted.

At this point it is not so much the common Cluster Transmit Q that is the issue, its the one common channel from any one QM to the other in the cluster.

You could mitigate by making multiple overlapped or stacked clusters so you have multiple channels and each queue is in its own cluster (Ack!!!). Now one channel message retrying only impacts that cluster and the queues in this cluster. BUT, if that continues for to long, the transmit queue gets deeper, and deeper, and deeper. Eventually the other channels find their select gets by Correl ID out of the transmit queue get slower and slower.

Yeah, if you get two apps, one sending missile launch codes (Critical) and the other one sending horoscopes (eh, waste of electrons), you probably don't want to share infrastructure including an MQ cluster.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message Visit poster's website
Vitor
PostPosted: Fri Feb 03, 2012 5:50 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 21151
Location: Ohio, USA

PeterPotkay wrote:
Yeah, if you get two apps, one sending missile launch codes (Critical) and the other one sending horoscopes (eh, waste of electrons), you probably don't want to share infrastructure including an MQ cluster.


I don't disagree with anything in this post, including the "Ack!!!" of multiple overlapped & stacked clusters. You need to weigh the likelyhood & impact of a failure against the administrative overhead of this setup. Where is overhead includes sticking it all back together at some future point after a less experienced member of staff makes an unfortunate change.

Launch codes and horoscopes? Absolutely. $5m stock trades and month end earning statements? Maybe. Invoices & warehouse orders? Less so.

It's a risk/reward judgement call. The bottom line is that it doesn't immediately jam up as you feared it would.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Fri Feb 03, 2012 9:14 pm    Post subject: Reply with quote

Grand Poobah

Joined: 18 Nov 2003
Posts: 16103
Location: LI,NY

You need to realize that a cluster queue full is a VERY BAD situation in any cluster.
With standard values for retry and retry interval it slows down the communications between the qmgrs in the cluster considerably and yes it will be very noticeable.

Quick fix is to increase the max qdepth of the queue filling up.
This is a VERY temporary fix. Get the application consuming back up, scale it to load.... IMMEDIATELY!

Have fun
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » Clustering » cluster xmit queue backup
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.