ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexClusteringWhy are messages piling up in my cluster transmit queue?

Post new topicReply to topic
Why are messages piling up in my cluster transmit queue? View previous topic :: View next topic
Author Message
issac
PostPosted: Mon Mar 05, 2018 9:32 pm Post subject: Why are messages piling up in my cluster transmit queue? Reply with quote

Disciple

Joined: 02 Oct 2008
Posts: 152
Location: Shanghai

Hi, experts.

Its WMQ V7.1.0.8 on AIX.
The scenario is a bit more complex than what the title suggests. It happens in 1 cluster and 3 outside qmgrs, involving bellow qmgrs.

These are within 1 cluster CLUSTER1.
MQPUB
MQGW6

These are outside the cluster:
MQ2
MQ3
MQ4

In MQPUB I define TOPIC and SUB objects, for example:
DEFINE TOPIC(TOPIC.MQ2) +
TOPICSTR('TOPICSTR.MQ2') +
DESCR('') +
REPLACE

DEFINE SUB(SUB.MQ2.GW6) +
TOPICSTR('TOPICSTR.MQ2') +
DESTQMGR(MQGW6) +
DEST(REMOTEQ.SUB.MQ2) +
REPLACE

In MQGW6, I define the remoteq
DEFINE REMOTEQ(REMOTEQ.SUB.MQ2) RNAME(LOCALQ.SUB) RQMNAME(MQ2) CLUSTER(CLUSTER1) XMITQ(MQ2)

Theres a pair of SDR-RCVR channels MQGW6.MQ2 connecting MQGW6 to MQ2

The same setting exists for MQ3, MQ4, that is, therere defines like this:
On MQPUB:
1. TOPIC: TOPIC.MQ3
2. SUB: SUB.MQ3.GW6

On MQGW6
1. REMOTEQ: REMOTEQ.SUB.MQ3
2. CHANNEL: MQGW6.MQ3

On MQPUB, therere constantly messages published to all the topics. The traffic is quite a lot, like 200,000 an hour, equally to all 3 destination qmgrs, namely MQ2, MQ3, MQ4.

Here comes the problem: One day, on MQGW6, channel MQGW6.MQ2 becomes RETRYING. But its fine, we know that the host of MQ2 is under maintenance.

Then I noticed deadq piling up on MQGW6, whose reason code is 0x803, MQRC_PUT_INHIBITED. I thought it should be expected, as when channel is RETRYING, XMITQ is set to PUT(DISABLED) automatically by MQ. As MQPUB attempts to put message to a remoteq, the xmitq is PUT(DISABLED), it should be reasonable that messages are moved to the deadq due to PUT_INHIBITED.

Then we noticed the message accumulation in SYSTEM.CLUSTER.TRANSMIT.QUEUE on MQPUB. There were about 100,000 messages accumulated there. The depth decreases at a very slow rate when no message is being sent. Here comes my first question: Q1: Why does message pile up on cluster transmitq in MQPUB? Ive verified that messages eventually end up in the deadq of MQGW6, what does MQPUB have to do with the whole process?

I browsed the accumulated messages in MQPUB, discovering that it includes messages destined for all MQ2, MQ3, MQ4, not just MQ2. Heres my second question: Q2: why are messages for MQ3 and MQ4 affected? The only retrying channel is MQGW6.MQ2.

It appears that the MQ network discussed here isnt robust enough. One channel failure to lead to a whole system performance degration. So Q3: Should this be considered a MQ network design problem? Do I need to alter the design for some improvement?

Thanks in advance!
_________________
Bazinga!
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Tue Mar 06, 2018 9:05 pm Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20106
Location: LI,NY

I'd say no. Your first order of business is find out why the messages are in the DLQ. If it is because the queue is put inhibited, find which of your subscription queues is put inhibited?

Note that any operation slowing down a channel (destination queue full, put inhibited etc...) will dramatically slow down comms between the source qmgr and target qmgr. So you can have a single queue being full impact all traffic including traffic that has nothing to do with the full target queue.

If you want to reduce that impact you might want to look in short and long retry intervals on the cluster channels.

Have fun
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
mvic
PostPosted: Fri Mar 09, 2018 3:56 pm Post subject: Re: Why are messages piling up in my cluster transmit queue? Reply with quote

Jedi

Joined: 09 Mar 2004
Posts: 2078

issac wrote:
Then I noticed deadq piling up on MQGW6, whose reason code is 0x803, MQRC_PUT_INHIBITED. I thought it should be expected, as when channel is RETRYING, XMITQ is set to PUT(DISABLED) automatically by MQ. As MQPUB attempts to put message to a remoteq, the xmitq is PUT(DISABLED), it should be reasonable that messages are moved to the deadq due to PUT_INHIBITED.

This does not solve your issue, but I wanted to say that the above comments do not ring true to me.
Maybe you are thinking of this link? https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.0.0/com.ibm.mq.con.doc/q015630_.htm
But it only talks about xmitqs becoming GET-disabled.
The SCTQ is different, as it is used by many channels, not only one as in the case of non-clustering DQM.
The SCTQ would also never be PUT-disabled on the basis that one channel had a problem. For the sake of one channel, then all channels using the SCTQ would be made to suffer - this does not happen.

Turning to your issue:
1.is the destination queue a clustered queue - the SCTQ can be used for messages bound for queues that are not known, if the MQOPEN specifies a qmgr name that is known in the cluster.
2.or, perhaps did one of your destination clustered queues become put-disabled AFTER the message was put to the SCTQ?

Knowledge of put-disabled on clustered queues is supposed to flow to all interested queue managers, so they can avoid choosing those queues as a destination for your message.
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexClusteringWhy are messages piling up in my cluster transmit queue?
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.