ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexGeneral IBM MQ SupportS.C.T.Q. not processing messages for S.I.Q.C.

Post new topicReply to topic
S.C.T.Q. not processing messages for S.I.Q.C. View previous topic :: View next topic
Author Message
tczielke
PostPosted: Wed Mar 07, 2018 7:29 am Post subject: S.C.T.Q. not processing messages for S.I.Q.C. Reply with quote

Yatiri

Joined: 08 Jul 2010
Posts: 691
Location: Illinois, USA

We have an issue where we have two queue managers where messages are very slowly building on the SYSTEM.CLUSTER.TRANSMIT.QUEUE (S.C.T.Q.) and are not being processed. These messages are destined for the SYSTEM.INTER.QMGR.CONTROL (S.I.Q.C.) for queue managers that these two queue managers have connectivity to send the messages to. For whatever reason, these messages just sit on the S.C.T.Q. For one queue manager, the messages have been there for a few months.

Has anyone seen this issue? Or does anyone have any ideas why this behavior is happening?

The two queue managers that are having this issue are at either 7.5.0.6 or 8.0.0.6.

I do have a PMR open, but was just curious if anyone else has seen this issue or has an idea why it is happening.
_________________
MQ administrator since 2010.
Back to top
View user's profile Send private message
tczielke
PostPosted: Wed Mar 07, 2018 9:32 am Post subject: Reply with quote

Yatiri

Joined: 08 Jul 2010
Posts: 691
Location: Illinois, USA

Here are some more details:

This is what is happening from a channel stand point. Lets say it is QM1 that has the messages on the S.C.T.Q. that want to go to QM2 for the S.I.Q.C. queue. QM1 and QM2 are both in cluster X. QM2 actually has two CLUSRCVR channels for cluster X. CLUSRCVR1 is for an internal network and CLUSRCVR2 is for an external network. QM1 is in an internal network and does not have the connectivity to talk to QM2 over CLUSRCVR2, but it can talk to QM2 over CLUSRCVR1. When the issue started, QM1 tried to start up CLUSRCVR2 to send these messages, which it could not use since QM1->QMG2 over CLUSRCVR2 is not allowed. I then got an alert about the CLUSRCVR2 channel on QM1 being in a retrying state, so I stopped that channel. However, QM1 is still not sending the messages to QM2 (for months now for one of the queue managers), even though it has a valid route with CLUSRCVR1 to send the message. This looks like a potential bug to me, but curious if anyone else has seen this behavior.
_________________
MQ administrator since 2010.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Wed Mar 07, 2018 5:08 pm Post subject: Reply with quote

Jedi Council

Joined: 15 May 2001
Posts: 7421

Look at the Correlation ID of the messages waiting in the S.C.T.Q.
Is it the channel name that you have stopped, that does not have the ability to get from QM1 to QM2?

If yes, the messages will sit forever because the other channel that is running ignores those messages. It sees the Correl ID is NOT the name of itself, and thus knows those messages are not for it to move.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
tczielke
PostPosted: Thu Mar 08, 2018 10:56 am Post subject: Reply with quote

Yatiri

Joined: 08 Jul 2010
Posts: 691
Location: Illinois, USA

Thanks, Peter. Yes, this the case. The CorrelId has the channel name of the stopped channel. I am surprised that MQ does not realize in this case that the one channel is stopped and there is another open cluster channel that it can use to send the messages to QM1->QM2.
_________________
MQ administrator since 2010.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Mar 08, 2018 11:01 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 24901
Location: Ohio, USA

tczielke wrote:
I am surprised that MQ does not realize in this case that the one channel is stopped


Why? MQ assumes that channels are self healing and/or will be manually healed in a timely manner.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
tczielke
PostPosted: Thu Mar 08, 2018 11:06 am Post subject: Reply with quote

Yatiri

Joined: 08 Jul 2010
Posts: 691
Location: Illinois, USA

It does not seem intuitive to me. I would think that MQ would realize after a little bit that channel 1 is now no longer available (stopped) and channel 2 is open and going to the same destination so reroute the message through channel 2.
_________________
MQ administrator since 2010.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Mar 08, 2018 11:25 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 24901
Location: Ohio, USA

tczielke wrote:
channel 2 is open and going to the same destination so reroute the message through channel 2.


How would it know it's the same destination? The network routing (which is what the channel MCA knows) must be different as (in your example) it's pointing to an externally hosted network. In MQ terms you typically define 2 different channels over 2 different networks because you want to segregate traffic (high priority over fiber, batch over copper is the classic example) and it might actually cause SLA breach / extra costs if the system "fixes" one problem by dumping everything into one network.

Obviously this doesn't hold if you simply have 2 paths for resiliency (which you seem to have) and if you raised an RFE I'd vote for it. I'd personally settle for a convenient utility to push stranded messages back through the cluster workload algorithm again.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
tczielke
PostPosted: Thu Mar 08, 2018 11:30 am Post subject: Reply with quote

Yatiri

Joined: 08 Jul 2010
Posts: 691
Location: Illinois, USA

Should have said "and channel 2 is open and going to the same queue manager as channel 1, so reroute the message through channel 2"
_________________
MQ administrator since 2010.
Back to top
View user's profile Send private message
mvic
PostPosted: Fri Mar 09, 2018 3:38 pm Post subject: Reply with quote

Jedi

Joined: 09 Mar 2004
Posts: 2009

tczielke wrote:
It does not seem intuitive to me. I would think that MQ would realize after a little bit that channel 1 is now no longer available (stopped) and channel 2 is open and going to the same destination so reroute the message through channel 2.

Having identified channel1 as the route for the message, the local cluster code can change its mind, but only if the message was put via a bind-not-fixed object handle.

Still, I can't immediately say what causes the change of mind to be considered. I think the channel has to be in retry, for that to happen, but I could be wrong.

Anyway, if your message was put via bind-on-open, MQ will never reconsider its first routing decision.

By the way, this config where a CLUSRCVR is advertised by a QM but architecturally cannot be a valid route to that QM, is testing the boundaries of the design and intentions of clustering, IMHO.
Back to top
View user's profile Send private message
tczielke
PostPosted: Fri Mar 09, 2018 7:48 pm Post subject: Reply with quote

Yatiri

Joined: 08 Jul 2010
Posts: 691
Location: Illinois, USA

mvic wrote:
By the way, this config where a CLUSRCVR is advertised by a QM but architecturally cannot be a valid route to that QM, is testing the boundaries of the design and intentions of clustering, IMHO.


Yes, I do now see that this configuration is flawed. I am working on changing it. Unfortunately, when I designed it, it did not test the boundaries of my preconceived notions or vivid imagination.
_________________
MQ administrator since 2010.
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexGeneral IBM MQ SupportS.C.T.Q. not processing messages for S.I.Q.C.
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.