ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » Clustering » One of 2 Full repositories being dropped periodically

Post new topic  Reply to topic
 One of 2 Full repositories being dropped periodically « View previous topic :: View next topic » 
Author Message
smeunier
PostPosted: Mon Nov 26, 2018 11:07 am    Post subject: One of 2 Full repositories being dropped periodically Reply with quote

Partisan

Joined: 19 Aug 2002
Posts: 305
Location: Green Mountains of Vermont

Environment

AIX with OS 5.3 Running MQ V6.0.2.9

Before the rants begin with "Upgrade", this is a production environment running a high volume Pub/Sub environment with IBM Extended support for the backlevel version. It is what it is for now..........

At issue, there are three QMGRs in a cluster(A,B,C) Queue manager A and B are the full repositories. They are clustered to allow Broker (QMGR A) gateway access to spread the connection load (QMGR B and C). What has happened (Third instance), is that QMGR B has randomly stopped receiving CLUSTER updates for two SYSTEM.BROKER queues(CONTROL and STREAM). After the 60 days of no receiving update MQ does its cleanup and removes these queues from the QMGR B's repository. Then all hell breaks loose.

On a previous instance, after validating, that the Repository Manager process looks alive and well, channels are active, and the listener port is alive and well, we refreshed the Repository, which resolved the immediate issue. But here we are again with the same issue 30 days later........

Not sure what is happening, but I noticed today that on the other full repository (QMGR A), there was a message AMQ9418 Indicating that is was the only QMGR as a full repository. that when we started receiving message on QMGR B, that these two queues are no longer receiving updates and thus the 60 day messages.

- It looks like QMGR B is being dropped as a Full repository. Not sure how, or what the signal would be for that to occur. Both CLUSSDR and RCVR channels to/from that QMGR are fine. I'm at a loss here. Doing a REFRESH is not a friendly answer, as it is very disruptive to all the applications. So a couple questions:

- Is there a way other than a REFRESH on QMGR B to get it to resubscribe to CLUSTER updates? I have tried to update an attribute of one of the cluster queues to see if it would force a change/update, but to no avail. I see a status change on the SYSTEM.CLUSTER.COMMAND.QUEUE that it has put a message, but never see the same on the other side (QMGR B).

- Would killing the Cluster Repository be a good idea? It seems like this is tightly coupled to the QMGR and may cause issues with shared memory?!

- What else can I look for that would/might be the re-occurring issue?

Right now, my immediate goal, is how to get QMGR B Full repository back in sync again outside of doing a refresh.




Any yes, a move to MQ V9 is on the horizon, after hardware refreshes.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Mon Nov 26, 2018 11:15 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

If you have an extended support agreement with IBM, what do they say?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
Vitor
PostPosted: Mon Nov 26, 2018 1:14 pm    Post subject: Re: One of 2 Full repositories being dropped periodically Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

smeunier wrote:
- Would killing the Cluster Repository be a good idea? It seems like this is tightly coupled to the QMGR and may cause issues with shared memory?!


Heck no! I accept your situation can't get much worse, but killing hostages is a really bad idea. I doubt (due to the shared kernel resources you mention) the queue manager would be happy with that, and you might go from a queue manager that's not in the cluster to a queue manager that won't start without a lot of fiddling by the AIX sysadmins.

smeunier wrote:
- What else can I look for


Your copy of the extended support contract and IBM's phone number.

I agree with my worthy associate - it's PMR time and at last you can justify all the money that you've been spending on extended support all these years.


_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Mon Nov 26, 2018 7:20 pm    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

How many clusters are A, B and C in? (hope you say one)

QMGRA has manually defined Cluster Sender channel(s) to which QMGR(s)?
QMGRB has manually defined Cluster Sender channel(s) to which QMGR(s)?
QMGRC has manually defined Cluster Sender channel(s) to which QMGR(s)?

Does a Display of QMGRA confirm it is still a Full Repository for your cluster?
Does a Display of QMGRB confirm it is still a Full Repository for your cluster?

If you do a minor update to the clustered queue, like changing the Description Field (add a period), do both Full Repositories show that updated description field, only one, or neither?
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
mvic
PostPosted: Thu Dec 20, 2018 8:55 am    Post subject: Re: One of 2 Full repositories being dropped periodically Reply with quote

Jedi

Joined: 09 Mar 2004
Posts: 2080

smeunier wrote:
QMGR B has randomly stopped receiving CLUSTER updates for two SYSTEM.BROKER queues(CONTROL and STREAM)

Why would Qmgr B ever want or receive updates for those qs? Each qmgr has its own queues of these names, and they are not clustered.
Quote:
After the 60 days of no receiving update MQ does its cleanup and removes these queues from the QMGR B's repository.

I assume you're still referring to queues named SYSTEM.*... I don't think the qmgr would remove its knowledge of these queues. They are defined locally to each QM.
Quote:
Then all hell breaks loose.

Presumably you see errors / bad return codes - what are they?
If you are supported at v6 then why not use the latest (6.0.2.12 I think)
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » Clustering » One of 2 Full repositories being dropped periodically
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.