ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » Clustering » Mother of All Cluster Problems

Post new topic  Reply to topic Goto page Previous  1, 2
 Mother of All Cluster Problems « View previous topic :: View next topic » 
Author Message
belchman
PostPosted: Mon May 21, 2018 3:46 am    Post subject: Reply with quote

Partisan

Joined: 31 Mar 2006
Posts: 386
Location: Ohio, USA

Bruce2359,

I discovered that 2 FP limit in the MQ doc and was surprised so I asked IBM via an ESR and they say the doc is incorrect and they are fixing it.

I can find the ESR if necessary. I hope it is a fact and not just someone "closing the ticket as quickly as possible"
_________________
Make three correct guesses consecutively and you will establish a reputation as an expert. ~ Laurence J. Peter
Back to top
View user's profile Send private message
belchman
PostPosted: Mon May 21, 2018 3:49 am    Post subject: Reply with quote

Partisan

Joined: 31 Mar 2006
Posts: 386
Location: Ohio, USA

IN RE to the MQP1 full repository thread, as far as I know, MQP1 was always an FR and was always one of the 2 FRs. But that was all done before I was around.
_________________
Make three correct guesses consecutively and you will establish a reputation as an expert. ~ Laurence J. Peter
Back to top
View user's profile Send private message
belchman
PostPosted: Mon May 21, 2018 3:57 am    Post subject: Reply with quote

Partisan

Joined: 31 Mar 2006
Posts: 386
Location: Ohio, USA

MVIC,

We do have a DR once a year and we just had one. I just said DR was not the issue because this has been going on for nearly 2 years. I will run down the seqnum idea though.

For the DR, we (mainframe MQ folks) spin up a copy of MQP1 on another IP node and on that system, the two full repositories are different IP nodes and the name of the DR cluster as a DR affinity in it (so different cluster) .

However, I will look at that seqnum idea because it does seem feasible.
_________________
Make three correct guesses consecutively and you will establish a reputation as an expert. ~ Laurence J. Peter
Back to top
View user's profile Send private message
belchman
PostPosted: Mon May 21, 2018 4:06 am    Post subject: Reply with quote

Partisan

Joined: 31 Mar 2006
Posts: 386
Location: Ohio, USA

gbaddeley,

I thought of a bug being the culprit as well. To eliminate IT12700 I asked the mainframe folks what patch level they are at. The reply was PUT1711/RSU1712.

I have not checked if that includes a fix to IT12700.

The other FR and the only PR is v8.0.0.8.

Due to the fact that we have a cluster with 9 FR and 1 PR, and others that are wrong but not as bad, it is almost impossible to not have version mismatches. I inherited this problem. I did not create it.

But this particular cluster has 2 FR and 1 PR and the AIX nodes are MQv8.0.0.8 and the z/OS node is PUT1711/RSU1712.
_________________
Make three correct guesses consecutively and you will establish a reputation as an expert. ~ Laurence J. Peter
Back to top
View user's profile Send private message
bruce2359
PostPosted: Mon May 21, 2018 5:17 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

belchman wrote:
Bruce2359,

I discovered that 2 FP limit in the MQ doc and was surprised so I asked IBM via an ESR and they say the doc is incorrect and they are fixing it.

I can find the ESR if necessary. I hope it is a fact and not just someone "closing the ticket as quickly as possible"

What doc? Please cite the URL.

The doc is incorrect as to what, specifically? What are they fixing?

Sorry if I wasn't precise. You can alter the REPOS (or REPOSNL) qmgr attribute all of your qmgrs can be FRs. The issue is: which of them will MQ clustering software use as FRs?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
belchman
PostPosted: Mon May 21, 2018 7:28 am    Post subject: Reply with quote

Partisan

Joined: 31 Mar 2006
Posts: 386
Location: Ohio, USA

In my ESR, I asked IBM this

Problem description and business impact
I found this sentence "Details of any updates to cluster resources made by a queue manager in a cluster; for example, clustered queues, are sent from that queue manager to two full repositories at most in that cluster " in the document https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_7.5.0/com.ibm.mq.con.doc/q004760_.htm#q004760___perf.

Is that sentence accurate? If I have a cluster with more than 2 full repositories, will cluster updates only be sent to 2 of the repositories? If yes, can you provide the algorithm MQ uses to determine which 2 repositories get the update?


Their last response was this

The page is incorrect. I have sent a message to the Knowledge Center
team to fix it. Any update to a cluster queue will be sent to all full
repositories in a cluster. IBM M L3 cluster experts would argue that
there is no reason to ever have more then 2 full repositories in a
cluster. The same engineers would also argue that there should never be
less then two full repositories in a cluster. It is perfectly supported
to have as many as you like but the general consensus among IBM MQ
clustering experts is that two is the magic number.

_________________
Make three correct guesses consecutively and you will establish a reputation as an expert. ~ Laurence J. Peter
Back to top
View user's profile Send private message
bruce2359
PostPosted: Mon May 21, 2018 8:53 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

The below quote is also from in the document you cited in your ESR https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_7.5.0/com.ibm.mq.con.doc/q004760_.htm#q004760___perf.

Quote:
Unless there is a good reason for using a third repository, such as using a geographically local full repository for geographical reasons, use the two repository design. Having three full repositories means that you never know which are the two that are currently in use, and there might be administrative problems caused by interactions between multiple workload management parameters. It is not recommend to have more than two full repositories.

and this:
Quote:
You must fully interconnect all the full repository queue managers with manually defined cluster sender channels. Particular care must be taken when the cluster does have, for some justifiable reason, more than two full repositories. In this situation it is often possible to miss one or more channels and for it not to be immediately apparent. When full interconnection does not occur, hard to diagnose problems often arise. They are hard to diagnose because some full repositories not holding all repository data and therefore resulting in queue managers in the cluster having different views of the cluster depending on the full repositories that they connect to.


This addresses my comment about MQ clustering software using at most two FRs.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
mvic
PostPosted: Mon May 21, 2018 11:47 am    Post subject: Reply with quote

Jedi

Joined: 09 Mar 2004
Posts: 2080

When a PR qmgr uses a clustered queue for the first time, it will make subscriptions to FRs, and later it will renew these occasionally.
If there is only one FR, then of course it makes 1 subscription.
If there is are 2 FRs, then it makes 2 subscriptions.
If there is are 3 FRs, then it still makes 2 subscriptions.

It's not easy (maybe it's impossible) to predict which 2 of the 3 will be chosen. I guess this is the reason for the statement on that page: "Having three full repositories means that you never know which are the two that are currently in use"

Now, about knowledge of the queues, this is indeed forwarded around all FRs. If there are 3 FRs, it is forwarded around all 3.

But still a PR only makes a subscription to 2 FRs.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Mon May 21, 2018 4:38 pm    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

Ever seen the advice for ‘Best Practice’ in a WMQ cluster to only have two Full Repositories and wondered why? This post is for you.

WMQ Clusters: Why only two Full Repositories?
https://www.ibm.com/developerworks/community/blogs/messaging/entry/wmq_clusters_why_only_two_full_repositories?lang=en_us
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
belchman
PostPosted: Wed Jun 20, 2018 3:54 am    Post subject: Reply with quote

Partisan

Joined: 31 Mar 2006
Posts: 386
Location: Ohio, USA

It appears we may have resolved our immediate issue. We created a new explicit cluster sender channel on MQP1 to the repository on the open systems queue manager and that appears to have worked.
_________________
Make three correct guesses consecutively and you will establish a reputation as an expert. ~ Laurence J. Peter
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Wed Jun 20, 2018 7:21 pm    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

belchman wrote:
It appears we may have resolved our immediate issue. We created a new explicit cluster sender channel on MQP1 to the repository on the open systems queue manager and that appears to have worked.

Sure but did you create a channel back?
Remember each FR MUST have a manually defined cluster sender channel to EACH of the other FRs in the cluster.
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
belchman
PostPosted: Thu Jun 21, 2018 2:37 am    Post subject: Reply with quote

Partisan

Joined: 31 Mar 2006
Posts: 386
Location: Ohio, USA

fjb_saper,

Yes, the full repos on the AIX host already had a sender to the full repos MQP1 that worked just fine. We just created another sender on MQP1 to the full repos on AIX.

I have no idea why the existing sender on MQP1 stopped working. Its not my purview to worry about MQP1 until it cause open systems business processes to malfunction.

Thanks everyone for your assistance on this.
_________________
Make three correct guesses consecutively and you will establish a reputation as an expert. ~ Laurence J. Peter
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu Jun 21, 2018 5:16 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

belchman wrote:
fjb_saper,

Yes, the full repos on the AIX host already had a sender to the full repos MQP1 that worked just fine. We just created another sender on MQP1 to the full repos on AIX.

I have no idea why the existing sender on MQP1 stopped working. Its not my purview to worry about MQP1 until it cause open systems business processes to malfunction.

Thanks everyone for your assistance on this.

Did you mean CLUSSDR?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
belchman
PostPosted: Thu Jun 21, 2018 5:19 am    Post subject: Reply with quote

Partisan

Joined: 31 Mar 2006
Posts: 386
Location: Ohio, USA

bruce2359,

Yes, I am talking about cluster senders.
_________________
Make three correct guesses consecutively and you will establish a reputation as an expert. ~ Laurence J. Peter
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page Previous  1, 2 Page 2 of 2

MQSeries.net Forum Index » Clustering » Mother of All Cluster Problems
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.