| Author |
Message
|
| svu |
 Posted: Sat Jan 22, 2011 3:36 pm Post subject: Enforce QM to forget everything about cluster |
 |
|
Voyager
Joined: 30 Jan 2006
Posts: 99
|
Hi all
There is a QM that was full cluster repo at some point. A couple of days ago it had to be demoted to partial repo (2 new full repos created instead of the old ones). But unfortunately, because of messing with cluster receiver channels, this QM got "lost" - it fell out of cluster .... and I cannot add it. The cluster still exists (with 2 new full repos), works reasonably ok.
But I cannot add that old QM to the cluster any more. WMQ Explorer performs all necessary actions: creates cluster sender/receiver channels, resumes the cluster - but the QM does not appear in any lists.
What makes me suspicious is that that QM keeps 2 records in clusqmgr - about itself and other QM that used to be 2nd full repo - and I cannot delete those 2 records, even with RESET CLUSTER.
Also, according to WMQ Explorer, that QM considers itself as partial cluster repository, even when I remove all channels, do suspend cluster etc.
So, I guess the only way to add QM to the cluster again is to ask it to forget everything about its "past". How could I do that? How can I convince the QM that it is not the partial repo any more (so that could make it partial repo again)?
Thanks for any hints |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Sat Jan 22, 2011 4:06 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9470
Location: US: west coast, almost. Otherwise, enroute.
|
Exactly how did you issue the RESET CLUSTER command? Exactly what command did you issue? And where did you issue the command?
| Quote: |
| But unfortunately, because of messing with cluster receiver channels, this QM got "lost" - it fell out of cluster .... and I cannot add it. |
How did you mess up cluster receiver channels? On which qmgr? Why did you mess up the cluster receiver channels?
The WMQ Cluster Queue Managers manual describes how, where, and under what circumstances to issue the cluster commands.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| svu |
| Posted: Sat Jan 22, 2011 4:46 pm Post subject: |
|
|
Voyager
Joined: 30 Jan 2006
Posts: 99
|
| bruce2359 wrote: |
| Exactly how did you issue the RESET CLUSTER command? Exactly what command did you issue? And where did you issue the command? |
I am running on QM1
reset cluster(<clustername>) ACTION(FORCEREMOVE) QMNAME('QM2') QUEUES(YES)
but it does not work - it still shows me the information about the QM2 (that was his peer Full Repo, before the change). I guess that is because QM1 has REPOS(' ')
| Quote: |
| How did you mess up cluster receiver channels? On which qmgr? Why did you mess up the cluster receiver channels? |
By mistake, the cluster receiver channel was removed on QM1 (these things happen when the script is written by one man, run by another man against wrong QM)
| Quote: |
| The WMQ Cluster Queue Managers manual describes how, where, and under what circumstances to issue the cluster commands. |
Thanks, I know. Errare humanum est. Now the question is how to recover, without recreating QM |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sat Jan 22, 2011 5:10 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9470
Location: US: west coast, almost. Otherwise, enroute.
|
I/we need your posts to be much more technical specific.
| Quote: |
| What makes me suspicious is that that QM keeps 2 records in clusqmgr - about itself and other QM that used to be 2nd full repo - and I cannot delete those 2 records, even with RESET CLUSTER. |
Records? Do you mean that you've browsed the SYSTEM.CLUSTER.REPOSITORY.QUEUE on the defective PR qmgr?
| Quote: |
| But I cannot add that old QM to the cluster any more. |
You don't ADD qmgrs to a cluster; rather you DEFINE objects to the cluster. So, exactly did you try? What were the results?
In order to be a qmgr in a cluster, the qmgr must have a clusrcvr channel to itself, AND a clussdr channel to an FR. What is the status of the cluster channels as observed from both the FR and PR?
As you work through this, you might want to use MQSC commands, and post the results here.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Jan 22, 2011 11:49 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Read up on the effect of the REFRESH CLUSTER command.
If all your objects are defined right a refresh cluster(clustername) command issued on the PR in question should allow it to rejoin the cluster.
And as bruce2359 said... IT IS ALL IN THE CLUSTER MANUAL.  
 WMQ Explorer is not the best way to administer a cluster especially in these circumstances. Use either runmqsc or mqsc (mo72)
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| svu |
| Posted: Sun Jan 23, 2011 12:37 am Post subject: |
|
|
Voyager
Joined: 30 Jan 2006
Posts: 99
|
| bruce2359 wrote: |
| Records? Do you mean that you've browsed the SYSTEM.CLUSTER.REPOSITORY.QUEUE on the defective PR qmgr? |
No, I did DISPLAY CLUSQMGR - it shows 2 QMs
| Quote: |
| You don't ADD qmgrs to a cluster; rather you DEFINE objects to the cluster. So, exactly did you try? What were the results? |
I tried two ways: with GUI (WMQ Explorer), with MQSC: DEFINEd two channels (cluster sender/receiver), RESUMEd QM in the cluster. Neither way effectively adds QM to the cluster
| Quote: |
| What is the status of the cluster channels as observed from both the FR and PR? |
On both PR and FR, all 4 channels are Inactive. If I switch them to "running" state - they do run.
| Quote: |
| As you work through this, you might want to use MQSC commands, and post the results here. |
I can do that, just give me the commands  |
|
| Back to top |
|
|
| svu |
| Posted: Sun Jan 23, 2011 12:41 am Post subject: |
|
|
Voyager
Joined: 30 Jan 2006
Posts: 99
|
| fjb_saper wrote: |
| Read up on the effect of the REFRESH CLUSTER command. |
I did. And I run the command. No luck so far. It seems PR does not recognize existing FR for some reason - refusing to load the info, even though the channels are fine.
| Quote: |
| If all your objects are defined right a refresh cluster(clustername) command issued on the PR in question should allow it to rejoin the cluster. |
I hope so. Perhaps my cluster channels are wrong in some way - I would like to know which way...
| Quote: |
| And as bruce2359 said... IT IS ALL IN THE CLUSTER MANUAL. |
I read it. Perhaps I did not get something important from it (other than obvious "do not mess with clusters")
| Quote: |
| WMQ Explorer is not the best way to administer a cluster especially in these circumstances. Use either runmqsc or mqsc (mo72) |
I try both.
What puzzles me is that even after I do RESUME CLUSTER the PR QM thinks of itself as ... suspended(!) According to DISPLAY CLUSQMGR(QM1): SUSPEND(YES). Perhaps if I manage to kick it out of this state, that would resolve my problem.
At the same time, SYSTEM.CLUSTER.COMMAND.QUEUE has a number of messages. The earliest is related to the day when that all started. |
|
| Back to top |
|
|
| svu |
| Posted: Sun Jan 23, 2011 2:46 am Post subject: |
|
|
Voyager
Joined: 30 Jan 2006
Posts: 99
|
| YESSS!!!! amqrrmfa was not running for some reason. I restarted QM, that process appeared, command queue is empty now - and I can see the QM in the cluster! Thank to everybody! |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sun Jan 23, 2011 7:12 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You probably want to follow up with applying maintenance.
amqrrmfa should be running pretty much all of the time, and if it's crashed at some point that's a defect.
If, however, you're actually at 7.0.1.3, then you could consider opening a PMR. If you can gather enough information about the system at the time that amqrrmfa crashed, that is... |
|
| Back to top |
|
|
| svu |
| Posted: Sun Jan 23, 2011 8:52 am Post subject: |
|
|
Voyager
Joined: 30 Jan 2006
Posts: 99
|
| Well, it is still 6.something:) |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sun Jan 23, 2011 9:33 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9470
Location: US: west coast, almost. Otherwise, enroute.
|
Re-reading this entire post, I gather that you have not been formally trained (attended a basic WMQ System Administration class).
IBM offers WM20 WMQ System Administration for midrange platforms (UNIX, Windoze). This 4-day lecture and hands-on lab exercises class covers the fundamentals you will need to do basic admin, AND basic problem- determination. This course includes lecture and lab exercises with qmgr clusters.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sun Jan 23, 2011 11:13 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| svu wrote: |
| Well, it is still 6.something:) |
And what are you going to do in October, when v6.0.1 goes out of support? |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sun Jan 23, 2011 11:34 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9470
Location: US: west coast, almost. Otherwise, enroute.
|
It seems to me that the issue here was a sysadmin without fundamental WMQ skills.
Sysadmins were breaking clusters at v6 through lack of experience with WMQ, lack of problem-determination experience, failure to understand how clusters work, and failure to follow the Advanced Tasks section of the WMQ Clusters manual.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| svu |
| Posted: Mon Jan 24, 2011 1:08 am Post subject: |
|
|
Voyager
Joined: 30 Jan 2006
Posts: 99
|
| mqjeff wrote: |
| svu wrote: |
| Well, it is still 6.something:) |
And what are you going to do in October, when v6.0.1 goes out of support? |
Upgrade, obviously |
|
| Back to top |
|
|
| svu |
| Posted: Mon Jan 24, 2011 1:12 am Post subject: |
|
|
Voyager
Joined: 30 Jan 2006
Posts: 99
|
| bruce2359 wrote: |
It seems to me that the issue here was a sysadmin without fundamental WMQ skills.
Sysadmins were breaking clusters at v6 through lack of experience with WMQ, lack of problem-determination experience, failure to understand how clusters work, and failure to follow the Advanced Tasks section of the WMQ Clusters manual. |
Thanks for the in-depth analysis. It is even simpler than that... |
|
| Back to top |
|
|
|
|
