| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| Queue Manager missing SYSTEM.CLUSTER.* queues... |
« View previous topic :: View next topic » |
| Author |
Message
|
| Allan Mackay |
 Posted: Fri Aug 27, 2010 6:35 am Post subject: Queue Manager missing SYSTEM.CLUSTER.* queues... |
 |
|
Apprentice
Joined: 08 Jul 2010
Posts: 29
|
Hi,
I'm looking to cluster two queue managers (1 on zLinux, 1 on z/OS). This is something I've had some practice (and success) with to date for other pairs of zLinux/zOS QMs. However, the z/OS queue manager I want to cluster (as a full repository) doesn't have the SYSTEM.CLUSTER.* queues defined for some reason!
Can I just manually define them? If I do, is there anything else that I should check to see is also not missing? Would I need to bounce the QM for it to realise that it could now deal with clustering?
You can probably guess that the reason I'm asking this is that I did manually define the 3 missing queues (COMMAND.QUEUE, REPOSITORY.QUEUE and TRANSMIT.QUEUE) but am having the devils own job replicating a cluster that works on 2 other QM pairs. The zLinux side works (I think), in so far that the REPOSITORY.QUEUE gets populated. The CLUSRCVR and CLUSSDR channels are communicating, although the z/OS CLUSSDR didn't start automatically. I have checked and double checked the definitions... all seem to be correct.
If I remove the cluster repository name from the zLinux QM, the CLUSSDR will disconnect after 10 minutes as it should. However, if I remove the cluster repository name from the z/OS queue manager, the CLUSSDR continues running forever... This behaviour is making me wonder if there is something else missing or whether the QM simply needs a bounce (I say simply but it's a production QM in constant use!).
Any thoughts or help would be greatly appreciated here.
Cheers,
-Al. |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Fri Aug 27, 2010 6:40 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Look at the INP2 concatenation in your qmgr proc. Is one (or more) of the DD statements commented out?
I'd suspect that the DD that points to the member that creates the cluster objects was commented out.
Un-comment the DD statements, stop and restart the qmgr.
[edit]
If you cannot restart the qmgr, you can recreate the cluster objects with the CSQUTIL batch job to process the INP2 member that contains the cluster objects.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Allan Mackay |
| Posted: Fri Aug 27, 2010 6:57 am Post subject: |
|
|
Apprentice
Joined: 08 Jul 2010
Posts: 29
|
Aye, thanks for that... that's pretty much what I had already done. The only two other objects in that member were for distributed queuing (SYSTEM.CHANNEL.SYNCQ and SYSTEM.CHANNEL.INITQ) which are defined in the QM.
I should have added that the SYSTEM.CLUSTER.COMMAND.QUEUE on z/OS has messages on it, placed there by the z/OS QM itself and the zLinux QM which I have clustered with the z/OS one. That queue is not emptying nor is anything appearing on the REPOSITORY.QUEUE... I'm flummoxed 
Cheers
-Al. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Aug 27, 2010 7:00 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Have you successfully defined the CLUSSDR/CLUSRCVR channels on both ends?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Allan Mackay |
| Posted: Fri Aug 27, 2010 7:05 am Post subject: |
|
|
Apprentice
Joined: 08 Jul 2010
Posts: 29
|
| Yep, and as mentioned, double-checked that their definitions are correct absolutely correct. Incidentally, stopping the CLUSSDR on each side does stop the corresponding CLUSRCVR on the other side. And restarting, starts the other... |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Aug 27, 2010 7:20 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Let's back up a moment...
How did you discover that some of the cluster objects were missing? An error message when you attempted to define cluster objects?
| Quote: |
| incidentally, stopping the CLUSSDR on each side does stop the corresponding CLUSRCVR on the other side. And restarting, starts the other... |
Works the same way on SDR/RCVR channels, as well.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Aug 27, 2010 7:21 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
A cluster qmgr missing the SYSTEM.CLUSTER queues is in an abysmal state.
Just defining the queues will not help. You need also the cluster repository process to run and check for the input count on the SYSTEM.CLUSTER queues.
My guess is talk with your system admin. Make sure all is set up correctly to enable cluster operations and then bounce the qmgr.
Now you need to look at translating that to zOS terminology... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Allan Mackay |
| Posted: Sat Aug 28, 2010 1:57 pm Post subject: |
|
|
Apprentice
Joined: 08 Jul 2010
Posts: 29
|
| bruce2359 wrote: |
Let's back up a moment...
How did you discover that some of the cluster objects were missing? An error message when you attempted to define cluster objects?
| Quote: |
| incidentally, stopping the CLUSSDR on each side does stop the corresponding CLUSRCVR on the other side. And restarting, starts the other... |
Works the same way on SDR/RCVR channels, as well. |
Yes, I got a missing queue message when I started defining the cluster bits on the mainframe.
I mentioned the starting/stopping only to illustrate that the definitions appear to be correct... |
|
| Back to top |
|
|
| Allan Mackay |
| Posted: Sat Aug 28, 2010 2:02 pm Post subject: |
|
|
Apprentice
Joined: 08 Jul 2010
Posts: 29
|
| fjb_saper wrote: |
A cluster qmgr missing the SYSTEM.CLUSTER queues is in an abysmal state.
Just defining the queues will not help. You need also the cluster repository process to run and check for the input count on the SYSTEM.CLUSTER queues.
My guess is talk with your system admin. Make sure all is set up correctly to enable cluster operations and then bounce the qmgr.
Now you need to look at translating that to zOS terminology... |
Anyone know what I should be looking for there? When I mention MQSeries to our sysadmins, they kind of glaze over... I'm starting to come to the conclusion that I'll need to bounce the QM but I don't want to do that unless there's something I can see to show that bouncing it might resolve my problem. Explaining I bounced the QM and took out Production (albeit biefly) for nought could be tricky |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sat Aug 28, 2010 2:03 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| Yes, I got a missing queue message when I started defining the cluster bits on the mainframe. |
Missing queue? Do you mean ReasonCode 2085 UNKNOWN_OBJECT_NAME? Or some other? Please be explicit.
| Quote: |
| I mentioned the starting/stopping only to illustrate that the definitions appear to be correct... |
The definitions appear to be correct...now? Or when you discovered that the cluster objects were missing? Or after you ran CSQUTIL with the missing object definitions? Please be explicit.
If the CSQUTIL method, did the SYSOUT indicate any errors? Did you cycle the qmgr? Did it help? Is the REPOSITORY queue still empty? Please be explicit.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sat Aug 28, 2010 2:08 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Earlier, you stated:
| Quote: |
| Yep, and as mentioned, double-checked that their definitions are correct absolutely correct. |
You also stated that the repository queue as empty. How did you determine that the CLUSSDR/CLUSRCVR channel defs are correct?
| Quote: |
| Incidentally, stopping the CLUSSDR on each side does stop the corresponding CLUSRCVR on the other side. And restarting, starts the other... |
Did the channels start successfully?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Allan Mackay |
| Posted: Thu Sep 02, 2010 1:38 am Post subject: |
|
|
Apprentice
Joined: 08 Jul 2010
Posts: 29
|
Thinking back to our first attempt to cluster, and in discussion with a colleague... The sequence of events was:* ran a script to alter mainframe qmgr to be a full repository, deleting and redefinng channels to be CLUSSDR/CLUSRCVR channels and deleting remote queues and redefining aliases to target local queues on remote (zLinux) qmgr
* ran a script to alter zLinux qmgr to be a full repository, deleting and redefining channels to be CLUSSDR/CLUSRCVR channels and altering local queues to register them with the cluster
* tested the cluster setup by putting a message on an alias queue on the mainframe and getting a 2082 (MQRC_UNKNOWN_ALIAS_BASE_Q) return code At this point we started to investigate the cluster. We noticed that, on the zLinux side, the mainframe qmgr hadn't registered properly (was showing as SYSTEM.TEMPQMGR.xx.xx.xx.xx(xxxx) and that, on the mainframe side, the CLUSSDR hadn't started. Our first thought was that we hadn't got the channel definitions correct. We double checked, they were correct. We deleted and redefined them again. And had exactly the same issue.
I don't know why we did (intuition?) but we then checked the system quues on the mainframe qmgr and spotted that the system cluster queue definitions were missing! So, tongue in cheek, we defined them (we actually grabbed the definitions from another qmgr and used them, but that is the same definitions that would have been pulled in from the original startup of the mainframe qmgr). This didn't resolve anything, even after dropping everything from the cluster and re-adding. The only thing that is different is that the mainframe CLUSSDR channel now starts.
The SYSTEM.CLUSTER.COMMAND.QUEUE on the mainframe has messages on it, the SYSTEM.CLUSTER.REPOSITORY.QUEUE is empty. Whereas, on the zLinux side, the COMMAND.QUEUE is empty and the REPOSITORY queue is populated (as it should be).
The CLUSSDR and CLUSRCVR channels on both sides are now inactive (I'm not at all sure they should be). If we start the CLUSSDRs, the corresponding CLUSRCVRs start...
We have, since, declustered and reclustered in smaller satges (alter qmgrs to be clustered first, define CLUSSDR/CLUSRCVR pairs on both sides and only then defining the queues).
The only thing we haven't tried is bouncing the mainframe qmgr but, as indicated earlier, this isn't something we can do on a whim and certainly not without any assurance it will improve matters... |
|
| Back to top |
|
|
| Mr Butcher |
| Posted: Thu Sep 02, 2010 2:09 am Post subject: |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
the repository manager process, that reads the system.cluster.command.queue runs in the CHIN address space.
If the CHIN was started without the cluster queues being present, the repository manager ends with that 2085 message.
| Code: |
CSQX410I xxxx CSQXREPO Repository manager started
CSQX036E xxxx CSQXREPO Unable to open SYSTEM.CLUSTER.COMMAND.QUEUE,
MQCC=2 MQRC=2085
CSQX411I xxxx CSQXREPO Repository manager stopped |
I do not know any other way to restart thje repository manager then restarting the CHIN. Maybe you can do that and leave the MSTR up.....
_________________
Regards, Butcher |
|
| Back to top |
|
|
| Allan Mackay |
| Posted: Thu Sep 02, 2010 2:45 am Post subject: |
|
|
Apprentice
Joined: 08 Jul 2010
Posts: 29
|
Bingo! That WAS the bit I was missing! Thank you ever so much Mr Butcher, the veil has been lifted  |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Sep 02, 2010 5:42 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| hinking back to our first attempt to cluster... |
You need to go back further, to the point where someone deleted the SYSTEM.CLUSTER queues. This is/was the cause of all your grief.
I've seen this type of behavior before, where someone - with the best of intentions, and lack of understanding - deletes some useful/vital objects.
Unfortunately, IBM suggests that once your qmgr is created (started the first time), the INP2 concatenation datasets that create the SYSTEM. objects definitions be commented out. At v2.x, this saved a few milliseconds at startup.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
|
  |
Goto page 1, 2 Next |
Page 1 of 2 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
