Author |
Message
|
gbaddeley |
Posted: Thu Dec 13, 2018 4:04 pm Post subject: |
|
|
 Jedi Knight
Joined: 25 Mar 2003 Posts: 2538 Location: Melbourne, Australia
|
|
Back to top |
|
 |
hughson |
Posted: Fri Dec 14, 2018 12:23 am Post subject: |
|
|
 Padawan
Joined: 09 May 2013 Posts: 1959 Location: Bay of Plenty, New Zealand
|
Jeff.VT wrote: |
I just did this trace and got:
Quote: |
001CC9FB 08:35:18.842735 3136.25 CONN:000032 ----------------{ rrxError
001CC9FE 08:35:18.842751 3136.25 CONN:000032 RetCode = 20009211, rc1 = 0, rc2 = 0, Comment1 = '', Comment2 = '', Comment3= '', File= 'F:\build\slot1\p900_P\src\lib\remote\amqrfica.c', Line= '2368'
001CCA02 08:35:18.842769 3136.25 CONN:000032 ----------------}! rrxError (rc=rrcE_NO_STORAGE) |
It keeps referencing F:\, but I don't have an F: drive and no clue where it's getting that.
My installation is on D:\Program Files
My ProgramData is on C:\ProgramData
and my Queue Manager is on M:\<QMName>\ |
Don't worry about the F: drive, that is the file name of the MQ source code that is writing out the error. The full path name is burned into the output, the F: drive is in Hursley!
Cheers,
Morag _________________ Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
Back to top |
|
 |
Jeff.VT |
Posted: Fri Dec 14, 2018 5:46 am Post subject: |
|
|
Acolyte
Joined: 02 Mar 2017 Posts: 71
|
bruce2359 wrote: |
Jeff.VT wrote: |
Quote: |
Regarding 1) above: are you saying that you do NOT have manually defined clussdr channels from PR's to one FR? |
The opposite - I have both FRs manually defined in all my PRs.
|
One more time for clarity.
Firstly,you have only two FR's?
Are you saying that on each PR you have manually defined CLUSSDR channels to BOTH FR's? Or, that you have only one CLUSSDR from PR to ONLY one of the FR's?
If you display from your PR's, you see CLUSSDRB channels to FR's? |
I set up the Cluster using MQ Explorer. When you do that, you are given an option to select which FR you want to manually configure.
If you select one, both still exist, but the one you didn't choose is configured automatically.
If you select both, (which I did), then both are configured manually.
Both appear in my channel list. Where if I had selected just one, only one would appear in my channel list, and the other I'd have to go to DIS CLUSQMGR or DIS CHSTATUS to see.
In DIS CLUSQMGR on my PR's both of my FR's are listed as DEFTYPE(CLUSSDRB).
---------------
As I was looking up this DEFTYPE, I came across this:
Quote: |
CLUSSDRB
A cluster sender-channel has been administratively defined on the local queue manager and accepted as a valid cluster channel by the target queue manager. This is the expected DEFTYPE of a partial repository queue manager's manually configured full repository queue manager. It should also be the DEFTYPE of any CLUSQMGR from one full repository to another full repository in the cluster. Manual cluster-sender channels should not be configured to partial repositories or from a partial repository queue manager to more than one full repository. If a DEFTYPE of CLUSSDRB is seen in either of these situations it should be investigated and corrected. |
So I'm going to go ahead and correct that.
But I don't know if that would explain why my FRs and PRs alike are getting this error. |
|
Back to top |
|
 |
bruce2359 |
Posted: Fri Dec 14, 2018 7:40 am Post subject: |
|
|
 Poobah
Joined: 05 Jan 2008 Posts: 9469 Location: US: west coast, almost. Otherwise, enroute.
|
“... correct that.”
What’s a “that?” Correct what, precisely? _________________ I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
Back to top |
|
 |
Jeff.VT |
Posted: Fri Dec 14, 2018 9:48 am Post subject: |
|
|
Acolyte
Joined: 02 Mar 2017 Posts: 71
|
bruce2359 wrote: |
“... correct that.”
What’s a “that?” Correct what, precisely? |
I removed the 2nd Full Repository from manual definitions...
Note - when you do that, you gotta do a refresh cluster... so... I had a bit of an outage today. All PRs refreshed fine, no errors, no problems. But one. One of those bastages...
One thing I will say - I refreshed one of my full repositories and the error went away there. And it went away on all the PRs I refreshed too.
The one PR caused me a bit of a cluster-f outage when I refreshed it. And it seems like it now doesn't understand how to resolve any older clustered objects... but NEW clustered objects it can resolve fine.
On pre-existing objects I get 2189 MQRC CLUSTER RESOLUTION ERROR
On NEW objects, it's fine.
So I'm going to go grab lunch and figure it out.
And I'm not going to refresh the other full repository until I'm sure that's not what caused my outage... which it probably did... today is frustrating... |
|
Back to top |
|
 |
Jeff.VT |
Posted: Fri Dec 14, 2018 9:59 am Post subject: |
|
|
Acolyte
Joined: 02 Mar 2017 Posts: 71
|
Okay - my new problem stems from the fact that it seems like my queue manager won't create new Cluster Transmit queues for certain destination queue managers...
howcomeisthat? |
|
Back to top |
|
 |
bruce2359 |
Posted: Fri Dec 14, 2018 10:08 am Post subject: |
|
|
 Poobah
Joined: 05 Jan 2008 Posts: 9469 Location: US: west coast, almost. Otherwise, enroute.
|
Please, oh please, be precise. Why can't/won’t it create xmit queues? Which "certain queue managers?" What do they have in common? Are the cluster channels in RUNNING state?
Is there an Error of some kind? _________________ I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
Back to top |
|
 |
Jeff.VT |
Posted: Fri Dec 14, 2018 10:46 am Post subject: |
|
|
Acolyte
Joined: 02 Mar 2017 Posts: 71
|
bruce2359 wrote: |
Please, oh please, be precise. Why can't/won’t it create xmit queues? Which "certain queue managers?" What do they have in common? Are the cluster channels in RUNNING state?
Is there an Error of some kind? |
I am trying to be precise. I'm sorry.
The channels don't go into running, they stay inactive.
Quote: |
"Program cannot open queue manager object.
The attempt to open either the queue or queue manager object 'SYSTEM.CLUSTER.TRANSMIT.PR02' on queue manager 'PR01' failed with reason code 2085.
Ensure that the queue is available and retry the operation." |
I created that xmit q manually and it worked with it. (using another cluster xmitq as a template)
And I'm not certain, but I'm at least 60% sure that maybe 30 minutes after I created that transmit queue, it was deleted by something... and I had to create it again. I don't believe I deleted it, and I don't see the log anywhere saying who or what deleted it. I'm not 100% certain. But I could have sworn... It could just be residual leftovers of the frantically trying to fix the issues I had earlier - Earlier I removed "PR01" from the cluster and re-added it to try to resolve the problems I was having. Probably did it too fast or something.
Anyway. I'm currently working on a script that will create a cluster queue unique to each queue manager in the cluster, then have every queue manager send a message to all the other queue managers to see if there are any other transmit queues that refuse to be created like this. |
|
Back to top |
|
 |
Vitor |
Posted: Fri Dec 14, 2018 12:05 pm Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
|
Back to top |
|
 |
Jeff.VT |
Posted: Fri Dec 14, 2018 12:28 pm Post subject: |
|
|
Acolyte
Joined: 02 Mar 2017 Posts: 71
|
Vitor wrote: |
Jeff.VT wrote: |
Okay - my new problem stems from the fact that it seems like my queue manager won't create new Cluster Transmit queues for certain destination queue managers...
howcomeisthat? |
I always thought that if you used "named" cluster xmitqs you had to define them manually and the queue manager [url=https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_7.5.0/com.ibm.mq.con.doc/q017260_.htm#q017260___s3[/url] picked one. |
I've never had to create the XMITQ's... And I just looked... the vast majority of my XMITQ's are created by the queue manager.
Quote: |
dis ql(SYSTEM.CLUSTER.TRANSMIT.*) DEFTYPE
1 : dis ql(SYSTEM.CLUSTER.TRANSMIT.*) DEFTYPE
AMQ8409I: Display Queue details.
QUEUE(SYSTEM.CLUSTER.TRANSMIT.FR2)
TYPE(QLOCAL) DEFTYPE(PREDEFINED)
AMQ8409I: Display Queue details.
QUEUE(SYSTEM.CLUSTER.TRANSMIT.PR4)
TYPE(QLOCAL) DEFTYPE(PERMDYN)
AMQ8409I: Display Queue details.
QUEUE(SYSTEM.CLUSTER.TRANSMIT.FR2)
TYPE(QLOCAL) DEFTYPE(PREDEFINED)
AMQ8409I: Display Queue details.
QUEUE(SYSTEM.CLUSTER.TRANSMIT.PR3)
TYPE(QLOCAL) DEFTYPE(PERMDYN) |
Some of them are PREDEFINED - the ones I manually created
And some are PERMDYN - the ones MQ created.
I have some that even though the queue doesn't exist... MQ refuses to create it. It just gives an error that it doesn't exist. |
|
Back to top |
|
 |
Jeff.VT |
Posted: Fri Dec 14, 2018 12:49 pm Post subject: |
|
|
Acolyte
Joined: 02 Mar 2017 Posts: 71
|
I just checked... I have 21 queue managers in the cluster - 2 are FRs.
I created a script that would create a unique clustered queue on every queue manager, then have all the other queue managers send a message to it via the cluster.
I only found this error on 2 queue managers. And those are ones I had to remove forcibly from the cluster at one point... then add a different queue manager but with the same name to the cluster.
I'd guess I probably did that wrong at some point. Luckily for me, these two are also ones that will be gone in the next 6-9 months.
So I guess I'm just going to create the dumb XMITQ's, and wait for it to burn me again at some point. |
|
Back to top |
|
 |
Jeff.VT |
Posted: Fri Dec 14, 2018 1:02 pm Post subject: |
|
|
Acolyte
Joined: 02 Mar 2017 Posts: 71
|
On a positive note...
* I removed the 2nd manually-defined FullRepo on all the PartialRepo's
* Then I did a Refresh Cluster(*) repo(yes) on all the PartialRepo's
* Then I did a Refresh Cluster(*) on FullRepo #1
* <Then I had a 30 minute outage where I ran around like a headless chicken>
BUT... Doing all that... My original error (No Repository storage) on the PartialRepo's and FullRepo#1 has gone away...
I don't know if it's gone forever.
And I don't know if it's actually *fixed*
And I'm terrified to refresh FullRepo #2 - I'm certainly not doing it at 4pm on Friday. So I'll give it a try Monday morning and report back.
I'll also set up an alert for that 9500 error and watch to see if it ever comes back.
Thanks for everybody's help and concern! Sorry for the anti-climatic un-resolution. |
|
Back to top |
|
 |
bruce2359 |
Posted: Fri Dec 14, 2018 4:03 pm Post subject: |
|
|
 Poobah
Joined: 05 Jan 2008 Posts: 9469 Location: US: west coast, almost. Otherwise, enroute.
|
|
Back to top |
|
 |
fjb_saper |
Posted: Fri Dec 14, 2018 10:42 pm Post subject: |
|
|
 Grand High Poobah
Joined: 18 Nov 2003 Posts: 20756 Location: LI,NY
|
When the Cluster XMITQ doesn't get created are you sure that you have the corresponding MODEL template queue available?  _________________ MQ & Broker admin |
|
Back to top |
|
 |
Jeff.VT |
Posted: Mon Dec 17, 2018 6:17 am Post subject: |
|
|
Acolyte
Joined: 02 Mar 2017 Posts: 71
|
So it came back... at 3:01am server time the next morning. A cluster refresh I assume cleans up the cluster slightly, and then it eventually just fills up this un-defined 'repository storage' again.
Ya, the XMITQ model queue exists. And it will create the XMITQ's for some cluster channels, but not for all of them.
I don't generally have a problem troubleshooting my queue manager system. I've been working with it for quite a while. The issues I've had over the years I've been able to troubleshoot very effectively and without IBM's involvement.
I attended the MQTC the last 4 years and have learned quite a bit.
The questions I have on my logging system are more of a hail Mary.
But if it were something simple, something that the above course mentions, I have a feeling you guys would have been able set me straight rather easily. Or IBM would have sent me to a whitepaper telling me the problem after I sent them all the data on the system.
I can peruse the material, but I'd guess it's a bit of an overview of things I either was trained on, have gleamed in my many other trainings, or in my 8 years administering this system. |
|
Back to top |
|
 |
|