ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexClusteringAMQ9500 - No Repository storage

Post new topicReply to topic Goto page Previous  1, 2, 3  Next
AMQ9500 - No Repository storage View previous topic :: View next topic
Author Message
gbaddeley
PostPosted: Thu Dec 13, 2018 4:04 pm Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2492
Location: Melbourne, Australia

I Googled for rrcE_NO_STORAGE, AMQ9211, IY20768

Seems to indicate that the MQ process could not request memory from the OS. There may be an issue that the MQ cluster respository memory is full, due to a leak, or the same info being repeatedly allocated.

What info does taskmgr.exe say and process explorer ( https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer ) provide about memory usage by amqzlaa0.exe ?

I found a post from 13 years ago - http://mqseries.net/phpBB/viewtopic.php?t=24597
_________________
Glenn
Back to top
View user's profile Send private message
hughson
PostPosted: Fri Dec 14, 2018 12:23 am Post subject: Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

Jeff.VT wrote:
I just did this trace and got:

Quote:
001CC9FB 08:35:18.842735 3136.25 CONN:000032 ----------------{ rrxError
001CC9FE 08:35:18.842751 3136.25 CONN:000032 RetCode = 20009211, rc1 = 0, rc2 = 0, Comment1 = '', Comment2 = '', Comment3= '', File= 'F:\build\slot1\p900_P\src\lib\remote\amqrfica.c', Line= '2368'
001CCA02 08:35:18.842769 3136.25 CONN:000032 ----------------}! rrxError (rc=rrcE_NO_STORAGE)


It keeps referencing F:\, but I don't have an F: drive and no clue where it's getting that.

My installation is on D:\Program Files
My ProgramData is on C:\ProgramData
and my Queue Manager is on M:\<QMName>\

Don't worry about the F: drive, that is the file name of the MQ source code that is writing out the error. The full path name is burned into the output, the F: drive is in Hursley!

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
Jeff.VT
PostPosted: Fri Dec 14, 2018 5:46 am Post subject: Reply with quote

Acolyte

Joined: 02 Mar 2017
Posts: 68

bruce2359 wrote:
Jeff.VT wrote:
Quote:
Regarding 1) above: are you saying that you do NOT have manually defined clussdr channels from PR's to one FR?


The opposite - I have both FRs manually defined in all my PRs.


One more time for clarity.

Firstly,you have only two FR's?

Are you saying that on each PR you have manually defined CLUSSDR channels to BOTH FR's? Or, that you have only one CLUSSDR from PR to ONLY one of the FR's?

If you display from your PR's, you see CLUSSDRB channels to FR's?


I set up the Cluster using MQ Explorer. When you do that, you are given an option to select which FR you want to manually configure.



If you select one, both still exist, but the one you didn't choose is configured automatically.

If you select both, (which I did), then both are configured manually.
Both appear in my channel list. Where if I had selected just one, only one would appear in my channel list, and the other I'd have to go to DIS CLUSQMGR or DIS CHSTATUS to see.

In DIS CLUSQMGR on my PR's both of my FR's are listed as DEFTYPE(CLUSSDRB).

---------------

As I was looking up this DEFTYPE, I came across this:

Quote:
CLUSSDRB
A cluster sender-channel has been administratively defined on the local queue manager and accepted as a valid cluster channel by the target queue manager. This is the expected DEFTYPE of a partial repository queue manager's manually configured full repository queue manager. It should also be the DEFTYPE of any CLUSQMGR from one full repository to another full repository in the cluster. Manual cluster-sender channels should not be configured to partial repositories or from a partial repository queue manager to more than one full repository. If a DEFTYPE of CLUSSDRB is seen in either of these situations it should be investigated and corrected.


So I'm going to go ahead and correct that.

But I don't know if that would explain why my FRs and PRs alike are getting this error.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Fri Dec 14, 2018 7:40 am Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

“... correct that.”

What’s a “that?” Correct what, precisely?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
Jeff.VT
PostPosted: Fri Dec 14, 2018 9:48 am Post subject: Reply with quote

Acolyte

Joined: 02 Mar 2017
Posts: 68

bruce2359 wrote:
“... correct that.”

What’s a “that?” Correct what, precisely?


I removed the 2nd Full Repository from manual definitions...

Note - when you do that, you gotta do a refresh cluster... so... I had a bit of an outage today. All PRs refreshed fine, no errors, no problems. But one. One of those bastages...

One thing I will say - I refreshed one of my full repositories and the error went away there. And it went away on all the PRs I refreshed too.

The one PR caused me a bit of a cluster-f outage when I refreshed it. And it seems like it now doesn't understand how to resolve any older clustered objects... but NEW clustered objects it can resolve fine.

On pre-existing objects I get 2189 MQRC CLUSTER RESOLUTION ERROR
On NEW objects, it's fine.

So I'm going to go grab lunch and figure it out.

And I'm not going to refresh the other full repository until I'm sure that's not what caused my outage... which it probably did... today is frustrating...
Back to top
View user's profile Send private message
Jeff.VT
PostPosted: Fri Dec 14, 2018 9:59 am Post subject: Reply with quote

Acolyte

Joined: 02 Mar 2017
Posts: 68

Okay - my new problem stems from the fact that it seems like my queue manager won't create new Cluster Transmit queues for certain destination queue managers...

howcomeisthat?
Back to top
View user's profile Send private message
bruce2359
PostPosted: Fri Dec 14, 2018 10:08 am Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

Please, oh please, be precise. Why can't/won’t it create xmit queues? Which "certain queue managers?" What do they have in common? Are the cluster channels in RUNNING state?

Is there an Error of some kind?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
Jeff.VT
PostPosted: Fri Dec 14, 2018 10:46 am Post subject: Reply with quote

Acolyte

Joined: 02 Mar 2017
Posts: 68

bruce2359 wrote:
Please, oh please, be precise. Why can't/won’t it create xmit queues? Which "certain queue managers?" What do they have in common? Are the cluster channels in RUNNING state?

Is there an Error of some kind?


I am trying to be precise. I'm sorry.

The channels don't go into running, they stay inactive.

Quote:
"Program cannot open queue manager object.

The attempt to open either the queue or queue manager object 'SYSTEM.CLUSTER.TRANSMIT.PR02' on queue manager 'PR01' failed with reason code 2085.

Ensure that the queue is available and retry the operation."


I created that xmit q manually and it worked with it. (using another cluster xmitq as a template)

And I'm not certain, but I'm at least 60% sure that maybe 30 minutes after I created that transmit queue, it was deleted by something... and I had to create it again. I don't believe I deleted it, and I don't see the log anywhere saying who or what deleted it. I'm not 100% certain. But I could have sworn... It could just be residual leftovers of the frantically trying to fix the issues I had earlier - Earlier I removed "PR01" from the cluster and re-added it to try to resolve the problems I was having. Probably did it too fast or something.

Anyway. I'm currently working on a script that will create a cluster queue unique to each queue manager in the cluster, then have every queue manager send a message to all the other queue managers to see if there are any other transmit queues that refuse to be created like this.
Back to top
View user's profile Send private message
Vitor
PostPosted: Fri Dec 14, 2018 12:05 pm Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

Jeff.VT wrote:
Okay - my new problem stems from the fact that it seems like my queue manager won't create new Cluster Transmit queues for certain destination queue managers...

howcomeisthat?


I always thought that if you used "named" cluster xmitqs you had to define them manually and the queue manager https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_7.5.0/com.ibm.mq.con.doc/q017260_.htm#q017260___s3 picked one.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Jeff.VT
PostPosted: Fri Dec 14, 2018 12:28 pm Post subject: Reply with quote

Acolyte

Joined: 02 Mar 2017
Posts: 68

Vitor wrote:
Jeff.VT wrote:
Okay - my new problem stems from the fact that it seems like my queue manager won't create new Cluster Transmit queues for certain destination queue managers...

howcomeisthat?


I always thought that if you used "named" cluster xmitqs you had to define them manually and the queue manager [url=https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_7.5.0/com.ibm.mq.con.doc/q017260_.htm#q017260___s3[/url] picked one.


I've never had to create the XMITQ's... And I just looked... the vast majority of my XMITQ's are created by the queue manager.

Quote:
dis ql(SYSTEM.CLUSTER.TRANSMIT.*) DEFTYPE
1 : dis ql(SYSTEM.CLUSTER.TRANSMIT.*) DEFTYPE
AMQ8409I: Display Queue details.
QUEUE(SYSTEM.CLUSTER.TRANSMIT.FR2)
TYPE(QLOCAL) DEFTYPE(PREDEFINED)
AMQ8409I: Display Queue details.
QUEUE(SYSTEM.CLUSTER.TRANSMIT.PR4)
TYPE(QLOCAL) DEFTYPE(PERMDYN)
AMQ8409I: Display Queue details.
QUEUE(SYSTEM.CLUSTER.TRANSMIT.FR2)
TYPE(QLOCAL) DEFTYPE(PREDEFINED)
AMQ8409I: Display Queue details.
QUEUE(SYSTEM.CLUSTER.TRANSMIT.PR3)
TYPE(QLOCAL) DEFTYPE(PERMDYN)


Some of them are PREDEFINED - the ones I manually created

And some are PERMDYN - the ones MQ created.

I have some that even though the queue doesn't exist... MQ refuses to create it. It just gives an error that it doesn't exist.
Back to top
View user's profile Send private message
Jeff.VT
PostPosted: Fri Dec 14, 2018 12:49 pm Post subject: Reply with quote

Acolyte

Joined: 02 Mar 2017
Posts: 68

I just checked... I have 21 queue managers in the cluster - 2 are FRs.

I created a script that would create a unique clustered queue on every queue manager, then have all the other queue managers send a message to it via the cluster.

I only found this error on 2 queue managers. And those are ones I had to remove forcibly from the cluster at one point... then add a different queue manager but with the same name to the cluster.

I'd guess I probably did that wrong at some point. Luckily for me, these two are also ones that will be gone in the next 6-9 months.

So I guess I'm just going to create the dumb XMITQ's, and wait for it to burn me again at some point.
Back to top
View user's profile Send private message
Jeff.VT
PostPosted: Fri Dec 14, 2018 1:02 pm Post subject: Reply with quote

Acolyte

Joined: 02 Mar 2017
Posts: 68

On a positive note...

* I removed the 2nd manually-defined FullRepo on all the PartialRepo's
* Then I did a Refresh Cluster(*) repo(yes) on all the PartialRepo's
* Then I did a Refresh Cluster(*) on FullRepo #1
* <Then I had a 30 minute outage where I ran around like a headless chicken>

BUT... Doing all that... My original error (No Repository storage) on the PartialRepo's and FullRepo#1 has gone away...

I don't know if it's gone forever.
And I don't know if it's actually *fixed*
And I'm terrified to refresh FullRepo #2 - I'm certainly not doing it at 4pm on Friday. So I'll give it a try Monday morning and report back.

I'll also set up an alert for that 9500 error and watch to see if it ever comes back.

Thanks for everybody's help and concern! Sorry for the anti-climatic un-resolution.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Fri Dec 14, 2018 4:03 pm Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

I'm guessing that you haven't attended MQ system administration training. IBM's WM15 course includes lecture, discussion, and hands-on lab exercises. Much time is spent on configuration and problem-determination, and includes basic cluster admin. Labs on Windows and UNIX flavors.
https://www-03.ibm.com/services/learning/ites.wss/zz-en?pageType=course_description&cc=&courseCode=WM153G&gtpcc=us
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Fri Dec 14, 2018 10:42 pm Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

When the Cluster XMITQ doesn't get created are you sure that you have the corresponding MODEL template queue available?
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
Jeff.VT
PostPosted: Mon Dec 17, 2018 6:17 am Post subject: Reply with quote

Acolyte

Joined: 02 Mar 2017
Posts: 68

So it came back... at 3:01am server time the next morning. A cluster refresh I assume cleans up the cluster slightly, and then it eventually just fills up this un-defined 'repository storage' again.

Ya, the XMITQ model queue exists. And it will create the XMITQ's for some cluster channels, but not for all of them.

I don't generally have a problem troubleshooting my queue manager system. I've been working with it for quite a while. The issues I've had over the years I've been able to troubleshoot very effectively and without IBM's involvement.

I attended the MQTC the last 4 years and have learned quite a bit.

The questions I have on my logging system are more of a hail Mary.

But if it were something simple, something that the above course mentions, I have a feeling you guys would have been able set me straight rather easily. Or IBM would have sent me to a whitepaper telling me the problem after I sent them all the data on the system.

I can peruse the material, but I'd guess it's a bit of an overview of things I either was trained on, have gleamed in my many other trainings, or in my 8 years administering this system.
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Goto page Previous  1, 2, 3  Next Page 2 of 3

MQSeries.net Forum IndexClusteringAMQ9500 - No Repository storage
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.