ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » Clustering » Need debug idea for CLUSTER

Post new topic  Reply to topic Goto page 1, 2  Next
 Need debug idea for CLUSTER « View previous topic :: View next topic » 
Author Message
cicsprog
PostPosted: Wed Jul 30, 2014 11:02 am    Post subject: Need debug idea for CLUSTER Reply with quote

Partisan

Joined: 27 Jan 2002
Posts: 314

We DO NOT have centralized MQ Support. We on the mainframe MQ side of the house are getting alerts for BUFFERPOOL 0 too small. Since we don't have any Qlocals other than some SYSTEM.REPOSITORY.* queues on that pageset, I suspect some non-mainframe mqm has automated a REFRESH CLUSTER command and we get flooded with subscription messages periodically. Does anyone have an idea on how to catch this bad actor?
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Jul 30, 2014 12:02 pm    Post subject: Re: Need debug idea for CLUSTER Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

cicsprog wrote:
I suspect some non-mainframe mqm has automated a REFRESH CLUSTER command and we get flooded with subscription messages periodically. Does anyone have an idea on how to catch this bad actor?

Why would you suspect this? To whom have you granted RACF authority to issue administrative commands?

Exactly what error are you getting? Post the complete error message here.

How small is bufferpool 0? Is it set to expand automatically? If so, by how much?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
cicsprog
PostPosted: Wed Jul 30, 2014 12:13 pm    Post subject: Reply with quote

Partisan

Joined: 27 Jan 2002
Posts: 314

DEFINE BUFFPOOL( 0 ) BUFFERS( 50000 )

14.52.52 STC35760 CSQP020E &MQL0 CSQP1RSW Buffer pool 0 is too small

SYSTEM.CLUSTER.REPOSITORY.QUEUE is on PAGESET 0

There are other TEST mainframe MQM's getting same CSQP020E and they all share the same set of clusters

RACF wouldn't be involved if the REFRESH CLUSTER was issued from a distributed MQM
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Jul 30, 2014 1:47 pm    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

Alter the buffer pool to allow auto expand.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
cicsprog
PostPosted: Wed Jul 30, 2014 1:55 pm    Post subject: Reply with quote

Partisan

Joined: 27 Jan 2002
Posts: 314

Don't see anything in the book about auto expanding. I can alter it to add or decrease buffers but thats a manual process.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Wed Jul 30, 2014 2:52 pm    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

Make sure that new queues or temporary queues etc do not get created in bufferpool 0. What is the bufferpool/pageset for SYSTEM.DEFAULT.LOCAL.QUEUE, SYSTEM.DEAD.LETTER.QUEUE, <QMGR.DLQ>, SYSTEM.DEFAULT.MODEL.QUEUE?

How can you be sure that it is one of the system queues and not some temporary queue or new queue created and deleted by a test app that then puts the load on Bufferpool 0?

Have fun
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
bruce2359
PostPosted: Wed Jul 30, 2014 2:54 pm    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

Sorry. Memory didn't serve. There is no auto expand on BUFFERPOOL. Rather, auto expand is on PSID.

Use the DISPLAY USAGE TYPE(PAGESET) command to display buffer pool information.

Use the ALTER BUFFERPOOL command to increase the number of buffers.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
cicsprog
PostPosted: Wed Jul 30, 2014 2:58 pm    Post subject: Reply with quote

Partisan

Joined: 27 Jan 2002
Posts: 314

ya...checked all of that. Everytime I monitored it, PAGESET 0 is empty by the time I get to look at it. Same for thes other test mainframe mqms. The only thing I can think of is to have my AO guys issue a SVC dump when the CSQP020E pops and go dump diggin to see whats in the Q's for PAGESET 0. But looking at the msg data in the SCRQ ain't going to be fruitful since its in a proprietary format.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Wed Jul 30, 2014 3:03 pm    Post subject: Re: Need debug idea for CLUSTER Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

cicsprog wrote:
Since we don't have any Qlocals other than some SYSTEM.REPOSITORY.* queues on that pageset,


Be precise. There are no queues by that name, and I'm sure there are more queues than 'some repository' queues there.

List ALL the queues that are in Page Set zero. I'd be very surprised if it was the SYSTEM.CLUSTER.REPOSITORY.QUEUE that is causing the issue.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Wed Jul 30, 2014 3:03 pm    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

Don't waste the time.
Find all the page sets using buffer pool 0.
Then have the qmgr disclose all queues using these page sets.
If any of them can be used as model for defining other queues you may be hitting a transitory problem that will go away as soon as you put those queues into a different page set.

Have fun
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
cicsprog
PostPosted: Wed Jul 30, 2014 3:08 pm    Post subject: Re: Need debug idea for CLUSTER Reply with quote

Partisan

Joined: 27 Jan 2002
Posts: 314

PeterPotkay wrote:
cicsprog wrote:
Since we don't have any Qlocals other than some SYSTEM.REPOSITORY.* queues on that pageset,


Be precise. There are no queues by that name, and I'm sure there are more queues than 'some repository' queues there.

List ALL the queues that are in Page Set zero. I'd be very surprised if it was the SYSTEM.CLUSTER.REPOSITORY.QUEUE that is causing the issue.


TYPO SYSTEM.CLUSTER.REPOSITORY.QUEUE is the only CLUSTER QUEUE on PSID 0. the other SYSTEM.CLUSTER.* queues are on PSID98

there are some of the SYSTEM.BROKER.* (not used) and SYSTEM.CHANNEL.* on PSID0

Think about it.....this is happening in TEST not PROD (much higher volumes) and the Q's are placed similiar in PROD. I've seen newbie MQ Admins before script commands after a problem, like REFRESH CLUSTER, fixed a problem not knowing the impact of such commands. I've seen them issue CLUSTER define commands everytime the server starts...so you can see why I am thinking some numskull has automated a REFRESH.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Wed Jul 30, 2014 3:35 pm    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

Is this mainframe QM one of the 2 Full Repository QMs for this cluster?
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
cicsprog
PostPosted: Wed Jul 30, 2014 3:49 pm    Post subject: Reply with quote

Partisan

Joined: 27 Jan 2002
Posts: 314

partial for two clusters - one cluster with SSL channels and the other cluster nonSSL channels. Clusters WHS/DEV and WHS/SSL/DEV. Large clusters with many test mqms and many cluster Q's
Back to top
View user's profile Send private message
cicsprog
PostPosted: Wed Jul 30, 2014 3:53 pm    Post subject: Reply with quote

Partisan

Joined: 27 Jan 2002
Posts: 314

no worries about this...I will figure it out...was just looking for an idea on how to tell if a flood of subscriptions come in how I might tell which mqm the REFRESH came from...it might not be possible to tell since I would think the FR would be sending the subscription msgs and might not be the mqm where the REFRESH was issued.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Thu Jul 31, 2014 4:34 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

Easy check.... What happens if you switch the SYSTEM.CLUSTER.REPOSITORY.QUEUE from a PSID tied to buffer pool 0 to a PSID tied to a different buffer pool?
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » Clustering » Need debug idea for CLUSTER
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.