| Author |
Message
|
| cicsprog |
 Posted: Wed Jul 30, 2014 11:02 am Post subject: Need debug idea for CLUSTER |
 |
|
Partisan
Joined: 27 Jan 2002
Posts: 347
|
| We DO NOT have centralized MQ Support. We on the mainframe MQ side of the house are getting alerts for BUFFERPOOL 0 too small. Since we don't have any Qlocals other than some SYSTEM.REPOSITORY.* queues on that pageset, I suspect some non-mainframe mqm has automated a REFRESH CLUSTER command and we get flooded with subscription messages periodically. Does anyone have an idea on how to catch this bad actor? |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Wed Jul 30, 2014 12:02 pm Post subject: Re: Need debug idea for CLUSTER |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| cicsprog wrote: |
| I suspect some non-mainframe mqm has automated a REFRESH CLUSTER command and we get flooded with subscription messages periodically. Does anyone have an idea on how to catch this bad actor? |
Why would you suspect this? To whom have you granted RACF authority to issue administrative commands?
Exactly what error are you getting? Post the complete error message here.
How small is bufferpool 0? Is it set to expand automatically? If so, by how much?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| cicsprog |
| Posted: Wed Jul 30, 2014 12:13 pm Post subject: |
|
|
Partisan
Joined: 27 Jan 2002
Posts: 347
|
DEFINE BUFFPOOL( 0 ) BUFFERS( 50000 )
14.52.52 STC35760 CSQP020E &MQL0 CSQP1RSW Buffer pool 0 is too small
SYSTEM.CLUSTER.REPOSITORY.QUEUE is on PAGESET 0
There are other TEST mainframe MQM's getting same CSQP020E and they all share the same set of clusters
RACF wouldn't be involved if the REFRESH CLUSTER was issued from a distributed MQM |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Jul 30, 2014 1:47 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Alter the buffer pool to allow auto expand.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| cicsprog |
| Posted: Wed Jul 30, 2014 1:55 pm Post subject: |
|
|
Partisan
Joined: 27 Jan 2002
Posts: 347
|
| Don't see anything in the book about auto expanding. I can alter it to add or decrease buffers but thats a manual process. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jul 30, 2014 2:52 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Make sure that new queues or temporary queues etc do not get created in bufferpool 0. What is the bufferpool/pageset for SYSTEM.DEFAULT.LOCAL.QUEUE, SYSTEM.DEAD.LETTER.QUEUE, <QMGR.DLQ>, SYSTEM.DEFAULT.MODEL.QUEUE?
How can you be sure that it is one of the system queues and not some temporary queue or new queue created and deleted by a test app that then puts the load on Bufferpool 0?
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Jul 30, 2014 2:54 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Sorry. Memory didn't serve. There is no auto expand on BUFFERPOOL. Rather, auto expand is on PSID.
Use the DISPLAY USAGE TYPE(PAGESET) command to display buffer pool information.
Use the ALTER BUFFERPOOL command to increase the number of buffers.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| cicsprog |
| Posted: Wed Jul 30, 2014 2:58 pm Post subject: |
|
|
Partisan
Joined: 27 Jan 2002
Posts: 347
|
| ya...checked all of that. Everytime I monitored it, PAGESET 0 is empty by the time I get to look at it. Same for thes other test mainframe mqms. The only thing I can think of is to have my AO guys issue a SVC dump when the CSQP020E pops and go dump diggin to see whats in the Q's for PAGESET 0. But looking at the msg data in the SCRQ ain't going to be fruitful since its in a proprietary format. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jul 30, 2014 3:03 pm Post subject: Re: Need debug idea for CLUSTER |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| cicsprog wrote: |
| Since we don't have any Qlocals other than some SYSTEM.REPOSITORY.* queues on that pageset, |
Be precise. There are no queues by that name, and I'm sure there are more queues than 'some repository' queues there.
List ALL the queues that are in Page Set zero. I'd be very surprised if it was the SYSTEM.CLUSTER.REPOSITORY.QUEUE that is causing the issue.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jul 30, 2014 3:03 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Don't waste the time.
Find all the page sets using buffer pool 0.
Then have the qmgr disclose all queues using these page sets.
If any of them can be used as model for defining other queues you may be hitting a transitory problem that will go away as soon as you put those queues into a different page set.
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| cicsprog |
| Posted: Wed Jul 30, 2014 3:08 pm Post subject: Re: Need debug idea for CLUSTER |
|
|
Partisan
Joined: 27 Jan 2002
Posts: 347
|
| PeterPotkay wrote: |
| cicsprog wrote: |
| Since we don't have any Qlocals other than some SYSTEM.REPOSITORY.* queues on that pageset, |
Be precise. There are no queues by that name, and I'm sure there are more queues than 'some repository' queues there.
List ALL the queues that are in Page Set zero. I'd be very surprised if it was the SYSTEM.CLUSTER.REPOSITORY.QUEUE that is causing the issue. |
TYPO SYSTEM.CLUSTER.REPOSITORY.QUEUE is the only CLUSTER QUEUE on PSID 0. the other SYSTEM.CLUSTER.* queues are on PSID98
there are some of the SYSTEM.BROKER.* (not used) and SYSTEM.CHANNEL.* on PSID0
Think about it.....this is happening in TEST not PROD (much higher volumes) and the Q's are placed similiar in PROD. I've seen newbie MQ Admins before script commands after a problem, like REFRESH CLUSTER, fixed a problem not knowing the impact of such commands. I've seen them issue CLUSTER define commands everytime the server starts...so you can see why I am thinking some numskull has automated a REFRESH. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jul 30, 2014 3:35 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Is this mainframe QM one of the 2 Full Repository QMs for this cluster?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| cicsprog |
| Posted: Wed Jul 30, 2014 3:49 pm Post subject: |
|
|
Partisan
Joined: 27 Jan 2002
Posts: 347
|
| partial for two clusters - one cluster with SSL channels and the other cluster nonSSL channels. Clusters WHS/DEV and WHS/SSL/DEV. Large clusters with many test mqms and many cluster Q's |
|
| Back to top |
|
|
| cicsprog |
| Posted: Wed Jul 30, 2014 3:53 pm Post subject: |
|
|
Partisan
Joined: 27 Jan 2002
Posts: 347
|
| no worries about this...I will figure it out...was just looking for an idea on how to tell if a flood of subscriptions come in how I might tell which mqm the REFRESH came from...it might not be possible to tell since I would think the FR would be sending the subscription msgs and might not be the mqm where the REFRESH was issued. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Jul 31, 2014 4:34 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Easy check.... What happens if you switch the SYSTEM.CLUSTER.REPOSITORY.QUEUE from a PSID tied to buffer pool 0 to a PSID tied to a different buffer pool?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
