| Author |
Message
|
| senMQ |
 Posted: Wed Jan 05, 2011 9:58 pm Post subject: REFRESH CLUSTER command |
 |
|
Acolyte
Joined: 14 Aug 2006
Posts: 66
Location: Palo Alto, CA
|
Hi,
We have a cluster with two FRs. One of our application developers ran the "REFRESH CLUSTER" command on the cluster first FR queue manager. During this time, the channel from the second FR to 1st FR was in retrying state due to unavailability of resources in the first FR .( there were too many server connection channel instances running on this queue manager).
The first FR lost all the cluster information after the execution of "refresh cluster" command. The SYSTEM.CLUSTER.REPOSITORY.QUEUE had only 13 messages in it. Later, I fixed the channel issue between the two full repositories. But, the first FR queue manager did not get the cluster information from the second FR, even after the sender channel from the 2nd FR to the 1st FR started running.
Finally, I changed the first FR to partial repository temporarily and then changed it to FR:
ALTER QMGR REPOS(' ')
ALTER QMGR REPOS(<cluster name>)
I saw messages coming into SYSTEM.CLUSTER.REPOSITORY.QUEUE at this point. Now, this queue has more than 500 messages. In short, everything is working fine now. But, can I conclude that the issue occurred due to the channel issues at the time of executing the refresh cluster command?
TIA |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Wed Jan 05, 2011 10:11 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
You should not execute a refresh cluster command if the channel to the FR is not working properly. You can potentially create more harm with it then if you let it be... Of course if you are a FR the connection that needs to work fine is to the "Other" FR...
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jan 05, 2011 10:19 pm Post subject: Re: REFRESH CLUSTER command |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| senMQ wrote: |
| One of our application developers ran the "REFRESH CLUSTER" command on the cluster first FR queue manager. |
Um, why do they even have that capability?!
| senMQ wrote: |
The SYSTEM.CLUSTER.REPOSITORY.QUEUE had only 13 messages in it.
.
.
.
Now, this queue has more than 500 messages. |
The number of messages in the SYSTEM.CLUSTER.REPOSITORY.QUEUE means absolutely nothing. On a working test cluster, issue the refresh command. Issue it again. Issue it again. The q depth is changing but the cluster keeps working normally. The number of messages in the SYSTEM.CLUSTER.REPOSITORY.QUEUE means absolutely nothing.
Well, if the q depth is zero I suppose that means something (bad), but otherwise don't try to deduce anything from the q depth of the S.C.T.Q.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Jan 05, 2011 11:18 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
I'd strongly suggest reading the WMQ Clusters manual.
I'd also suggest taking application developers out of the mqm group. Developers have no legitimate business doing system administrative tasks - in test, qa or production.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| senMQ |
| Posted: Wed Jan 05, 2011 11:21 pm Post subject: |
|
|
Acolyte
Joined: 14 Aug 2006
Posts: 66
Location: Palo Alto, CA
|
Ref:
"The number of messages in the SYSTEM.CLUSTER.REPOSITORY.QUEUE means absolutely nothing. On a working test cluster, issue the refresh command. Issue it again. Issue it again. The q depth is changing but the cluster keeps working normally. The number of messages in the SYSTEM.CLUSTER.REPOSITORY.QUEUE means absolutely nothing. "
If that is the case, from where would the queue manager get the necessary cluster object information? Also, I checked the other FR and it has about 500 messages now. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Jan 06, 2011 2:38 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
One would expect more messages in S.C.R.Q for a large cluster than a small cluster.
As FJ says, if REFRESH CLUSTER is issued, the channel to the FR needs to be working for any new information to be gained. If it's an FR that's been REFRESHed, then it's the channel to the OTHER FR that needs to be working.
As Bruce said - why did the app developers have the ability to run REFRESH CLUSTER ? |
|
| Back to top |
|
|
| Mr Butcher |
| Posted: Thu Jan 06, 2011 2:57 am Post subject: |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
AFAIK there is some kind of "compression" in the S.C.R.Q. 500 messages caused by object creation or refresh cluster commands or whatever may be compressed to 10-20 messages by the mq cluster itself. i dont know details or when this happens, but i remember this from some discussion / error situation with IBM.
I think this makes sense from a performance point of view.
so, the number of messages in the SCRQ does not mean anything. maybe check the message length ...... or the content
_________________
Regards, Butcher |
|
| Back to top |
|
|
| senMQ |
| Posted: Fri Jan 07, 2011 7:44 am Post subject: REFRESH CLUSTER command |
|
|
Acolyte
Joined: 14 Aug 2006
Posts: 66
Location: Palo Alto, CA
|
I just came to know of another event that happened on that day. Looks like someone added a new queue manager into the cluster that has the same name as another queue manager which is already in the cluster. Does MQ allow 2 queue managers with the same name on one cluster? It seems like that the application team ran the "refresh cluster" command to fix this. But unfortunately it led to slew of other issues. 
TIA |
|
| Back to top |
|
|
| exerk |
| Posted: Fri Jan 07, 2011 7:52 am Post subject: Re: REFRESH CLUSTER command |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| senMQ wrote: |
| ...Looks like someone added a new queue manager into the cluster that has the same name as another queue manager which is already in the cluster. Does MQ allow 2 queue managers with the same name on one cluster?... |
Oh dear...but that's what you get when you allow applications people to do admin work - would they allow you the same level of access to their stuff? However, in answer to your question, you might wish to go and read the Clustering manual, and specifically the section on Naming Convention, which will answer your question.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Jan 07, 2011 7:54 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
WMQ does nothing to prevent adding a duplicate qmgr name to a cluster.
How oo successfully remove cluster object definitions for the wrong qmgr added to the cluster is described in the WMQ Clusters manual RESET command. You must use the QMID of the qmgr you intend to remove from the cluster.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Jan 07, 2011 7:58 am Post subject: Re: REFRESH CLUSTER command |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| senMQ wrote: |
| Does MQ allow 2 queue managers with the same name on one cluster? |
Yes. The problem is it doesn't work and is likely to break your cluster.
| senMQ wrote: |
| It seems like that the application team ran the "refresh cluster" command to fix this. |
So they felt having made an invalid change to a cluster, issuing a command to propagate the change throughout the cluster was the best move?
| senMQ wrote: |
| But unfortunately it led to slew of other issues. |
Yes, it would do. Doing that takes you from "likely to break your cluster" to "will certainly break your cluster". It does explain why the FRs lost the will to go on.
One has to ask why untrained, insufficiently cautious application developers have the authority to do this.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
