| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| what if SYSTEM.CLUSTER.TRANSMIT.QUEUE has 300.000 msg's |
« View previous topic :: View next topic » |
| Author |
Message
|
| serpota |
 Posted: Sat Dec 13, 2014 10:45 am Post subject: what if SYSTEM.CLUSTER.TRANSMIT.QUEUE has 300.000 msg's |
 |
|
Voyager
Joined: 26 May 2006
Posts: 85
|
We have a nice cluster that's been working fine for a while, few years (mq v6 in fact). It has 4 FR's (we cant change that) and 10 PR's.
There's been an incident last week (a phantom FR defined and deleted with some trouble) and after that, one of the FR's SYSTEM.CLUSTER.TRANSMIT.QUEUE has started growing slowly by slowly and now has 150.000 messages.
Channels to destination PR are up and running.
It is a production environment, so radical actions are not allowed - data recovery and survival is of main importance.
What would you recommend ? |
|
| Back to top |
|
 |
| PeterPotkay |
| Posted: Sat Dec 13, 2014 2:19 pm Post subject: Re: what if SYSTEM.CLUSTER.TRANSMIT.QUEUE has 300.000 msg's |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| serpota wrote: |
| What would you recommend ? |
Engage someone officially who is an MQ Cluster expert to clean up your topology.
You need to to get rid of the extra FRs. There is no reason to have more than 2.
You need to implement policies, procedures and technical solutions that prevent "phantom" FRs from being defined.
It seems like one of your legit FRs seems to think the phantom FR is still valid, and so wants to let that FR know about all changes in the cluster, so it puts messages to the S.C.T.Q. to go to the phantom, except the cluster sender channel to the phantom is retrying since you guys got rid of the phantom, apparently not 100% in a manner that officially had it leave the cluster.
I'm assuming all the messages piling up in the S.C.T.Q. are administrative FR to FR messages. Is that true, or are there application messages queued up?
Do you have any cluster channels retrying on the FR that has all these messages in the S.C.T.Q.?
In order to avoid making a bad situation worse, I think you should open up a PMR right away and follow their guidance.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| serpota |
| Posted: Sat Dec 13, 2014 4:42 pm Post subject: |
|
|
Voyager
Joined: 26 May 2006
Posts: 85
|
Thanks for you (large) answer, Peter.
We do have an open PMR - I can send you the number, if you want to, but labs answer is ... evasive, as the problem was caused (started) by our creation of the "phantom" FR. It is not a product problem, it seems.
But now Severity is "1", jejeje and the "recovery" does not work.
I think admin messages have gone from SCTQ, as we do "browse" (using MO71) and see applications messages that look stuck/frozen there.
No channels are retrying, sure.
Customer architects have been told only 2 FRs are required, but the change will take time, you know.
Scripts to create qmgrs will be improved, of course,
but there was a small human error on the "input" file, you see
"runmqsc QMGR-NAME < wrong-file.MQSC" is always possible, and it was a FR.
We are stopping message sources to see what remains in the queue.
Saturday nite will be a long nite here.
Thanks. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
