ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » Clustering » what if SYSTEM.CLUSTER.TRANSMIT.QUEUE has 300.000 msg's

Post new topic  Reply to topic
 what if SYSTEM.CLUSTER.TRANSMIT.QUEUE has 300.000 msg's « View previous topic :: View next topic » 
Author Message
serpota
PostPosted: Sat Dec 13, 2014 10:45 am    Post subject: what if SYSTEM.CLUSTER.TRANSMIT.QUEUE has 300.000 msg's Reply with quote

Voyager

Joined: 26 May 2006
Posts: 85

We have a nice cluster that's been working fine for a while, few years (mq v6 in fact). It has 4 FR's (we cant change that) and 10 PR's.
There's been an incident last week (a phantom FR defined and deleted with some trouble) and after that, one of the FR's SYSTEM.CLUSTER.TRANSMIT.QUEUE has started growing slowly by slowly and now has 150.000 messages.
Channels to destination PR are up and running.
It is a production environment, so radical actions are not allowed - data recovery and survival is of main importance.
What would you recommend ?
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Sat Dec 13, 2014 2:19 pm    Post subject: Re: what if SYSTEM.CLUSTER.TRANSMIT.QUEUE has 300.000 msg's Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

serpota wrote:
What would you recommend ?

Engage someone officially who is an MQ Cluster expert to clean up your topology.
You need to to get rid of the extra FRs. There is no reason to have more than 2.
You need to implement policies, procedures and technical solutions that prevent "phantom" FRs from being defined.

It seems like one of your legit FRs seems to think the phantom FR is still valid, and so wants to let that FR know about all changes in the cluster, so it puts messages to the S.C.T.Q. to go to the phantom, except the cluster sender channel to the phantom is retrying since you guys got rid of the phantom, apparently not 100% in a manner that officially had it leave the cluster.

I'm assuming all the messages piling up in the S.C.T.Q. are administrative FR to FR messages. Is that true, or are there application messages queued up?

Do you have any cluster channels retrying on the FR that has all these messages in the S.C.T.Q.?

In order to avoid making a bad situation worse, I think you should open up a PMR right away and follow their guidance.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
serpota
PostPosted: Sat Dec 13, 2014 4:42 pm    Post subject: Reply with quote

Voyager

Joined: 26 May 2006
Posts: 85

Thanks for you (large) answer, Peter.
We do have an open PMR - I can send you the number, if you want to, but labs answer is ... evasive, as the problem was caused (started) by our creation of the "phantom" FR. It is not a product problem, it seems.
But now Severity is "1", jejeje and the "recovery" does not work.

I think admin messages have gone from SCTQ, as we do "browse" (using MO71) and see applications messages that look stuck/frozen there.

No channels are retrying, sure.

Customer architects have been told only 2 FRs are required, but the change will take time, you know.

Scripts to create qmgrs will be improved, of course,
but there was a small human error on the "input" file, you see
"runmqsc QMGR-NAME < wrong-file.MQSC" is always possible, and it was a FR.

We are stopping message sources to see what remains in the queue.
Saturday nite will be a long nite here.

Thanks.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » Clustering » what if SYSTEM.CLUSTER.TRANSMIT.QUEUE has 300.000 msg's
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.