ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » Clustering » Cluster communication broken when cluster queue full

Post new topic  Reply to topic
 Cluster communication broken when cluster queue full « View previous topic :: View next topic » 
Author Message
abd.wsu
PostPosted: Tue Apr 28, 2015 6:18 am    Post subject: Cluster communication broken when cluster queue full Reply with quote

Acolyte

Joined: 12 Sep 2012
Posts: 65

We ran into an issue with our cluster setup recently. We are on MQ 7504 on RHEL 5.6.
Now, we have a cluster queue on our gateway where messages are put by a broker flow/broker queue manager in the cluster with gateway. This queue got full and we saw the SCTQ on the broker backup with messages supposed to go to this queue. This broke our entire cluster communication.
Now i was under the impression that the messages would end up on the DLQ when a queue got full, but that didn't happen here.
Am i wrong in assuming that? I didn't see any errors in the logs of either queue manager. No FDCs. Some errors on the broker queue manager that there was a resource shortage on the gateway side.
That kinda threw us off as we were thinking the cluster channels were fighting with the svrconn channels for resources.
The cluster communication was restored after the queue was cleaned up.

My question is , why did the messages not go to DLQ? The DLQ was empty.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Tue Apr 28, 2015 11:30 am    Post subject: Re: Cluster communication broken when cluster queue full Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

abd.wsu wrote:
We ran into an issue with our cluster setup recently. We are on MQ 7504 on RHEL 5.6.
Now, we have a cluster queue on our gateway where messages are put by a broker flow/broker queue manager in the cluster with gateway. This queue got full and we saw the SCTQ on the broker backup with messages supposed to go to this queue. This broke our entire cluster communication.
Now i was under the impression that the messages would end up on the DLQ when a queue got full, but that didn't happen here.
Am i wrong in assuming that? I didn't see any errors in the logs of either queue manager. No FDCs. Some errors on the broker queue manager that there was a resource shortage on the gateway side.
That kinda threw us off as we were thinking the cluster channels were fighting with the svrconn channels for resources.
The cluster communication was restored after the queue was cleaned up.

My question is , why did the messages not go to DLQ? The DLQ was empty.

Here are a few questions of our own:
  • Was the DLQ configured on the destination / gateway qmgr?
  • What was the consumption rate on the queue that was full?

The thing is that the cluster channel will do a number of retries at a specific interval (all specified in the channel def and configurable). If the get rate on the queue is such that you consume as fast or faster than the timing for retries, there will be no message on the DLQ. In the meantime however everything backs up because the retries slow down the channel tremendously.

So what you do want to avoid at all costs in your cluster is a queue full.

Have fun
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
abd.wsu
PostPosted: Tue Apr 28, 2015 12:05 pm    Post subject: Reply with quote

Acolyte

Joined: 12 Sep 2012
Posts: 65

Hi,

Yes the DLQ was configured. But your explanation makes absolute sense. This could have been the case with my issue. The messages were being read from the queue in question, but just that the queue was backed up.
We could tune the cluster retries but that would impact the entire cluster communication. The best way to avoid this is to keep the queues clean.
Yeah, we are trying to keep the queues all empty to prevent the impact on the cluster. This is more like a one off issue that caught me by surprise.

Would one more symptom of this mechanism be the channel going into mixed state or paused state?


Thanks a lot for the brilliant explanation.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Tue Apr 28, 2015 5:37 pm    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

abd.wsu wrote:
Hi,

Yes the DLQ was configured. But your explanation makes absolute sense. This could have been the case with my issue. The messages were being read from the queue in question, but just that the queue was backed up.
We could tune the cluster retries but that would impact the entire cluster communication. The best way to avoid this is to keep the queues clean.
Yeah, we are trying to keep the queues all empty to prevent the impact on the cluster. This is more like a one off issue that caught me by surprise.

Would one more symptom of this mechanism be the channel going into mixed state or paused state?


Thanks a lot for the brilliant explanation.

paused totally possible.... like I said, everything backs up, and the consumption rate on the full queue is just enough to avoid seeing messages on the DLQ. Best make the queue bigger and start a few more instances of the consuming app.

Have fun
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » Clustering » Cluster communication broken when cluster queue full
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.