| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Cluster communication broken when cluster queue full |
« View previous topic :: View next topic » |
| Author |
Message
|
| abd.wsu |
 Posted: Tue Apr 28, 2015 6:18 am Post subject: Cluster communication broken when cluster queue full |
 |
|
Acolyte
Joined: 12 Sep 2012
Posts: 65
|
We ran into an issue with our cluster setup recently. We are on MQ 7504 on RHEL 5.6.
Now, we have a cluster queue on our gateway where messages are put by a broker flow/broker queue manager in the cluster with gateway. This queue got full and we saw the SCTQ on the broker backup with messages supposed to go to this queue. This broke our entire cluster communication.
Now i was under the impression that the messages would end up on the DLQ when a queue got full, but that didn't happen here.
Am i wrong in assuming that? I didn't see any errors in the logs of either queue manager. No FDCs. Some errors on the broker queue manager that there was a resource shortage on the gateway side.
That kinda threw us off as we were thinking the cluster channels were fighting with the svrconn channels for resources.
The cluster communication was restored after the queue was cleaned up.
My question is , why did the messages not go to DLQ? The DLQ was empty. |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Tue Apr 28, 2015 11:30 am Post subject: Re: Cluster communication broken when cluster queue full |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| abd.wsu wrote: |
We ran into an issue with our cluster setup recently. We are on MQ 7504 on RHEL 5.6.
Now, we have a cluster queue on our gateway where messages are put by a broker flow/broker queue manager in the cluster with gateway. This queue got full and we saw the SCTQ on the broker backup with messages supposed to go to this queue. This broke our entire cluster communication.
Now i was under the impression that the messages would end up on the DLQ when a queue got full, but that didn't happen here.
Am i wrong in assuming that? I didn't see any errors in the logs of either queue manager. No FDCs. Some errors on the broker queue manager that there was a resource shortage on the gateway side.
That kinda threw us off as we were thinking the cluster channels were fighting with the svrconn channels for resources.
The cluster communication was restored after the queue was cleaned up.
My question is , why did the messages not go to DLQ? The DLQ was empty. |
Here are a few questions of our own:
- Was the DLQ configured on the destination / gateway qmgr?
- What was the consumption rate on the queue that was full?
The thing is that the cluster channel will do a number of retries at a specific interval (all specified in the channel def and configurable). If the get rate on the queue is such that you consume as fast or faster than the timing for retries, there will be no message on the DLQ. In the meantime however everything backs up because the retries slow down the channel tremendously.
So what you do want to avoid at all costs in your cluster is a queue full.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| abd.wsu |
| Posted: Tue Apr 28, 2015 12:05 pm Post subject: |
|
|
Acolyte
Joined: 12 Sep 2012
Posts: 65
|
Hi,
Yes the DLQ was configured. But your explanation makes absolute sense. This could have been the case with my issue. The messages were being read from the queue in question, but just that the queue was backed up.
We could tune the cluster retries but that would impact the entire cluster communication. The best way to avoid this is to keep the queues clean.
Yeah, we are trying to keep the queues all empty to prevent the impact on the cluster. This is more like a one off issue that caught me by surprise.
Would one more symptom of this mechanism be the channel going into mixed state or paused state?
Thanks a lot for the brilliant explanation. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Apr 28, 2015 5:37 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| abd.wsu wrote: |
Hi,
Yes the DLQ was configured. But your explanation makes absolute sense. This could have been the case with my issue. The messages were being read from the queue in question, but just that the queue was backed up.
We could tune the cluster retries but that would impact the entire cluster communication. The best way to avoid this is to keep the queues clean.
Yeah, we are trying to keep the queues all empty to prevent the impact on the cluster. This is more like a one off issue that caught me by surprise.
Would one more symptom of this mechanism be the channel going into mixed state or paused state?
Thanks a lot for the brilliant explanation. |
paused totally possible.... like I said, everything backs up, and the consumption rate on the full queue is just enough to avoid seeing messages on the DLQ. Best make the queue bigger and start a few more instances of the consuming app.
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
