| Author |
Message
|
| bhatnprasanna |
 Posted: Fri Sep 11, 2009 1:20 am Post subject: MQ Cluster setup problem |
 |
|
Newbie
Joined: 11 Sep 2009
Posts: 3
|
Hi,
In our current MQ cluster setup, we have 3 full repository queue managers. We have following cluster sender receiver channels between these full repos.
Q Managers - NODE1, NODE2 NODE3
On NODE1:
TO.NODE1 - ClusRcvr
TO.NODE2 - ClusSdr
TO.NODE3 - ClusSdr
On NODE2:
TO.NODE2 - ClusRcvr
TO.NODE1 - ClusSdr
TO.NODE3 - ClusSdr
On NODE3:
TO.NODE3 - ClusRcvr
TO.NODE2 - ClusSdr
TO.NODE1 - ClusSdr
This setup is there in production from few months. Now we are seing the one issue wherein messages in these cluster queues piling up everyday at a perticular time.
Can anyone please help me to check whether the above cluster setup of 3 full repos are correct?
Thanks in advance,
Prasanna |
|
| Back to top |
|
 |
| Mr Butcher |
| Posted: Fri Sep 11, 2009 2:00 am Post subject: |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
FR queuemanagers should be full connected, so your cluster setup looks fine. it may be worth reviewing why you need 3 FRs instead of 2, but this is not related to your problem.
which queues to you mean by "these queues" ? if messages are piling up in the SYSTEM.CLUSTER.TRANSMIT.QUEUE then you should check for transfer problems / network shortage / bandwidth / concurrent file transfers / ...... . (any channel events btw ?!?)
If messages are piling up in local queues on any of these queuemanagers, then you habve a problem with the consuming application .either they are too slow (slower as the sender) or bad application design or or or.
maybe you have a missunderstanding from the workload mechanism in the cluster... ?!? what are you expecting the cluster to do if messages pile up in a local queue?
_________________
Regards, Butcher |
|
| Back to top |
|
|
| sumit |
| Posted: Fri Sep 11, 2009 2:32 am Post subject: Re: MQ Cluster setup problem |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
| bhatnprasanna wrote: |
| Now we are seing the one issue wherein messages in these cluster queues piling up everyday at a perticular time. |
In all the 3 queue managers and in which queue? What's special about this particular time, is it heavy volume of messages flow and your end site application is not able to consume it in time? What are these messages?
_________________
Regards
Sumit |
|
| Back to top |
|
|
| exerk |
| Posted: Fri Sep 11, 2009 2:35 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
You may also want to have a read of THIS. And, please, information as to whether it is a SYSTEM queue or APPLICATION queue that is filling would be of more use.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| bhatnprasanna |
| Posted: Fri Sep 11, 2009 3:26 am Post subject: |
|
|
Newbie
Joined: 11 Sep 2009
Posts: 3
|
| Mr Butcher wrote: |
FR queuemanagers should be full connected, so your cluster setup looks fine. it may be worth reviewing why you need 3 FRs instead of 2, but this is not related to your problem.
which queues to you mean by "these queues" ? if messages are piling up in the SYSTEM.CLUSTER.TRANSMIT.QUEUE then you should check for transfer problems / network shortage / bandwidth / concurrent file transfers / ...... . (any channel events btw ?!?)
If messages are piling up in local queues on any of these queuemanagers, then you habve a problem with the consuming application .either they are too slow (slower as the sender) or bad application design or or or.
maybe you have a missunderstanding from the workload mechanism in the cluster... ?!? what are you expecting the cluster to do if messages pile up in a local queue? |
Hi,
Thanks for the replies...
The queues I am talking about is the local queues which are in the cluster.
This problem of messages pilingup, happening at perticular times in a day (9am, 1pm and 6pm). All other time it is fine. When this happened last two times, restart of the full repo which hosted the cluster queues, solved the problem for around a month. After a month this prob again has started. There is no problem at the network or the load on the server. So i am thinking if there is any problem in the cluster setup. |
|
| Back to top |
|
|
| exerk |
| Posted: Fri Sep 11, 2009 3:33 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
Ensure that when it happens that all queue managers in the cluster have 'clean' communication with each other, and whether there are network/resource issues etc.
Also, consider dropping down to 2 FR's and see if the problem persists, and how many PR's are there in the cluster?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| sumit |
| Posted: Fri Sep 11, 2009 4:33 am Post subject: |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
| bhatnprasanna wrote: |
| When this happened last two times, restart of the full repo which hosted the cluster queues, solved the problem for around a month. |
Was there no way out other than starting the FR? What are these queues used for? Why application didn't pick the messages from the queue?
| bhatnprasanna wrote: |
| After a month this prob again has started. |
How are you tackling it now?
Any error logs?
_________________
Regards
Sumit |
|
| Back to top |
|
|
| Mr Butcher |
| Posted: Fri Sep 11, 2009 4:39 am Post subject: |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
* messages piling up in the local queues
* restart of FR hosting these queues fixes the issue
Sorry, but to me this is not related. When message are already in the local queue, they reached their destination. no more cluster work here to do for mqseries.
restarting that queuemanager will not make these messages being processed (they may be discarded during restart if non persistent), but maybe you had an application issue, and restarting the queuemanager also caused a restart of the application?
or .... did you expect the messages being workloaded in the cluster to 3 queuemanagers and they only hit one qmgr? in that case, yes, you may have a cluster issue. maybe you tell us more about your queue layout in the cluster.
but on the first hand ... when messages pile up on a local queue, i would look at the consuming application. is it running? if running, is it also working? are messages being read off the queue at the expected rate? any errors in application logfiles? .......
_________________
Regards, Butcher |
|
| Back to top |
|
|
| exerk |
| Posted: Fri Sep 11, 2009 4:47 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| Mr Butcher wrote: |
| ...but on the first hand ... when messages pile up on a local queue, i would look at the consuming application. is it running? if running, is it also working? are messages being read off the queue at the expected rate? any errors in application logfiles? ....... |
I'm so glad someone is awake 
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
|
|
