| Author |
Message
|
| MB |
 Posted: Tue Jun 05, 2007 10:11 am Post subject: Broker Issues |
 |
|
Acolyte
Joined: 25 Jun 2004
Posts: 52
|
Hi,
We have 6 Brokers in our Production Environment running on 6 servers. At the same time all the Brokers stopped responding to messages. The Input queue depths were increasing.
Observations:
1. The QM was up and Running.
2. We were able to connect to Broker using tool kit.
3. All the data flow engines were running.
4. No Errors in any of the MQ or MB logs.
5. No errors in errors queues.
6. No Hardware or software specific changes on servers
7. No database related errors in any of the logs
Have anyone faced this kind of issue before? Could you please let me know what was the cause and solution for the issue?
Thanks,
MB. |
|
| Back to top |
|
 |
| andypiper |
| Posted: Tue Jun 05, 2007 11:52 pm Post subject: |
|
|
Acolyte
Joined: 29 Oct 2001
Posts: 59
Location: London
|
Any messages on dead letter queues?
Any channels in retry state?
Any changes to the type of data that the brokers were being sent?
We'd need to know a lot more about your setup to help to debug. I'd suggest that there is more troubleshooting that you could do yet. |
|
| Back to top |
|
|
| elvis_gn |
| Posted: Wed Jun 06, 2007 12:03 am Post subject: |
|
|
Padawan
Joined: 08 Oct 2004
Posts: 1905
Location: Dubai
|
Hi MB,
Quite weird, but did this really happen or are you trying to do an analysis of what you would do in such a situation ? Its impossible to believe that all flows are running but not picking messages.
In any case, i'm hoping are have a clustered environment. If all the flows are running then they should definitely be picking up messages from the queues....if the depth is not decreasing, then its most probably that the flow is throwing back the message to the Input queue or looping.
Perhaps your error handling is not proper or not catching the exception....
How is the CPU doing ? It is peaking for the exe groups ?
Regards. |
|
| Back to top |
|
|
| MB |
| Posted: Wed Jun 06, 2007 2:45 pm Post subject: |
|
|
Acolyte
Joined: 25 Jun 2004
Posts: 52
|
Thanks for your responses.
CPU was idle.
No looping just that flow were not picking up the messages.
The Kind of data sent to queues is same and has not changed in the recent past.
Channels were all running fine
Dead letter queue did have some messages. But will that be the cause of the problem? |
|
| Back to top |
|
|
| jbanoop |
| Posted: Wed Jun 06, 2007 2:56 pm Post subject: |
|
|
Chevalier
Joined: 17 Sep 2005
Posts: 401
Location: SC
|
Dead letter queue if it is filling up is actually a sign that the messages are being processed ..
have a look at the init queues to see if somthing is clogging something up ..
if the 6 QMs are on the same cluster .. i would strongly suspect some issue with the cluster itself..
I am no MQ Admin but if all the QMs are on the cluster , thats the only common thing .. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Jun 07, 2007 3:03 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Quote: |
| Its impossible to believe that all flows are running but not picking messages. |
Typically this happens if somenone get inhibited the flows input queues.... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mvarghese |
| Posted: Thu Jun 07, 2007 3:59 pm Post subject: |
|
|
Centurion
Joined: 27 Sep 2006
Posts: 141
|
Since its a production Environment..looks some bug.. ...better to raise a PMR...and solve it immediately..call IBM websphere support Team <number is in IBM site>they will help u...
_________________
Jain Varghese |
|
| Back to top |
|
|
| sessionsa |
| Posted: Mon Jun 11, 2007 5:42 am Post subject: |
|
|
Novice
Joined: 03 Aug 2006
Posts: 13
Location: Lansing, MI
|
| Did you get a resolution on this. I am having the same issue. Nothing in the DLQ, the QM, CM and Brkr are running fine. I end up having to stop and start or redeploy the process. It runs once after that I have to stop and restart the process to run it again. |
|
| Back to top |
|
|
| sessionsa |
| Posted: Mon Jun 11, 2007 7:26 am Post subject: |
|
|
Novice
Joined: 03 Aug 2006
Posts: 13
Location: Lansing, MI
|
We ended up restarting the QMgr and that resolved the current issue. The only explaination I could get is that the connectivity between the QMgr, ConfigMgr and queues somehow become dis jointed and will not work until the QMgr is restarted.
Try it and see what happens.
Regards, |
|
| Back to top |
|
|
|
|
