| Author |
Message
|
| vaishnav.piyush |
 Posted: Tue Aug 18, 2009 12:10 am Post subject: Queue Manager Restarts -- OS Cluster in Windows 2003 |
 |
|
Novice
Joined: 16 Mar 2009
Posts: 14
|
Hi
I am facing a peculiar problem where Queue Manager Restarts itself approximately the same time everyday. Same problem I am facing on two different Queue Managers which are running on OS Level Cluster with two nodes each on Windows 2003.
The queue manager logs (below pasted) show that the queue manager was restarted but no particular reason seems to be logged. In one of them, MQRC 2009 is logged at the end (after QM is ended), which is probably because a call to the QM object was rejected as it has ended. In other case it is MQRC 2162 while the QM is still Queiscing.
Below are the logs from one of the two QMs:
-------------------------------------------------------------------------------
8/8/2009 23:47:36 - Process(10732.133) User(MUSR_MQADMIN) Program(amqrmppa.exe)
AMQ9001: Channel 'MQMB_ESEALMW577_1' ended normally.
EXPLANATION:
Channel 'MQMB_ESEALMW577_1' ended normally.
ACTION:
None.
-------------------------------------------------------------------------------
8/9/2009 00:02:56 - Process(6988.1) User(MUSR_MQADMIN) Program(amqrrmfa.exe)
AMQ9411: Repository manager ended normally.
EXPLANATION:
The repository manager ended normally.
ACTION:
None.
-------------------------------------------------------------------------------
8/9/2009 00:02:57 - Process(12960.1) User(MUSR_MQADMIN) Program(runmqchi.exe)
AMQ9542: Queue manager is ending.
EXPLANATION:
The program will end because the queue manager is quiescing.
ACTION:
None.
----- amqrimna.c : 941 --------------------------------------------------------
8/9/2009 00:02:57 - Process(11000.1) User(MUSR_MQADMIN) Program(amqzmgr0.exe)
AMQ5029: The Server 'TriggerMonitor' has ended. ProcessId(11728).
EXPLANATION:
The Server process has ended.
ACTION:
None.
-------------------------------------------------------------------------------
8/9/2009 00:02:57 - Process(11000.1) User(MUSR_MQADMIN) Program(amqzmgr0.exe)
AMQ5023: The Channel Initiator has ended. ProcessId(12960).
EXPLANATION:
The Channel Initiator process has ended.
ACTION:
None.
-------------------------------------------------------------------------------
8/9/2009 00:02:57 - Process(6620.1) User(MUSR_MQADMIN) Program(amqzmuc0.exe)
AMQ5041: The Queue Manager task 'ASYNCQ' has ended.
EXPLANATION:
The Queue Manager task ASYNCQ has ended.
ACTION:
None.
-------------------------------------------------------------------------------
8/9/2009 00:02:57 - Process(11000.1) User(MUSR_MQADMIN) Program(amqzmgr0.exe)
AMQ5025: The Command Server has ended. ProcessId(11584).
EXPLANATION:
The Command Server process has ended.
ACTION:
None.
-------------------------------------------------------------------------------
8/9/2009 00:02:57 - Process(13352.1) User(MUSR_MQADMIN) Program(amqzmur0.exe)
AMQ5041: The Queue Manager task 'ERROR-LOG' has ended.
EXPLANATION:
The Queue Manager task ERROR-LOG has ended.
ACTION:
None.
-------------------------------------------------------------------------------
8/9/2009 00:02:58 - Process(11000.1) User(MUSR_MQADMIN) Program(amqzmgr0.exe)
AMQ5027: The Listener 'LISTENER.TCP' has ended. ProcessId(13292).
EXPLANATION:
The Listener process has ended.
ACTION:
None.
-------------------------------------------------------------------------------
8/9/2009 00:03:00 - Process(7832.1) User(MUSR_MQADMIN) Program(runmqchl.exe)
AMQ9509: Program cannot open queue manager object.
EXPLANATION:
The attempt to open either the queue or queue manager object 'ESEALMW577' on
queue manager 'ESEALMW577' failed with reason code 2009.
ACTION:
Ensure that the queue is available and retry the operation.
----- amqrevna.c : 254 --------------------------------------------------------
8/9/2009 00:03:00 - Process(7832.1) User(MUSR_MQADMIN) Program(runmqchl.exe)
AMQ9999: Channel program ended abnormally.
EXPLANATION:
Channel program 'ESEALMW577_MQMB_1' ended abnormally.
ACTION:
Look at previous error messages for channel program 'ESEALMW577_MQMB_1' in the
error files to determine the cause of the failure.
----- amqrccca.c : 777 --------------------------------------------------------
8/9/2009 00:03:03 - Process(10024.1) User(MUSR_MQADMIN) Program(amqzxma0.exe)
AMQ8004: WebSphere MQ queue manager 'ESEALMW577' ended.
EXPLANATION:
WebSphere MQ queue manager 'ESEALMW577' ended.
ACTION:
None.
-------------------------------------------------------------------------------
Does anybody know the solution to this problem ?
Do tell me if you need to know the OS Event Logs and Cluster Configuration.
Thanks in Advance
Piyush Vaishnav |
|
| Back to top |
|
 |
| exerk |
| Posted: Tue Aug 18, 2009 12:19 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
Go talk to the Sys Admins and ask what is scheduled around that time. A controlled shut-down of the queue manager will only give you what you see in the logs - what else could it give you?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| vaishnav.piyush |
| Posted: Tue Aug 18, 2009 2:02 am Post subject: |
|
|
Novice
Joined: 16 Mar 2009
Posts: 14
|
Already checked, there is no event scheduled on the Server for that time.
Moreover, we are facing the same issue on two different independent cluster servers (with two nodes each) both running Windows 2003.
And the time when the restarts happens is not exactly the same each time it happens (not everyday exactly but once in two or three days ramdomly), but mostly varies in 2 Hrs of period. |
|
| Back to top |
|
|
| exerk |
| Posted: Tue Aug 18, 2009 2:26 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
Any FDC's? Anything in EventVwr?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| vaishnav.piyush |
| Posted: Tue Aug 18, 2009 2:56 am Post subject: |
|
|
Novice
Joined: 16 Mar 2009
Posts: 14
|
| There are no FDCs generated. However the event viewer (System) shows that the MQ Cluster Resource failed and was unavailable during that time. |
|
| Back to top |
|
|
| exerk |
| Posted: Tue Aug 18, 2009 3:03 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
And the cause of that failure?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Aug 18, 2009 3:13 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| exerk wrote: |
| And the cause of that failure? |
Are we sure this is a failure? The qmgr seems to enter a controlled shutdown and it happens at the same time every day... a daily scheduled job?? Not necessarily an MQ job but a job using resources making the hardware cluster believe that MQ is unavailable?? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| vaishnav.piyush |
| Posted: Tue Aug 18, 2009 6:17 am Post subject: |
|
|
Novice
Joined: 16 Mar 2009
Posts: 14
|
The Application Event Logs show that the problem was initiated by the fact that a particular remote host was not available due to with the respective sender channel ended abnormally. However, this remote host is MQ hub and is available 24/7 without faliure. Also the System Admin here says that there is no network problem on this server at this time.
Moreover, we are facing similar restart problem on another hardware cluster with same OS and cluster config.
So it seems to me that the issue has something to do with a possible bug in MQ in this particular cluster.
The MQ version on both clusters is 6.2.2.0.
Does anybody know about any bug which was fixed in later fix-packs / service-packs..
We need to be sure that installing a service pack will solve the problem, before we request for it.
Guys, thanks so much for your support. |
|
| Back to top |
|
|
| exerk |
| Posted: Tue Aug 18, 2009 6:25 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| vaishnav.piyush wrote: |
| ...The MQ version on both clusters is 6.2.2.0... |
I'm assuming you mean 6.0.2.2
| vaishnav.piyush wrote: |
| ...We need to be sure that installing a service pack will solve the problem, before we request for it. |
I suspect that you will be advised to apply the latest FixPack (6.0.2.7 IIRC), and see if it clears the problem, so you may wish to pre-empt by going to it.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| sumit |
| Posted: Tue Aug 18, 2009 6:35 am Post subject: |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
Hard to believe that a bug can lead to a clean shutdown of a queue manager. 
_________________
Regards
Sumit |
|
| Back to top |
|
|
| vaishnav.piyush |
| Posted: Tue Aug 18, 2009 11:42 pm Post subject: |
|
|
Novice
Joined: 16 Mar 2009
Posts: 14
|
| Yes the version is 6.0.2.2. |
|
| Back to top |
|
|
| sri |
| Posted: Thu May 20, 2010 2:08 am Post subject: Re: Queue Manager Restarts -- OS Cluster in Windows 2003 |
|
|
Apprentice
Joined: 14 Mar 2004
Posts: 32
Location: Chennai - India
|
Hi
We have similar issue.. do you have the solution for this issue.. or atleast the exact cause for the event
AMQ9411: Repository manager ended normally
thanks
Srini
| vaishnav.piyush wrote: |
Hi
I am facing a peculiar problem where Queue Manager Restarts itself approximately the same time everyday. Same problem I am facing on two different Queue Managers which are running on OS Level Cluster with two nodes each on Windows 2003.
The queue manager logs (below pasted) show that the queue manager was restarted but no particular reason seems to be logged. In one of them, MQRC 2009 is logged at the end (after QM is ended), which is probably because a call to the QM object was rejected as it has ended. In other case it is MQRC 2162 while the QM is still Queiscing.
Below are the logs from one of the two QMs:
-------------------------------------------------------------------------------
8/8/2009 23:47:36 - Process(10732.133) User(MUSR_MQADMIN) Program(amqrmppa.exe)
AMQ9001: Channel 'MQMB_ESEALMW577_1' ended normally.
EXPLANATION:
Channel 'MQMB_ESEALMW577_1' ended normally.
ACTION:
None.
-------------------------------------------------------------------------------
8/9/2009 00:02:56 - Process(6988.1) User(MUSR_MQADMIN) Program(amqrrmfa.exe)
AMQ9411: Repository manager ended normally.
EXPLANATION:
The repository manager ended normally.
ACTION:
None.
-------------------------------------------------------------------------------
8/9/2009 00:02:57 - Process(12960.1) User(MUSR_MQADMIN) Program(runmqchi.exe)
AMQ9542: Queue manager is ending.
EXPLANATION:
The program will end because the queue manager is quiescing.
ACTION:
None.
----- amqrimna.c : 941 --------------------------------------------------------
8/9/2009 00:02:57 - Process(11000.1) User(MUSR_MQADMIN) Program(amqzmgr0.exe)
AMQ5029: The Server 'TriggerMonitor' has ended. ProcessId(11728).
EXPLANATION:
The Server process has ended.
ACTION:
None.
-------------------------------------------------------------------------------
8/9/2009 00:02:57 - Process(11000.1) User(MUSR_MQADMIN) Program(amqzmgr0.exe)
AMQ5023: The Channel Initiator has ended. ProcessId(12960).
EXPLANATION:
The Channel Initiator process has ended.
ACTION:
None.
-------------------------------------------------------------------------------
8/9/2009 00:02:57 - Process(6620.1) User(MUSR_MQADMIN) Program(amqzmuc0.exe)
AMQ5041: The Queue Manager task 'ASYNCQ' has ended.
EXPLANATION:
The Queue Manager task ASYNCQ has ended.
ACTION:
None.
-------------------------------------------------------------------------------
8/9/2009 00:02:57 - Process(11000.1) User(MUSR_MQADMIN) Program(amqzmgr0.exe)
AMQ5025: The Command Server has ended. ProcessId(11584).
EXPLANATION:
The Command Server process has ended.
ACTION:
None.
-------------------------------------------------------------------------------
8/9/2009 00:02:57 - Process(13352.1) User(MUSR_MQADMIN) Program(amqzmur0.exe)
AMQ5041: The Queue Manager task 'ERROR-LOG' has ended.
EXPLANATION:
The Queue Manager task ERROR-LOG has ended.
ACTION:
None.
-------------------------------------------------------------------------------
8/9/2009 00:02:58 - Process(11000.1) User(MUSR_MQADMIN) Program(amqzmgr0.exe)
AMQ5027: The Listener 'LISTENER.TCP' has ended. ProcessId(13292).
EXPLANATION:
The Listener process has ended.
ACTION:
None.
-------------------------------------------------------------------------------
8/9/2009 00:03:00 - Process(7832.1) User(MUSR_MQADMIN) Program(runmqchl.exe)
AMQ9509: Program cannot open queue manager object.
EXPLANATION:
The attempt to open either the queue or queue manager object 'ESEALMW577' on
queue manager 'ESEALMW577' failed with reason code 2009.
ACTION:
Ensure that the queue is available and retry the operation.
----- amqrevna.c : 254 --------------------------------------------------------
8/9/2009 00:03:00 - Process(7832.1) User(MUSR_MQADMIN) Program(runmqchl.exe)
AMQ9999: Channel program ended abnormally.
EXPLANATION:
Channel program 'ESEALMW577_MQMB_1' ended abnormally.
ACTION:
Look at previous error messages for channel program 'ESEALMW577_MQMB_1' in the
error files to determine the cause of the failure.
----- amqrccca.c : 777 --------------------------------------------------------
8/9/2009 00:03:03 - Process(10024.1) User(MUSR_MQADMIN) Program(amqzxma0.exe)
AMQ8004: WebSphere MQ queue manager 'ESEALMW577' ended.
EXPLANATION:
WebSphere MQ queue manager 'ESEALMW577' ended.
ACTION:
None.
-------------------------------------------------------------------------------
Does anybody know the solution to this problem ?
Do tell me if you need to know the OS Event Logs and Cluster Configuration.
Thanks in Advance
Piyush Vaishnav |
|
|
| Back to top |
|
|
| sumit |
| Posted: Fri May 21, 2010 6:27 am Post subject: |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
And have you also checked that there is no scheduled Job or activity running which may be responsible?
More info like OS and MQ version?
_________________
Regards
Sumit |
|
| Back to top |
|
|
| BBM |
| Posted: Mon May 24, 2010 12:57 pm Post subject: |
|
|
Master
Joined: 10 Nov 2005
Posts: 217
Location: London, UK
|
Although you have stated that its a clean shutdown I would look at the possibility of a memory leak on the server. This being Windows I've seen memory leaks many times that affect MQ adversely.
Try firing up task manager and monitor handle count and memory for all processes.
I would also check other obvious things like disk space for the volume containing the logs.
BBM |
|
| Back to top |
|
|
| raz |
| Posted: Tue May 25, 2010 7:43 am Post subject: |
|
|
Acolyte
Joined: 10 Jul 2007
Posts: 70
|
| what version of MQ you are using? |
|
| Back to top |
|
|
|
|
