| Author |
Message
|
| aahmad8 |
 Posted: Fri Jul 08, 2011 4:47 pm Post subject: Probe Id MC005021 MQMOnlineProcessing MQ v7.0.1.5 |
 |
|
Apprentice
Joined: 12 Sep 2005
Posts: 27
Location: Malaysia
|
MQ queue manager runs under Microsoft Cluster Server (MSCS) control. Queue manager failed over to server "B" from server "A" which cannot start the queue manager and gives FDC with probe ID MC005021 (Server "A"). Manual fail over to server "A" will bring back to server "B" as queue manager cannot be started at server "A". Below is the FDC file.
+-----------------------------------------------------------------------------+
| |
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- Sat July 09 2011 03:23:58 Malay Peninsula Standard Ti |
| UTC Time :- 1310153038.336000 |
| UTC Time Offset :- 480 ((UNKNWOWN) |
| Host Name :- xxxxxxMQ1 |
| Operating System :- Windows Ver 6.1 (7) Server Enterprise x64 Edition, |
| Build 7600 |
| PIDS :- 5724H7220 |
| LVLS :- 7.0.1.5 |
| Product Long Name :- WebSphere MQ for Windows |
| Vendor :- IBM |
| Probe Id :- MC005021 |
| Application Name :- MQM |
| Component :- MQMOnlineProcessing |
| SCCS Info :- lib/admin/pc/winnt/mscs/amqmclut/amqmdbgn.cpp, 1.8.3.1 |
| Line Number :- 134 |
| Build Date :- Apr 19 2011 |
| CMVC level :- p701-105-110419 |
| Build Type :- IKAP - (Production) |
| UserID :- SYSTEM |
| Process Name :- C:\Windows\SysWOW64\rhs.exe |
| Addressing mode :- 32-bit |
| Process :- 00004044 |
| Thread :- 00000002 |
| UserApp :- FALSE |
| Last HQC :- 0.0.0-0 |
| Last HSHMEMB :- 0.0.0-0 |
| Major Errorcode :- xecF_E_UNEXPECTED_SYSTEM_RC |
| Minor Errorcode :- OK |
| Probe Type :- MSGAMQ6119 |
| Probe Severity :- 2 |
| Probe Description :- AMQ6119: An internal WebSphere MQ error has occurred |
| (***CMQMException Caught***: Failed to start IBM MQSeries Service) |
| FDCSequenceNumber :- 0 |
| Comment1 :- ***CMQMException Caught***: Failed to start IBM |
| MQSeries Servic |
| |
+-----------------------------------------------------------------------------+
and the error log file.
----- amqxfdcp.c : 777 --------------------------------------------------------
7/9/2011 03:25:30 - Process(4044.5) User(SYSTEM) Program(rhs.exe)
Host(xxxxxxMQ1)
AMQ6183: An internal WebSphere MQ error has occurred.
EXPLANATION:
An error has been detected, and the WebSphere MQ error recording routine has
been called. The failing process is process 4044.
ACTION:
Use the standard facilities supplied with your system to record the problem
identifier and to save any generated output files. Use either the WMQ Support
site: http://www.ibm.com/software/integration/wmq/support/, or IBM Support
Assistant (ISA): http://www.ibm.com/software/support/isa/, to see whether a
solution is already available. If you are unable to find a match, contact your
IBM support center. Do not discard these files until the problem has been
resolved.
----- amqxfdcp.c : 821 --------------------------------------------------------
I cannot find any hit from internet for probe id MC005021. Is there anyone encounter similar issue or suggestion to resolve the issue?
We also applied Microsoft hot fix from http://support.microsoft.com/kb/978527 but the problem still persists. |
|
| Back to top |
|
 |
| exerk |
| Posted: Sat Jul 09, 2011 12:26 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
Is this a first-time failure, i.e. it has failed over successfully in the past but now these symptoms have occurred?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| aahmad8 |
| Posted: Sat Jul 09, 2011 7:10 am Post subject: |
|
|
Apprentice
Joined: 12 Sep 2005
Posts: 27
Location: Malaysia
|
Yes you are right. It has failed over successfully in the past but once this probe id occurred I cannot fail over back to server "A".
_________________
Past, Present & Future of MQ |
|
| Back to top |
|
|
| Vitor |
| Posted: Sat Jul 09, 2011 8:16 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| aahmad8 wrote: |
| It has failed over successfully in the past but once this probe id occurred I cannot fail over back to server "A". |
What was the original problem that triggered a failover from A -> B?
What steps did you take to resolve it, i.e. what's changed on server A since the last time the queue manager successfully started?
It's also my interpretation of that FDC that it's the MQ Service that's failing not the queue manager. So any changes in terms of server A's registry, user ids or software would be of particular interest.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| aahmad8 |
| Posted: Sat Jul 09, 2011 8:35 am Post subject: |
|
|
Apprentice
Joined: 12 Sep 2005
Posts: 27
Location: Malaysia
|
1. Trying to start sender channel from Server "A".
2. Error message appeared that I am not authorize to perform "runmqsc" command.
3. While troubleshooting failover takes place from Server "A" to Server "B".
4. Manual failover back to Server "A" from Server "B".
5. Trying to start queue manager but also message appeared saying that I am not authorize. The ID is a member of mqm group.
6. The problem did not resolve and there was no change as far as MQ is concern.
7. Queue Manager failed to start at Server "A". No changes to Server "A" registery, user id or software.
8. Yesterday the MQ was upgrade to latest fix pack 7.0.1.5 but the issue still persist. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Jul 09, 2011 8:40 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| aahmad8 wrote: |
1. Trying to start sender channel from Server "A".
2. Error message appeared that I am not authorize to perform "runmqsc" command.
3. While troubleshooting failover takes place from Server "A" to Server "B".
4. Manual failover back to Server "A" from Server "B".
5. Trying to start queue manager but also message appeared saying that I am not authorize. The ID is a member of mqm group.
6. The problem did not resolve and there was no change as far as MQ is concern.
7. Queue Manager failed to start at Server "A". No changes to Server "A" registery, user id or software.
8. Yesterday the MQ was upgrade to latest fix pack 7.0.1.5 but the issue still persist. |
Remember on Windows you need both mqm and Administrator group authorities...
Also how did you start the qmgr?
Did you try strmqm -si <qmgr>?
Sounds to me like the upgrade was run with the wrong user/permissions
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| aahmad8 |
| Posted: Sat Jul 09, 2011 8:59 am Post subject: |
|
|
Apprentice
Joined: 12 Sep 2005
Posts: 27
Location: Malaysia
|
1. Since the MQ setup is using MSCS, the ID is a member of Domain mqm and Administrator group.
2. Most of the time through MQ Explorer but sometime type strmqm <qmgr?> at command line.
3. User permission issue was happen before applying fix pack v7.0.1.5
4. I am using similar ID for both Server "A" and "B" upgrade. Server "B" is working fine.
5. Based on FDC which gave Probe ID MC005021, I suspect C:\Windows\SysWOW64\rhs.exe is giving me problem starting queue manager in Server "A". Similar filename also appear in AMQERR01.LOG file:
7/9/2011 03:25:30 - Process(4044.5) User(SYSTEM) Program(rhs.exe)
Host(xxxxxxMQ1)
AMQ6183: An internal WebSphere MQ error has occurred. |
|
| Back to top |
|
|
| aahmad8 |
| Posted: Sat Jul 09, 2011 10:02 am Post subject: |
|
|
Apprentice
Joined: 12 Sep 2005
Posts: 27
Location: Malaysia
|
I am getting the following message after issuing strmqm -si <qmgr> at command line.
AMQ7257: The MQ service must be running to perform the requested operation.
C:\Users\mymq>net start MQSeriesServices
The service is starting or stopping. Please try again later.
C:\Users\mymq>net start MQSeriesServices
The IBM MQSeries service is starting.....................................
The IBM MQSeries service could not be started.
A service specific error occurred: 2195.
More help is available by typing NET HELPMSG 3547.
_________________
Past, Present & Future of MQ |
|
| Back to top |
|
|
| exerk |
| Posted: Sat Jul 09, 2011 12:44 pm Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| aahmad8 wrote: |
| 2. Most of the time through MQ Explorer but sometime type strmqm <qmgr?> at command line. |
No, no, and no again. This is an MSCS controlled queue manager, so nothing you do do to this queue manager should be done through the command line - only through the MSCS cluster manager snap-in.
| aahmad8 wrote: |
| 3. User permission issue was happen before applying fix pack v7.0.1.5 |
What user permission issue? You didn't state there was such an issue in your original or subsequent posts.
| aahmad8 wrote: |
| 4. I am using similar ID for both Server "A" and "B" upgrade. Server "B" is working fine. |
Similar, or the same userid on each node?
| aahmad8 wrote: |
| 5. Based on FDC which gave Probe ID MC005021, I suspect C:\Windows\SysWOW64\rhs.exe is giving me problem starting queue manager in Server "A". Similar filename also appear in AMQERR01.LOG file: |
That's the Resource Hosting Subsystem executable, so I'd ask the System Administrators about recent changes made to each/either node recently.
Your only resolution may be to be to drop the queue manager from MSCS control, blow away the resource group, and reinstate the resource group - plan for that eventuality.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| aahmad8 |
| Posted: Sat Jul 09, 2011 2:15 pm Post subject: |
|
|
Apprentice
Joined: 12 Sep 2005
Posts: 27
Location: Malaysia
|
| exerk wrote: |
| No, no, and no again. This is an MSCS controlled queue manager, so nothing you do do to this queue manager should be done through the command line - only through the MSCS cluster manager snap-in. |
Thanks. Take note of it.
| exerk wrote: |
| What user permission issue? You didn't state there was such an issue in your original or subsequent posts. |
I didn't state as I suspected our security team has reinstate the authority require to administer queue manager. I do not have authorization issue thereafter.
| exerk wrote: |
| Similar, or the same userid on each node? |
Same user ID.
| exerk wrote: |
| That's the Resource Hosting Subsystem executable, so I'd ask the System Administrators about recent changes made to each/either node recently. |
There was no change according to System Administrator. The only changes was applying HotFix "The Resource Hosting Subsystem (Rhs.exe) process stops unexpectedly when you start a cluster resource in Windows Server 2008 R2" which can be found at http://support.microsoft.com/kb/978527 after looking at MQ FDC and error log file.
| exerk wrote: |
| Your only resolution may be to be to drop the queue manager from MSCS control, blow away the resource group, and reinstate the resource group - plan for that eventuality. |
Can I maintain the queue manager in SAN storage without recreating it? |
|
| Back to top |
|
|
| exerk |
| Posted: Sat Jul 09, 2011 2:27 pm Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| aahmad8 wrote: |
| Can I maintain the queue manager in SAN storage without recreating it? |
If you can I suggest you migrate the queue manger out of MSCS control, to local storage, then delete/recreate the MSCS resource group and migrate the queue manager back to MSCS control.
Can you please clarify your statement regarding what the Sys Admins did to 'reinstate' your authorities, thank you?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Sun Jul 10, 2011 2:41 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Before you go messing with the QM, or removing resources from MSCS....
On Server A, is the MQ Service set to Automatic and is it Running in the Windows Services window?
If not, can you set it to Automatic and / or start it?
When you say it is the same ID, is it the SAME domain ID used across both servers, or is it two different ID on two different servers that just happen to have the same friendly name (but the underlying Windows SIDs are different)?
I assume you've opened a PMR with IBM Support already. What are they saying?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
|
|
