| Author |
Message
|
| ccooper |
 Posted: Mon Jul 26, 2010 8:06 am Post subject: Unable to start Queue Manager - Windows Platform |
 |
|
Newbie
Joined: 26 Jul 2010
Posts: 9
|
Hello,
I have a server that is running Windows Server 2003 and has MQ installed on it. This server used to be a domain controller but was recently demoted to a member server. Once the server was demoted MQ stopped running and will no longer start. When I try to start it I get the following message:
****************************************
* Command: amqmdain qmgr start S38840001
****************************************
WebSphere MQ queue manager 'S38840001' starting.
WebSphere MQ was unable to display an error message 893.
exitvalue = 71
I checked the AMQ2596.0.FDC log file and found the following entry:
+-----------------------------------------------------------------------------+
| |
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- Mon July 26 11:58:07 Eastern Daylight Time 2010 |
| Host Name :- S38840001 (Windows Server 2003, Build 3790: Service |
| Pack 2) |
| PIDS :- 5724H7200 |
| LVLS :- 6.0.2.2 |
| Product Long Name :- WebSphere MQ for Windows |
| Vendor :- IBM |
| Probe Id :- ZX000001 |
| Application Name :- MQM |
| Component :- ExecCtrlrMain |
| SCCS Info :- cmd/zmain/amqzxma0.c, 1.192.1.9 |
| Line Number :- 852 |
| Build Date :- Aug 1 2007 |
| CMVC level :- p600-202-070801 |
| Build Type :- IKAP - (Production) |
| UserID :- MUSR_MQADMIN_RETAIL |
| Process Name :- C:\MQM\bin\amqzxma0.exe |
| Addressing mode :- 32-bit |
| Process :- 00002596 |
| Thread :- 00000001 |
| QueueManager :- S38840001 |
| ConnId(1) IPCC :- 2 |
| ConnId(2) QM :- 2 |
| ConnId(3) QM-P :- 2 |
| ConnId(4) App :- 2 |
| Major Errorcode :- xecF_E_UNEXPECTED_RC |
| Minor Errorcode :- MQRC_UNEXPECTED_ERROR |
| Probe Type :- MSGAMQ6118 |
| Probe Severity :- 2 |
| Probe Description :- AMQ6118: An internal WebSphere MQ error has occurred |
| (893) |
| FDCSequenceNumber :- 4 |
| Arith1 :- 2195 893 |
| |
+-----------------------------------------------------------------------------+
MQM Function Stack
ExecCtrlrMain
xcsFFST
MQM Trace History
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsCheckPointer
---} xcsCheckPointer rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsCheckPointer
---} xcsCheckPointer rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsIsEnvironment
---} xcsIsEnvironment rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsFmtOpen
---} xcsFmtOpen rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsReallocMem
---} xcsReallocMem rc=OK
---{ xcsFmtClose
---} xcsFmtClose rc=OK
---{ xcsReleaseThreadMutexSem
---} xcsReleaseThreadMutexSem rc=OK
---{ xgmWriteNTEventLog
----{ xgmCheckMsgRequired
----} xgmCheckMsgRequired rc=OK
----{ xgmCheckMsgSource
----} xgmCheckMsgSource rc=OK
----{ xeeConvertInserts
-----{ xcsGetMem
-----} xcsGetMem rc=OK
----} xeeConvertInserts rc=OK
----{ xcsFreeMem
----} xcsFreeMem rc=OK
---} xgmWriteNTEventLog rc=OK
---{ xcsFreeMem
---} xcsFreeMem rc=OK
---{ xcsFreeMem
---} xcsFreeMem rc=OK
--} xcsFFST rc=OK
-} kpiStartup rc=krcE_UNEXPECTED_ERROR
-{ xcsFFST
MQ is installed before it is promoted to a domain controller. Once promoted we configure all the trigger services and define the queue manager and queues. At this point MQ is fine and has no issues.
However, like I stated above, once we demote the server MQ is no longer functional.
Below are a few lines from the AMQERR01.log file. Please let me know if anyone has encountered this before and what I could possibly try to resolve this.
----- amqxfdcp.c : 804 --------------------------------------------------------
7/26/2010 11:58:07 - Process(2596.1) User(MUSR_MQADMIN_RETAIL) Program(amqzxma0.exe)
AMQ6118: An internal WebSphere MQ error has occurred (893)
EXPLANATION:
An error has been detected, and the MQ error recording routine has been called.
ACTION:
Use the standard facilities supplied with your system to record the problem
identifier, and to save the generated output files. Contact your IBM support
center. Do not discard these files until the problem has been resolved.
----- amqxfdcp.c : 766 --------------------------------------------------------
7/26/2010 11:58:07 - Process(2596.1) User(MUSR_MQADMIN_RETAIL) Program(amqzxma0.exe)
AMQ6184: An internal WebSphere MQ error has occurred on queue manager
S38840001.
EXPLANATION:
An error has been detected, and the WebSphere MQ error recording routine has
been called. The failing process is process 2596.
ACTION:
Use the standard facilities supplied with your system to record the problem
identifier, and to save the generated output files. Contact your IBM support
center. Do not discard these files until the problem has been resolved.
----- amqxfdcp.c : 804 --------------------------------------------------------
7/26/2010 11:58:11 - Process(8112.3) User(MUSR_MQADMIN_RETAIL) Program(amqmsrvn.exe)
AMQ7880: Error code 71 starting S38840001/Queue Manager WebSphere MQ service.
EXPLANATION:
The service was unable to start S38840001/Queue Manager. The error message
reported was as follows: WebSphere MQ queue manager 'S38840001' starting.
WebSphere MQ was unable to display an error message 893.
ACTION:
Use WebSphere MQ Explorer to investigate why the service could not begin. If
recovery for this service is active, MQ will attempt to recover. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Mon Jul 26, 2010 8:18 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Is the user MUSR_MQADMIN_RETAIL domain or local? Has the demotion affected his rights and/or permissions? Could his SID have changed?
The WMQ error indicates corrupt or inaccessable logs. If the user does have all the right permissions to run the queue manager & access the files, you could try a restore, or rebuilding the logs a la cold start (as described on this forum from time to time).
But that should be a last resort.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| exerk |
| Posted: Mon Jul 26, 2010 8:18 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
No need to paste the whole function stack, just the header will do...
Can the user (MUSR_MQADMIN_RETAIL) be resolved by the demoted server? How was WMQ originally installed, i.e. in domain mode, or 'local' mode?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| ccooper |
| Posted: Mon Jul 26, 2010 8:40 am Post subject: |
|
|
Newbie
Joined: 26 Jul 2010
Posts: 9
|
The MUSR_MQADMIN_RETAIL is a domain user and can be resolved by the server. I double checked this by performing a dsquery and the net user commad. I do not beleive his SID should have changed. One thing I did notice is that once the demotionwas complete I tried to start it the first time and the incorrect identity error displayed. I went into DCOM and made sure the identity was correct. I also re-applied the password. Once I did that I started receiving the errors from my previous response.
MQ is installed after the serverhas joined our domain. I was also leaning toward some security issue but do not know where else to look? |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jul 26, 2010 8:48 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ccooper wrote: |
| I was also leaning toward some security issue but do not know where else to look? |
It is notoriously difficult to run the WMQ service as a domain user. Despite you "fixing" it I don't think the service believes it. I certainly don't think the access to the log files is working properly.
Are you stopping and starting the service or rebooting the machine? If you're restarting the machine (or if you ever restart the machine) you may get this problem if WMQ tries to start before the DOCM & associated services have connected to the domain controller. Not a problem where WMQ is running on the domain controller of course!
If it was me, I'd run the services as a local user. Much easier & safer.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| ccooper |
| Posted: Mon Jul 26, 2010 8:52 am Post subject: |
|
|
Newbie
Joined: 26 Jul 2010
Posts: 9
|
| The IBM MQSeries service runs under the Local System account. I can stop/start the service without any issues. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jul 26, 2010 9:00 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ccooper wrote: |
| The IBM MQSeries service runs under the Local System account. I can stop/start the service without any issues. |
So who's this MUSR_MQADMIN_RETAIL user? What's he supposed to be doing architectually.
I also wouldn't use LocalSystem to run the service. Define a local user and use that.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Jul 26, 2010 9:03 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| ccooper wrote: |
| The IBM MQSeries service runs under the Local System account. I can stop/start the service without any issues. |
So who's this MUSR_MQADMIN_RETAIL user? What's he supposed to be doing architectually.
I also wouldn't use LocalSystem to run the service. Define a local user and use that. |
More specifically, the IBM MQSeries service likely doesn't actually RUN as the Local System user.
It runs as the user setup in the DCOM Config. It's just started by the local system user and then forks itself to the user specified in the DCOM Config.
Rerun the Prepare WebSphere MQ wizard and specify a new and valid user. Or use amqmdain to reset the same. |
|
| Back to top |
|
|
| exerk |
| Posted: Mon Jul 26, 2010 9:04 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| ccooper wrote: |
| The IBM MQSeries service runs under the Local System account. I can stop/start the service without any issues. |
OK, now I'm confused! You have the service running under a local user, but a domain user is trying to start the queue manager - and presumably your set-up is such that local user accounts are authorized to query the group membership of the domain user accounts?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| ccooper |
| Posted: Mon Jul 26, 2010 10:05 am Post subject: |
|
|
Newbie
Joined: 26 Jul 2010
Posts: 9
|
Ok, it looks like I got it to start. It seems I overlooked the permissions on the MQMDATA directory. The local mqm group was missing. I added the local mqm group and added the musr_mqadmin_retail user to the local group. Restarted the service and then started the Queue Manager.
To answer the questions asked....
We do not have many local accounts and they do not have the ability to query group membership of domain user accounts. The musr_mqadmin_retail is the account that starts the queue manager. I believe mqjeff explained it in an above post.
I have messed with this server quite a bit and want to test this again to see if it was just missig that local mqm group all along? I think vitor was in the right direction with the changing of the SID. I would of assumed it would of given a better error message than what was given from exlorer?
Hopefully this resolves the problem. Thanks everyone for your help. This was my first post and will definitely be back again if I have another queston. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jul 26, 2010 10:30 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ccooper wrote: |
| I would of assumed it would of given a better error message than what was given from exlorer? |
Bad assumption. In cases like this Windows doesn't return too much helpful information to the calling application.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
