| Author |
Message
|
| MQChela |
 Posted: Mon Feb 13, 2012 12:32 pm Post subject: Queue Manager stays up only in interactive session |
 |
|
Novice
Joined: 07 Feb 2006
Posts: 20
|
Hi all,
We just moved to our DR site. When Windows 2K3 server was rebooted post migration, lone queue manager on the server did not come up. Below is top section of FDC file created:
+----------------------------------------------------+
| WebSphere MQ First Failure Symptom Report
==============================
| Date/Time :- Fri February 10 19:29:15 Eastern Standard Time 2012
| Host Name :- DR_SERVER (Windows Ver 5.2 Build 3790: Service Pack 2)
| PIDS :- 5724H7200
| LVLS :- 6.0.0.0
| Product Long Name :- WebSphere MQ for Windows
| Vendor :- IBM
| Probe Id :- HL081010
| Application Name :- MQM
| Component :- mqlOpenLogControlFile
| SCCS Info :- lib/logger/amqholf0.c, 1.28
| Line Number :- 973
| Build Date :- May 19 2005
| CMVC level :- p000-L050519
| Build Type :- IKAP - (Production)
| UserID :- MUSR_MQADMIN
| Process Name :- C:\Program Files\IBM\WebSphere MQ\bin\amqzxma0.exe |
| Process :- 00006780
| Thread :- 00000001
| QueueManager :- Q_MGR_1
| ConnId(1) IPCC :- 2
| ConnId(2) QM :- 2
| ConnId(3) QM-P :- 2
| ConnId(4) App :- 2
| Major Errorcode :- xecF_E_UNEXPECTED_RC
| Minor Errorcode :- hrcE_MQLO_ACCD
| Probe Type :- MSGAMQ6118
| Probe Severity :- 2
| Probe Description :- AMQ6118: An internal WebSphere MQ error has occurred
| (20806822)
| FDCSequenceNumber :- 0
| Arith1 :- 545286178 20806822
When logged in, and started from command line using strmqm, queue manager stays up. We had to keep the console locked.
We checked using DCOMCNFG and MQ service is configured to run under MUSR_MQADMIN (not interactive user). Comparison with primary server revealed that local MQM group did not have correct permissions to folder for queue manager objects (it's on different drive that MQ Software). But even after fixing these permission, the error persists. In fact now there are 3 FDCs getting created.
Any suggestions?
TIA |
|
| Back to top |
|
 |
| Vitor |
| Posted: Mon Feb 13, 2012 12:46 pm Post subject: Re: Queue Manager stays up only in interactive session |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| MQChela wrote: |
| Any suggestions? |
Apply some maintenance; that's a base version of WMQ.
Better still, upgrade to a modern version.
Then work out what's different between the Windows confirguration this normally runs under & the one at the DR site. My guess would be the domains are not the same & the SIDs don't match up.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| MQChela |
| Posted: Mon Feb 13, 2012 12:50 pm Post subject: Re: Queue Manager stays up only in interactive session |
|
|
Novice
Joined: 07 Feb 2006
Posts: 20
|
| Vitor wrote: |
| MQChela wrote: |
| Any suggestions? |
Apply some maintenance; that's a base version of WMQ.
Better still, upgrade to a modern version.
Then work out what's different between the Windows confirguration this normally runs under & the one at the DR site. My guess would be the domains are not the same & the SIDs don't match up. |
It's too late to apply service packs. Primary site also has exact same version and has been running fine. Domains are same.
Queue manager data files are replicated from primary server on a daily basis. Could that be a problem? |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Feb 13, 2012 12:57 pm Post subject: Re: Queue Manager stays up only in interactive session |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| MQChela wrote: |
| Domains are same. |
I've heard that before from some Windows admins. They may have the same names, but are they the same at a SID level?
| MQChela wrote: |
| Queue manager data files are replicated from primary server on a daily basis. Could that be a problem? |
It wouldn't be my first guess. That error is a fault trying to allocate a file handle to the queue manager logs, not a problem with the logs themselves. I see 3 principle sources of trouble:
- The SIDs are hosed up and the user id can't get to the files
- The user on the DR site is incorrectly configured & the log path is hosed up
- It's the bug in WMQ that results in that problem which was fixed in V6.0.1.0, which because no 2 Windows installs are identical doesn't affect your primary site due to some subtlety in the configuration.
Again, see what's different in the Windows installs. Not just the WMQ installs on Windows.
_________________
Honesty is the best policy.
Insanity is the best defence.
Last edited by Vitor on Mon Feb 13, 2012 12:58 pm; edited 1 time in total |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Feb 13, 2012 12:58 pm Post subject: Re: Queue Manager stays up only in interactive session |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| MQChela wrote: |
| Queue manager data files are replicated from primary server on a daily basis. Could that be a problem? |
Yes, because you likely replicated them while the queue manager was running.
In which case the files are in an inconsistent state. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Feb 13, 2012 1:00 pm Post subject: Re: Queue Manager stays up only in interactive session |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| Yes, because you likely replicated them while the queue manager was running. |
Good point. Dim enough to be using V6 base, dim enough to copy files while they're being used.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| exerk |
| Posted: Mon Feb 13, 2012 1:19 pm Post subject: Re: Queue Manager stays up only in interactive session |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| Vitor wrote: |
| mqjeff wrote: |
| Yes, because you likely replicated them while the queue manager was running. |
Good point. Dim enough to be using V6 base, dim enough to copy files while they're being used. |
But if they're using SAN replication they'll have little choice. About the only way I've seen to get this to work is to explicitly set FULL CONTROL permissions to the directories, for the user under which WMQ runs, in the OP's case, MUSR_MQADMIN. I neither encourage nor endorse what is being done by the OP, but sometimes people get painted into corners.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Feb 13, 2012 3:30 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
What happens when you start the QM using the amqmdain command and then log off?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Tue Feb 14, 2012 3:11 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| Quote: |
| When logged in, and started from command line using strmqm, queue manager stays up. We had to keep the console locked. |
You should start Windows qmgrs using amqmdain qmgr start, not strmqm. If the qmgr is set up for automatic start up using amqmdain auto, it should start immediately after the MQ service starts (at system boot or using amqsvc -start command or via Windows Services window).
_________________
Glenn |
|
| Back to top |
|
|
| MQChela |
| Posted: Wed Feb 15, 2012 5:08 pm Post subject: Re: Queue Manager stays up only in interactive session |
|
|
Novice
Joined: 07 Feb 2006
Posts: 20
|
Sorry folks. Got busy with other issues.
Unfortunately applying fix packs or version is not an option right now since we are slated to upgrade to W2K8R2 and MQ7 soon. I could be wrong but I didn't see any fix packs addressing this issue explicitely.
What I found was local user MUSR_MQADMIN was part of local administrators group on primary server but not on DR server. We duplicated this on a test server and were able to fix by adding MUSR_MQADMIN to local admin group, restarting "IBM MQSeries" service and starting queue manager in MQExplorer. It stayed up even after logging off.
When we tried same exact fix on DR server, it didn't work. We attempted to start queue manager from MQExplorer (which I knew fail earlier) and got same (?) error.
****************************************
Command: amqmdain qmgr start Q_MGR_1
****************************************
WebSphere MQ queue manager 'Q_MGR_1' starting.
AMQ6109: An internal WebSphere MQ error has occurred.
exitvalue = 71
Interestingly, FDCs are still showing AMQ6118
For now, we are back to starting queue manager by interactive user |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Feb 16, 2012 2:46 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Did you restart the machine after adding the user to the local admins group? |
|
| Back to top |
|
|
| JasonE |
| Posted: Thu Feb 16, 2012 4:59 am Post subject: |
|
|
Grand Master
Joined: 03 Nov 2003
Posts: 1220
Location: Hursley
|
>hrcE_MQLO_ACCD
>mqlOpenLogControlFile
>UserID :- MUSR_MQADMIN
Means access is denied when MUSR_MQADMIN is opening the .LFH file.
A simple way to see what is going on would be to use process monitor on the failing server, and log the act of amqmdain start Q_MGR_1, and look for access denied errors, especially on that file.
In effect, on the DR server, both the local MQM group and the local MUSR_MQADMIN user are different SID's from the primary server. (There is also not usually the need to put the MQ configured userid into the local administrators group)
Once you make the change, restarting the service may not be enough as things like the taskbar will hold open the amqmsrvn dcom object - you want to ensure the amqmsrvn.exe process isnt running for the change to be effective, I believe. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Feb 16, 2012 5:12 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| JasonE wrote: |
| In effect, on the DR server, both the local MQM group and the local MUSR_MQADMIN user are different SID's from the primary server. (There is also not usually the need to put the MQ configured userid into the local administrators group) |
I said it was the SIDs! Yay me! 
(That will be the total level of my truimph for today) 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| MQChela |
| Posted: Thu Feb 16, 2012 6:12 am Post subject: |
|
|
Novice
Joined: 07 Feb 2006
Posts: 20
|
Unfortunately restarting the server is not an option during weekdays.
File "H:\Program Files\IBM\WebSphere MQ\log\Q_MGR_1\amqhlctl.lfh" is indeed mentioned in couple of FDCs. But MUSR_MQADMIN has full access to the file.
amqmsrvn.exe is running under MUSR_MQADMIN
Several other MQ processes including amqzxma0.exe that is mentioned in FDCs are running under interactive user's account. |
|
| Back to top |
|
|
| JasonE |
| Posted: Thu Feb 16, 2012 6:42 am Post subject: |
|
|
Grand Master
Joined: 03 Nov 2003
Posts: 1220
Location: Hursley
|
| Quote: |
| Unfortunately restarting the server is not an option during weekdays. |
Sure... I more was thinking of stopping the taskbar icon, stop the service, and if amqmsrvn.exe is still running then kill it. If the process token of amqmsrvn is not administrative, adding MUSR_MQADMIN to be an administrator is ineffective until that process restarts.
| Quote: |
File "H:\Program Files\IBM\WebSphere MQ\log\Q_MGR_1\amqhlctl.lfh" is indeed mentioned in couple of FDCs. But MUSR_MQADMIN has full access to the file.
|
The FDC was explicitly saying MQ tried to open the file under that userid and did not have access to the file. What makes you think it does have access to that file? On the failing machine you need to ensure that local mqm has full control over the folder, and ensure it was reapplied to all subdirs and contents. Explore through to the file, and see what the effective permissions of musr_mqadmin would be.
Putting musr_mqadmin in administrators group really isnt needed for normal mq operation but if you are trying to do a form of DR onto other boxes rather than use MSCS which sorts it all out for you, then by making it administrative you are probablyt benefitting from the fact admins have full control by default. The alternative is to add the mqm group in manually on the failing node for this to work.
| Quote: |
| amqmsrvn.exe is running under MUSR_MQADMIN |
This is a dcom object used to launch other processes if not run in interactive mode, so will be what a service start or amqmdain start QM will do.
| Quote: |
| Several other MQ processes including amqzxma0.exe that is mentioned in FDCs are running under interactive user's account. |
This means you did an interactive strmqm rather than one going via the dcom object, and hence will die when you sign off. The benefit is that you are probably signed on as an administrator and hencethe qmgr is running administratively.[/quote] |
|
| Back to top |
|
|
|
|
