| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Can't start Queue Manager - Error |
« View previous topic :: View next topic » |
| Author |
Message
|
| LouML |
 Posted: Mon Aug 27, 2012 6:39 am Post subject: Can't start Queue Manager - Error |
 |
|
Partisan
Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY
|
We are running MQ Server 7.0.1.5 on a Solaris 10 blade. This morning we noticed that a queue manager was not running. This has been running without issue for years. I tried to manually start it but got a 545284129 error. There are no entries in any MQ logs and no FDC files created.
Our Unix team took a look and saw some shared memory issues, but they say they’re cleared them. The strmqm command still fails. Then they rebooted the server, but it still fails.
I’ve searched for 545284129 here and in Google but can’t find a relevant answer.
We do weekly reboots Sundays at 1:00 AM. In checking the logs, I see that MQ started to come back up at 1:06 AM. I do not know if the queue manager ran for some time before ending or died right away. The last queue manager log entries are below:
| Code: |
bash-3.00$ uname -a
SunOS stmpbdr1 5.10 Generic_141445-09 i86pc i386 i86pc
bash-3.00$ dspmqver
Name: WebSphere MQ
Version: 7.0.1.5
CMVC level: p701-105-110419
BuildType: IKAP - (Production)
bash-3.00$ dspmq
QMNAME(QM.PBDR1.01) STATUS(Ended normally)
bash-3.00$ strmqm QM.PBDR1.01
WebSphere MQ queue manager 'QM.PBDR1.01' starting.
The Queue Manager ended for reason 545284129 ''
bash-3.00$ ls -la /var/mqm/errors/ /var/mqm/qmgrs/QM\!PBDR1\!01/errors/
/var/mqm/errors/:
total 540
drwxrwsrwx 2 mqm mqm 6 Aug 27 08:59 .
drwxrwsr-x 12 mqm mqm 15 Aug 27 08:58 ..
-rw-r----- 1 mqm mqm 158470 Apr 8 2011 AMQ26878.0.FDC
-rw-rw-r-- 1 mqm mqm 6797 Feb 1 2012 AMQERR01.LOG
-rw-rw-r-- 1 mqm mqm 0 Oct 25 2010 AMQERR02.LOG
-rw-rw-r-- 1 mqm mqm 0 Oct 25 2010 AMQERR03.LOG
/var/mqm/qmgrs/QM!PBDR1!01/errors/:
total 2069
drwxrws--- 2 mqm mqm 5 Aug 21 22:00 .
drwxrwsr-x 21 mqm mqm 26 Aug 27 10:00 ..
-rw-rw---- 1 mqm mqm 179037 Aug 26 01:10 AMQERR01.LOG
-rw-rw---- 1 mqm mqm 262426 Aug 21 22:00 AMQERR02.LOG
-rw-rw---- 1 mqm mqm 262205 Aug 13 22:17 AMQERR03.LOG
bash-3.00$
-------------------------------------------------------------------------------
08/26/12 01:00:32 - Process(1214.1) User(mqm) Program(amqzxma0_nd)
Host(stmpbdr1)
AMQ8004: WebSphere MQ queue manager 'QM.PBDR1.01' ended.
EXPLANATION:
WebSphere MQ queue manager 'QM.PBDR1.01' ended.
ACTION:
None.
-------------------------------------------------------------------------------
08/26/12 01:06:21 - Process(1212.1) User(mqm) Program(amqzxma0_nd)
Host(stmpbdr1)
AMQ7229: 5 log records accessed on queue manager 'QM.PBDR1.01' during the log
replay phase.
EXPLANATION:
5 log records have been accessed so far on queue manager QM.PBDR1.01 during the
log replay phase in order to bring the queue manager back to a previously known
state.
ACTION:
None.
-------------------------------------------------------------------------------
08/26/12 01:06:21 - Process(1212.1) User(mqm) Program(amqzxma0_nd)
Host(stmpbdr1)
AMQ7230: Log replay for queue manager 'QM.PBDR1.01' complete.
EXPLANATION:
The log replay phase of the queue manager restart process has been completed
for queue manager QM.PBDR1.01.
ACTION:
None.
-------------------------------------------------------------------------------
08/26/12 01:06:21 - Process(1212.1) User(mqm) Program(amqzxma0_nd)
Host(stmpbdr1)
AMQ7231: 0 log records accessed on queue manager 'QM.PBDR1.01' during the
recovery phase.
EXPLANATION:
0 log records have been accessed so far on queue manager QM.PBDR1.01 during the
recovery phase of the transactions manager state.
ACTION:
None.
-------------------------------------------------------------------------------
08/26/12 01:06:21 - Process(1212.1) User(mqm) Program(amqzxma0_nd)
Host(stmpbdr1)
AMQ7232: Transaction manager state recovered for queue manager 'QM.PBDR1.01'.
EXPLANATION:
The state of transactions at the time the queue manager ended has been
recovered for queue manager QM.PBDR1.01.
ACTION:
None.
------------------------------------------------------------------------------- |
_________________
Yeah, well, you know, that's just, like, your opinion, man. - The Dude |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Mon Aug 27, 2012 7:07 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
The joys of strmqtrc.
EDIT.
Slightly more constructively, confirm that the reboot didn't do things like forget to mount the right file systems in the right place at the right time, and that someone didn't do something fun like put in security changes at the system or file system level and the usual suspects. |
|
| Back to top |
|
|
| LouML |
| Posted: Mon Aug 27, 2012 7:42 am Post subject: |
|
|
Partisan
Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY
|
Unix team swears there were no changes.
Ran a trace and have two trace files.
| Code: |
bash-3.00$ more AMQ7954.0.FMT
Timestamp Process.Thread Trace Ident Trace Data
===========================================================
11:18:04.438888 7954.1 : Header.v02:7.0:SunOS 5.10:64:4:1:EST
11:18:04.438916 7954.1 : Version : 7.0.1.5 Level : p701-105-110419
11:18:04.438936 7954.1 : UTC Date : 08/27/12 Time : 15:18:04.438869
11:18:04.438948 7954.1 : Local Date : 08/27/12 Time : 11:18:04.438869 EST
11:18:04.438992 7954.1 : PID : 7954 Process : strmqm (64-bit)
11:18:04.439003 7954.1 : Host : stmpbdr1
11:18:04.439010 7954.1 : Operating System : SunOS 5.10
11:18:04.439018 7954.1 : Product Long Name : WebSphere MQ for Solaris (x86-64 platform)
11:18:04.439031 7954.1 : --------------------------------
11:18:04.439039 7954.1 : xtrNullFd: 4, xihTraceFileNum: 5
11:18:04.439061 7954.1 : Thread stack
11:18:04.439081 7954.1 : -> xcsTerminate
11:18:04.439094 7954.1 : -> xihCSThreadClear
11:18:04.439104 7954.1 : -{ xihCSThreadClear
11:18:04.439113 7954.1 : --{ xcsEmptyCBM
11:18:04.439120 7954.1 : ---{ xcsFreeMemFn
11:18:04.439133 7954.1 : component:23 pointer:42fc08
11:18:04.439147 7954.1 : cbmindex:-1
11:18:04.439156 7954.1 : ---} xcsFreeMemFn rc=OK
11:18:04.439163 7954.1 : ---{ xcsFreeMemFn
11:18:04.439171 7954.1 : component:23 pointer:455338
11:18:04.439179 7954.1 : cbmindex:-1
11:18:04.439196 7954.1 : ---} xcsFreeMemFn rc=OK
11:18:04.439204 7954.1 : ---{ xcsFreeMemFn
11:18:04.439212 7954.1 : component:23 pointer:455ba8
11:18:04.439220 7954.1 : cbmindex:-1
11:18:04.439227 7954.1 : ---} xcsFreeMemFn rc=OK
11:18:04.439234 7954.1 : ---{ xcsFreeMemFn
11:18:04.439255 7954.1 : component:23 pointer:456418
11:18:04.439263 7954.1 : cbmindex:-1
11:18:04.439270 7954.1 : ---} xcsFreeMemFn rc=OK
11:18:04.439277 7954.1 : ---{ xcsFreeMemFn
11:18:04.439285 7954.1 : component:23 pointer:456c88
11:18:04.439292 7954.1 : cbmindex:-1
11:18:04.439299 7954.1 : ---} xcsFreeMemFn rc=OK
11:18:04.439306 7954.1 : ---{ xcsFreeMemFn
11:18:04.439313 7954.1 : component:23 pointer:4574f8
11:18:04.439326 7954.1 : cbmindex:-1
11:18:04.439334 7954.1 : ---} xcsFreeMemFn rc=OK
11:18:04.439346 7954.1 : ---{ xcsFreeMemFn
11:18:04.439354 7954.1 : component:23 pointer:453178
11:18:04.439361 7954.1 : cbmindex:-1
11:18:04.439368 7954.1 : ---} xcsFreeMemFn rc=OK
11:18:04.439375 7954.1 : ---{ xcsFreeMemFn
11:18:04.439383 7954.1 : component:23 pointer:439448
11:18:04.439390 7954.1 : cbmindex:-1
11:18:04.439397 7954.1 : ---} xcsFreeMemFn rc=OK
11:18:04.439404 7954.1 : ---{ xcsFreeMemFn
11:18:04.439411 7954.1 : component:23 pointer:439d88
11:18:04.439419 7954.1 : cbmindex:-1
11:18:04.439429 7954.1 : ---} xcsFreeMemFn rc=OK
11:18:04.439436 7954.1 : ---{ xcsFreeMemFn
11:18:04.439449 7954.1 : component:23 pointer:43f9e8
11:18:04.439458 7954.1 : cbmindex:-1
11:18:04.439465 7954.1 : ---} xcsFreeMemFn rc=OK
11:18:04.439472 7954.1 : --} xcsEmptyCBM rc=OK
11:18:04.439479 7954.1 : -} xihCSThreadClear rc=OK
11:18:04.439487 7954.1 : -{ TermPrivateServices
11:18:04.439494 7954.1 : --{ xxxTerminate
11:18:04.439501 7954.1 : ---{ xcsDestroyThreadMutexSem
11:18:04.439509 7954.1 : ---} xcsDestroyThreadMutexSem rc=OK
11:18:04.439516 7954.1 : ---{ xcsFreeMemFn
11:18:04.439528 7954.1 : component:23 pointer:429d78
11:18:04.439536 7954.1 : cbmindex:0
11:18:04.439543 7954.1 : ---} xcsFreeMemFn rc=OK
11:18:04.439550 7954.1 : ---{ xcsFreeMemFn
11:18:04.439558 7954.1 : component:23 pointer:431f58
11:18:04.439571 7954.1 : cbmindex:-1
11:18:04.439579 7954.1 : ---} xcsFreeMemFn rc=OK
11:18:04.439586 7954.1 : --} xxxTerminate rc=OK
11:18:04.439593 7954.1 : --{ xehTerminateAsySignalHandling
11:18:04.439600 7954.1 : ---{ xehStopAsySignalMonitor
11:18:04.439607 7954.1 : xihAsySignalMonitorMutex holder tid(0) mod(0)
11:18:04.439624 7954.1 : ----{ xcsRequestThreadMutexSem
11:18:04.439632 7954.1 : hmtx is 425bb0
11:18:04.439639 7954.1 : ----} xcsRequestThreadMutexSem rc=OK
11:18:04.439646 7954.1 : ----{ xcsFreeMemFn
11:18:04.439654 7954.1 : component:23 pointer:43b608
11:18:04.439662 7954.1 : cbmindex:1
11:18:04.439669 7954.1 : ----} xcsFreeMemFn rc=OK
11:18:04.439676 7954.1 : ----{ xcsFreeMemFn
11:18:04.439683 7954.1 : component:23 pointer:428348
11:18:04.439701 7954.1 : cbmindex:2
11:18:04.439709 7954.1 : ----} xcsFreeMemFn rc=OK
11:18:04.439735 7954.1 : ----{ xcsFreeMemFn
11:18:04.439743 7954.1 : component:23 pointer:4282e8
11:18:04.439751 7954.1 : cbmindex:3
11:18:04.439758 7954.1 : ----} xcsFreeMemFn rc=OK
11:18:04.439765 7954.1 : ----{ xcsFreeMemFn
11:18:04.439772 7954.1 : component:23 pointer:428288
11:18:04.439780 7954.1 : cbmindex:4
11:18:04.439786 7954.1 : ----} xcsFreeMemFn rc=OK
11:18:04.439793 7954.1 : ----{ xcsFreeMemFn
11:18:04.439801 7954.1 : component:23 pointer:429d18
11:18:04.439808 7954.1 : cbmindex:5
11:18:04.439818 7954.1 : ----} xcsFreeMemFn rc=OK
11:18:04.439825 7954.1 : ----{ xcsReleaseThreadMutexSem
11:18:04.439837 7954.1 : hmtx is 425bb0
11:18:04.439845 7954.1 : ----} xcsReleaseThreadMutexSem rc=OK
11:18:04.439852 7954.1 : ----{ xcsThreadRaise
11:18:04.439860 7954.1 : tid(3) sig(19)
11:18:04.439868 7954.1 : xcsThreadRaise: Before Lock
11:18:04.439875 7954.1 : xcsThreadRaise: After Lock
11:18:04.439882 7954.1 : top of loop
11:18:04.439890 7954.1 : In Loop: pCtl = 0x00431070
11:18:04.439898 7954.1 : calling pthread_kill for pCtl(428450)
11:18:04.439924 7954.1 : thread kill ok
11:18:04.439995 7954.1 : ----} xcsThreadRaise rc=OK
11:18:04.440003 7954.1 : Successfully raised SIGPWR for tid(3)
11:18:04.440013 7954.1 : Destroyed xihAsyStarted condition variable
11:18:04.440022 7954.1 : ---} xehStopAsySignalMonitor rc=OK
11:18:04.440029 7954.1 : --} xehTerminateAsySignalHandling rc=OK
11:18:04.440045 7954.1 : FPE Handler deinstalled, New=0, Old=fe8b3b60
11:18:04.440055 7954.1 : SEGV Handler deinstalled, New=0, Old=fe8b3b60
11:18:04.440068 7954.1 : ABRT Handler deinstalled, New=0, Old=0
11:18:04.440077 7954.1 : BUS Handler deinstalled, New=0, Old=fe8b3b60
11:18:04.440086 7954.1 : ILL Handler deinstalled, New=0, Old=fe8b3b60
11:18:04.440093 7954.1 : Synchronous signal handling deactivated - previous handlers restored
11:18:04.440101 7954.1 : -} TermPrivateServices rc=OK
11:18:04.440107 7954.1 : -{ xcsReleaseThreadMutexSem
11:18:04.440115 7954.1 : hmtx is 425ac0
11:18:04.440122 7954.1 : -} xcsReleaseThreadMutexSem rc=OK
11:18:04.440129 7954.1 : } xcsTerminate rc=OK
11:18:04.440144 7954.1 : { zutMapRCTo8bit
11:18:04.440153 7954.1 : } zutMapRCTo8bit rc=OK
11:18:04.440161 7954.1 : { xcsInitialize
11:18:04.440169 7954.1 : -{ xihCSThreadClear
11:18:04.440182 7954.1 : --{ xcsEmptyCBM
11:18:04.440189 7954.1 : ---{ xcsFreeMemFn
11:18:04.440198 7954.1 : component:23 pointer:429d78
11:18:04.440206 7954.1 : cbmindex:-1
11:18:04.440213 7954.1 : ---} xcsFreeMemFn rc=OK
11:18:04.440220 7954.1 : ---{ xcsFreeMemFn
11:18:04.440227 7954.1 : component:23 pointer:43b608
11:18:04.440240 7954.1 : cbmindex:-1
11:18:04.440247 7954.1 : ---} xcsFreeMemFn rc=OK
11:18:04.440254 7954.1 : ---{ xcsFreeMemFn
11:18:04.440261 7954.1 : component:23 pointer:428348
11:18:04.440269 7954.1 : cbmindex:-1
11:18:04.440276 7954.1 : ---} xcsFreeMemFn rc=OK
11:18:04.440283 7954.1 : ---{ xcsFreeMemFn
11:18:04.440290 7954.1 : component:23 pointer:4282e8
11:18:04.440303 7954.1 : cbmindex:-1
11:18:04.440310 7954.1 : ---} xcsFreeMemFn rc=OK
11:18:04.440317 7954.1 : ---{ xcsFreeMemFn
11:18:04.440325 7954.1 : component:23 pointer:428288
11:18:04.440345 7954.1 : cbmindex:-1
11:18:04.440353 7954.1 : ---} xcsFreeMemFn rc=OK
11:18:04.440359 7954.1 : ---{ xcsFreeMemFn
11:18:04.440367 7954.1 : component:23 pointer:429d18
11:18:04.440374 7954.1 : cbmindex:-1
11:18:04.440381 7954.1 : ---} xcsFreeMemFn rc=OK
11:18:04.440388 7954.1 : --} xcsEmptyCBM rc=OK
11:18:04.440395 7954.1 : -} xihCSThreadClear rc=OK
11:18:04.440402 7954.1 : -{ xcsRequestThreadMutexSem
11:18:04.440409 7954.1 : hmtx is 425ac0
11:18:04.440416 7954.1 : -} xcsRequestThreadMutexSem rc=OK
11:18:04.440439 7954.1 : -{ InitPrivateServices
11:18:04.440447 7954.1 : --{ xcsGetEnvironmentString
11:18:04.440456 7954.1 : xcsGetEnvironmentString[MQS_ACTION_ON_EXCEPTION] = NULL
11:18:04.440463 7954.1 : --}! xcsGetEnvironmentString rc=xecE_E_ENV_VAR_NOT_FOUND
11:18:04.440470 7954.1 : --{ xcsIsEnvironment
11:18:04.440477 7954.1 : xcsIsEnvironment[AMQ_SIGCHLD_SIGACTION] = FALSE
11:18:04.440484 7954.1 : --} xcsIsEnvironment rc=OK
11:18:04.440491 7954.1 : AMQ_SIGCHLD_SIGACTION is not set
11:18:04.440498 7954.1 : --{ xcsIsEnvironment
11:18:04.440506 7954.1 : xcsIsEnvironment[MQS_NO_SYNC_SIGNAL_HANDLING] = FALSE
11:18:04.440530 7954.1 : --} xcsIsEnvironment rc=OK
11:18:04.440538 7954.1 : --{ xcsIsEnvironment
11:18:04.440546 7954.1 : xcsIsEnvironment[MQS_FORCE_NO_SYNC_SIGNAL_HANDLING] = FALSE
11:18:04.440553 7954.1 : --} xcsIsEnvironment rc=OK
11:18:04.440562 7954.1 : FPE Handler installed, New=fe8b3b60, Old=0
11:18:04.440576 7954.1 : SEGV Handler installed, New=fe8b3b60, Old=0
11:18:04.440586 7954.1 : BUS Handler installed, New=fe8b3b60, Old=0
11:18:04.440594 7954.1 : ILL Handler installed, New=fe8b3b60, Old=0
11:18:04.440605 7954.1 : Synchronous Signal Handling Activated
11:18:04.440612 7954.1 : --{ xehInitialiseAsySignalHandling
11:18:04.440620 7954.1 : ---{ xcsRequestThreadMutexSem
11:18:04.440627 7954.1 : hmtx is 425bb0
11:18:04.440634 7954.1 : ---} xcsRequestThreadMutexSem rc=OK
11:18:04.440641 7954.1 : ---{ xcsReleaseThreadMutexSem
11:18:04.440648 7954.1 : hmtx is 425bb0
11:18:04.440655 7954.1 : ---} xcsReleaseThreadMutexSem rc=OK
11:18:04.440662 7954.1 : --} xehInitialiseAsySignalHandling rc=OK
11:18:04.440669 7954.1 : --{ xcsRegisterAsySigHandler
11:18:04.440677 7954.1 : Signal(2 : SIGINT), funcP(fffffd7ffe8b30f0)
11:18:04.440688 7954.1 : xihAsySignalMonitorMutex holder tid(0) mod(0)
11:18:04.440700 7954.1 : ---{ xcsRequestThreadMutexSem
11:18:04.440709 7954.1 : hmtx is 425bb0
11:18:04.440716 7954.1 : ---} xcsRequestThreadMutexSem rc=OK
11:18:04.440723 7954.1 : ---{ xcsGetMemFn
11:18:04.440732 7954.1 : component:23 function:344 length:24 options:0 cbmindex:-1 *pointer:429cb8
11:18:04.440739 7954.1 : ---} xcsGetMemFn rc=OK
11:18:04.440746 7954.1 : ---{ xcsReleaseThreadMutexSem
11:18:04.440753 7954.1 : hmtx is 425bb0
11:18:04.440763 7954.1 : ---} xcsReleaseThreadMutexSem rc=OK
11:18:04.440770 7954.1 : --} xcsRegisterAsySigHandler rc=OK
11:18:04.440777 7954.1 : --{ xcsRegisterAsySigHandler
11:18:04.440785 7954.1 : Signal(3 : SIGQUIT), funcP(fffffd7ffe8b30f0)
11:18:04.440793 7954.1 : xihAsySignalMonitorMutex holder tid(0) mod(0)
11:18:04.440800 7954.1 : ---{ xcsRequestThreadMutexSem
11:18:04.440807 7954.1 : hmtx is 425bb0
11:18:04.440819 7954.1 : ---} xcsRequestThreadMutexSem rc=OK
11:18:04.440827 7954.1 : ---{ xcsGetMemFn
11:18:04.440840 7954.1 : component:23 function:344 length:24 options:0 cbmindex:-1 *pointer:429d18
11:18:04.440853 7954.1 : ---} xcsGetMemFn rc=OK
11:18:04.440860 7954.1 : ---{ xcsReleaseThreadMutexSem
11:18:04.440867 7954.1 : hmtx is 425bb0
11:18:04.440874 7954.1 : ---} xcsReleaseThreadMutexSem rc=OK
11:18:04.440881 7954.1 : --} xcsRegisterAsySigHandler rc=OK
11:18:04.440888 7954.1 : --{ xcsRegisterAsySigHandler
11:18:04.440896 7954.1 : Signal(15 : SIGTERM), funcP(fffffd7ffe8b30f0)
11:18:04.440904 7954.1 : xihAsySignalMonitorMutex holder tid(0) mod(0)
11:18:04.440911 7954.1 : ---{ xcsRequestThreadMutexSem
11:18:04.440939 7954.1 : hmtx is 425bb0
11:18:04.440946 7954.1 : ---} xcsRequestThreadMutexSem rc=OK
11:18:04.440953 7954.1 : ---{ xcsGetMemFn
11:18:04.440967 7954.1 : component:23 function:344 length:24 options:0 cbmindex:-1 *pointer:429d78
11:18:04.440975 7954.1 : ---} xcsGetMemFn rc=OK
11:18:04.440982 7954.1 : ---{ xcsReleaseThreadMutexSem
11:18:04.440989 7954.1 : hmtx is 425bb0
11:18:04.440996 7954.1 : ---} xcsReleaseThreadMutexSem rc=OK
11:18:04.441003 7954.1 : --} xcsRegisterAsySigHandler rc=OK
11:18:04.441010 7954.1 : --{ xcsRegisterAsySigHandler
11:18:04.441018 7954.1 : Signal(1 : SIGHUP), funcP(fffffd7ffe8b30f0)
11:18:04.441029 7954.1 : xihAsySignalMonitorMutex holder tid(0) mod(0)
11:18:04.441036 7954.1 : ---{ xcsRequestThreadMutexSem
11:18:04.441043 7954.1 : hmtx is 425bb0
11:18:04.441051 7954.1 : ---} xcsRequestThreadMutexSem rc=OK
11:18:04.441058 7954.1 : ---{ xcsGetMemFn
11:18:04.441066 7954.1 : component:23 function:344 length:24 options:0 cbmindex:-1 *pointer:43f4f8
11:18:04.441073 7954.1 : ---} xcsGetMemFn rc=OK
11:18:04.441092 7954.1 : ---{ xcsReleaseThreadMutexSem
11:18:04.441101 7954.1 : hmtx is 425bb0
11:18:04.441111 7954.1 : ---} xcsReleaseThreadMutexSem rc=OK
11:18:04.441118 7954.1 : --} xcsRegisterAsySigHandler rc=OK
11:18:04.441125 7954.1 : --{ xehStartAsySignalMonitor
11:18:04.441133 7954.1 : xihAsySignalMonitorMutex holder tid(0) mod(0)
11:18:04.441140 7954.1 : ---{ xcsRequestThreadMutexSem
11:18:04.441147 7954.1 : hmtx is 425bb0
11:18:04.441154 7954.1 : ---} xcsRequestThreadMutexSem rc=OK
11:18:04.441161 7954.1 : xehStartAsySignalMonitor for threaded build
11:18:04.441168 7954.1 : ---{ xppTestAndSetupInitCondition
11:18:04.441175 7954.1 : ---} xppTestAndSetupInitCondition rc=OK
11:18:04.441188 7954.1 : ---{ xcsCreateThread
11:18:04.441196 7954.1 : Attr(260) pThread(fffffd7ffe8b6fd0) pArg(0)
11:18:04.441203 7954.1 : ----{ xcsGetMemFn
11:18:04.441218 7954.1 : component:23 function:35 length:24 options:0 cbmindex:-1 *pointer:43f558
11:18:04.441226 7954.1 : ----} xcsGetMemFn rc=OK
11:18:04.441233 7954.1 : pthread_attr_setstacksize ok
11:18:04.441240 7954.1 : pthread_attr_setdetachstate ok
11:18:04.441328 7954.1 : pthread_t from pthread_create is: (4), returning tid: (4)
11:18:04.441346 7954.1 : ---} xcsCreateThread rc=OK
11:18:04.441354 7954.1 : New monitor thread tid(4)
11:18:04.441364 7954.1 : ---{ xcsReleaseThreadMutexSem
11:18:04.441373 7954.1 : hmtx is 425bb0
11:18:04.441381 7954.1 : ---} xcsReleaseThreadMutexSem rc=OK
11:18:04.441388 7954.1 : ---{ xppWaitForAsySigMonThread
11:18:04.441426 7954.4 : Thread stack
11:18:04.441444 7954.4 : -> xcsFreeMemFn
11:18:04.441450 7954.4 : { xcsFreeMemFn
11:18:04.441461 7954.4 : component:23 pointer:43f558
11:18:04.441467 7954.4 : cbmindex:0
11:18:04.441490 7954.4 : } xcsFreeMemFn rc=OK
11:18:04.441496 7954.4 : { xcsIsEnvironment
11:18:04.441505 7954.4 : xcsIsEnvironment[AMQ_NO_SIGWAIT_SIGTRAP] = FALSE
11:18:04.441510 7954.4 : } xcsIsEnvironment rc=OK
11:18:04.441515 7954.4 : Async signals blocked on thread
11:18:04.441520 7954.4 : Calling thread function (fffffd7ffe8b6fd0) with args(0)
11:18:04.441524 7954.4 : { xehAsySignalMonitor
11:18:04.441529 7954.4 : -{ xcsRequestThreadMutexSem
11:18:04.441533 7954.4 : hmtx is 425bb0
11:18:04.441537 7954.4 : -} xcsRequestThreadMutexSem rc=OK
11:18:04.441544 7954.4 : -{ xcsReleaseThreadMutexSem
11:18:04.441549 7954.4 : hmtx is 425bb0
11:18:04.441553 7954.4 : -} xcsReleaseThreadMutexSem rc=OK
11:18:04.441557 7954.4 : -{ xcsIsEnvironment
11:18:04.441567 7954.4 : xcsIsEnvironment[AMQ_NO_SIGWAIT_SIGTRAP] = FALSE
11:18:04.441576 7954.4 : -} xcsIsEnvironment rc=OK
11:18:04.441581 7954.4 : Async signals blocked on thread
11:18:04.441588 7954.4 : sigaction to clear SIG_IGN for signal (18 : SIGCHLD) successful
11:18:04.441594 7954.4 : sigaction to clear SIG_IGN for signal (19 : SIGPWR) successful
11:18:04.441604 7954.4 : Calling xppPostAsySigMonThread
11:18:04.441608 7954.4 : -{ xppPostAsySigMonThread
11:18:04.441616 7954.4 : -} xppPostAsySigMonThread rc=OK
11:18:04.441672 7954.4 : Monitor sigwaiting for next asy. signal
11:18:04.441697 7954.1 : ---} xppWaitForAsySigMonThread rc=OK
11:18:04.441721 7954.1 : --} xehStartAsySignalMonitor rc=OK
11:18:04.441737 7954.1 : Retrieved uid 1600 from getuid() [errno=0]
11:18:04.441746 7954.1 : -} InitPrivateServices rc=OK
11:18:04.441756 7954.1 : -{ xtrTAisMatch
11:18:04.441775 7954.1 : --{ xcsQueryProcessDetails
11:18:04.441834 7954.1 : Data: 0x00000006
11:18:04.441854 7954.1 : --} xcsQueryProcessDetails rc=OK
11:18:04.441863 7954.1 : --{ xihGetQueueManagerNames
11:18:04.441870 7954.1 : --} xihGetQueueManagerNames rc=OK
11:18:04.441877 7954.1 : -}! xtrTAisMatch rc=Unknown(1)
11:18:04.441885 7954.1 : -{ xxxInitialize
11:18:04.441892 7954.1 : --{ xcsSetlocale
11:18:04.441901 7954.1 : category(6) locale() buffer(fffffd7fffdfc150) buflen(1280)
11:18:04.441922 7954.1 : locale changed, now set to C
11:18:04.441931 7954.1 : --} xcsSetlocale rc=OK
11:18:04.441944 7954.1 : --{ xcsSetlocale
11:18:04.441953 7954.1 : category(6) locale((nil)) buffer(fffffd7fffdfb7a0) buflen(1280)
11:18:04.441963 7954.1 : locale not changed, currently set to C
11:18:04.441970 7954.1 : --} xcsSetlocale rc=OK
11:18:04.441977 7954.1 : --{ xcsGetEnvironmentString
11:18:04.441985 7954.1 : xcsGetEnvironmentString[LANG] = NULL
11:18:04.441998 7954.1 : --}! xcsGetEnvironmentString rc=xecE_E_ENV_VAR_NOT_FOUND
11:18:04.442006 7954.1 : --{ xcsGetMemFn
11:18:04.442016 7954.1 : component:23 function:207 length:56 options:0 cbmindex:-1 *pointer:440538
11:18:04.442029 7954.1 : --} xcsGetMemFn rc=OK
11:18:04.442036 7954.1 : --{ xcsCreateThreadMutexSem
11:18:04.442044 7954.1 : hmtx is 0
11:18:04.442052 7954.1 : --} xcsCreateThreadMutexSem rc=OK
11:18:04.442302 7954.1 : --{ xcsGetMemFn
11:18:04.442323 7954.1 : component:23 function:207 length:23392 options:0 cbmindex:-1 *pointer:447ef8
11:18:04.442332 7954.1 : --} xcsGetMemFn rc=OK
11:18:04.443137 7954.1 : Information: No default set for EBCDIC data conversion in ccsid.tbl.
11:18:04.443156 7954.1 : Information: No default set for ASCII data conversion in ccsid.tbl.
11:18:04.443165 7954.1 : Codeset for locale is ISO8859-1, common services CCSID selected is 819.
11:18:04.443172 7954.1 : -} xxxInitialize rc=OK
11:18:04.443179 7954.1 : -{ xcsGetEnvironmentString
11:18:04.443194 7954.1 : xcsGetEnvironmentString[AMQ_SERVICE_MODULE] = NULL
11:18:04.443202 7954.1 : -}! xcsGetEnvironmentString rc=xecE_E_ENV_VAR_NOT_FOUND
11:18:04.443209 7954.1 : -{ xcsReleaseThreadMutexSem
11:18:04.443216 7954.1 : hmtx is 425ac0
11:18:04.443223 7954.1 : -} xcsReleaseThreadMutexSem rc=OK
11:18:04.443251 7954.1 : } xcsInitialize rc=OK
11:18:04.443259 7954.1 : { xcsTerminate
11:18:04.443266 7954.1 : -{ xcsRequestThreadMutexSem
11:18:04.443274 7954.1 : hmtx is 425ac0
11:18:04.443281 7954.1 : -} xcsRequestThreadMutexSem rc=OK
11:18:04.443289 7954.1 : xihThreadInitsCount: 0, xihAllInitsCount: 0
11:18:04.443297 7954.1 : -{ xtrTAisMatch
11:18:04.443304 7954.1 : --{ xcsQueryProcessDetails
11:18:04.443344 7954.1 : Data: 0x00000006
11:18:04.443357 7954.1 : --} xcsQueryProcessDetails rc=OK
11:18:04.443371 7954.1 : --{ xihGetQueueManagerNames
11:18:04.443384 7954.1 : --} xihGetQueueManagerNames rc=OK
11:18:04.443392 7954.1 : -}! xtrTAisMatch rc=Unknown(1)
11:18:04.443399 7954.1 : -{ xihCSThreadClear
11:18:04.443406 7954.1 : --{ xcsEmptyCBM
11:18:04.443413 7954.1 : --} xcsEmptyCBM rc=OK
11:18:04.443420 7954.1 : -} xihCSThreadClear rc=OK
11:18:04.443428 7954.1 : -{ TermPrivateServices
11:18:04.443435 7954.1 : --{ xxxTerminate
11:18:04.443442 7954.1 : ---{ xcsDestroyThreadMutexSem
11:18:04.443450 7954.1 : ---} xcsDestroyThreadMutexSem rc=OK
11:18:04.443457 7954.1 : ---{ xcsFreeMemFn
11:18:04.443465 7954.1 : component:23 pointer:440538
11:18:04.443478 7954.1 : cbmindex:0
11:18:04.443485 7954.1 : ---} xcsFreeMemFn rc=OK
11:18:04.443497 7954.1 : ---{ xcsFreeMemFn
11:18:04.443506 7954.1 : component:23 pointer:447ef8
11:18:04.443514 7954.1 : cbmindex:-1
11:18:04.443521 7954.1 : ---} xcsFreeMemFn rc=OK
11:18:04.443528 7954.1 : --} xxxTerminate rc=OK
11:18:04.443535 7954.1 : --{ xehTerminateAsySignalHandling
11:18:04.443541 7954.1 : ---{ xehStopAsySignalMonitor
11:18:04.443549 7954.1 : xihAsySignalMonitorMutex holder tid(0) mod(0)
11:18:04.443556 7954.1 : ----{ xcsRequestThreadMutexSem
11:18:04.443568 7954.1 : hmtx is 425bb0
11:18:04.443576 7954.1 : ----} xcsRequestThreadMutexSem rc=OK
11:18:04.443582 7954.1 : ----{ xcsFreeMemFn
11:18:04.443590 7954.1 : component:23 pointer:43f4f8
11:18:04.443597 7954.1 : cbmindex:1
11:18:04.443604 7954.1 : ----} xcsFreeMemFn rc=OK
11:18:04.443616 7954.1 : ----{ xcsFreeMemFn
11:18:04.443625 7954.1 : component:23 pointer:429d78
11:18:04.443632 7954.1 : cbmindex:2
11:18:04.443639 7954.1 : ----} xcsFreeMemFn rc=OK
11:18:04.443646 7954.1 : ----{ xcsFreeMemFn
11:18:04.443653 7954.1 : component:23 pointer:429d18
11:18:04.443665 7954.1 : cbmindex:3
11:18:04.443672 7954.1 : ----} xcsFreeMemFn rc=OK
11:18:04.443679 7954.1 : ----{ xcsFreeMemFn
11:18:04.443687 7954.1 : component:23 pointer:429cb8
11:18:04.443694 7954.1 : cbmindex:4
11:18:04.443701 7954.1 : ----} xcsFreeMemFn rc=OK
11:18:04.443707 7954.1 : ----{ xcsReleaseThreadMutexSem
11:18:04.443715 7954.1 : hmtx is 425bb0
11:18:04.443722 7954.1 : ----} xcsReleaseThreadMutexSem rc=OK
11:18:04.443734 7954.1 : ----{ xcsThreadRaise
11:18:04.443743 7954.1 : tid(4) sig(19)
11:18:04.443755 7954.1 : xcsThreadRaise: Before Lock
11:18:04.443763 7954.1 : xcsThreadRaise: After Lock
11:18:04.443770 7954.1 : top of loop
11:18:04.443778 7954.1 : In Loop: pCtl = 0x00431070
11:18:04.443786 7954.1 : calling pthread_kill for pCtl(43f5a0)
11:18:04.443798 7954.1 : thread kill ok
11:18:04.443859 7954.1 : ----} xcsThreadRaise rc=OK
11:18:04.443867 7954.1 : Successfully raised SIGPWR for tid(4)
11:18:04.443868 7954.4 : Data: 0x00000013
11:18:04.443909 7954.4 : Sigwait returned with signal 19 : SIGPWR
11:18:04.443919 7954.4 : Monitor stopping: xihProcess.xihAsySignalMonitorTid[0] not self[4]
11:18:04.443879 7954.1 : Destroyed xihAsyStarted condition variable
11:18:04.443966 7954.1 : ---} xehStopAsySignalMonitor rc=OK
11:18:04.443986 7954.1 : --} xehTerminateAsySignalHandling rc=OK
11:18:04.443929 7954.4 : sigaction to restore SIG_IGN for signal (18 : SIGCHLD) successful
11:18:04.444001 7954.1 : FPE Handler deinstalled, New=0, Old=fe8b3b60
11:18:04.444016 7954.4 : sigaction to restore SIG_IGN for signal (19 : SIGPWR) successful
11:18:04.444057 7954.4 : Async signals unblocked on thread
11:18:04.444065 7954.4 : } xehAsySignalMonitor rc=OK
11:18:04.444040 7954.1 : SEGV Handler deinstalled, New=0, Old=fe8b3b60
11:18:04.444073 7954.4 : Returned from thread function (fffffd7ffe8b6fd0) for pCtl(43f5a0)
11:18:04.444096 7954.1 : ABRT Handler deinstalled, New=0, Old=0
11:18:04.444142 7954.1 : BUS Handler deinstalled, New=0, Old=fe8b3b60
11:18:04.444163 7954.1 : ILL Handler deinstalled, New=0, Old=fe8b3b60
11:18:04.444173 7954.1 : Synchronous signal handling deactivated - previous handlers restored
11:18:04.444181 7954.1 : -} TermPrivateServices rc=OK
11:18:04.444191 7954.1 : -{ xcsReleaseThreadMutexSem
11:18:04.444201 7954.1 : hmtx is 425ac0
11:18:04.444217 7954.1 : -} xcsReleaseThreadMutexSem rc=OK
11:18:04.444227 7954.1 : } xcsTerminate rc=OK
11:18:04.444124 7954.4 : { xppRunDestructors
11:18:04.444243 7954.1 : }! zsllpiStartQM rc=Unknown(47)
11:18:04.444255 7954.4 : tP(43f5a0)
11:18:04.444289 7954.4 : } xppRunDestructors rc=OK
11:18:04.444300 7954.4 : { xcsDisconnectSubpoolConnectionsForThrea
11:18:04.444309 7954.4 : } xcsDisconnectSubpoolConnectionsForThrea rc=OK
11:18:04.444319 7954.4 : { xihCSThreadClear
11:18:04.444327 7954.4 : -{ xcsEmptyCBM
11:18:04.444333 7954.4 : --{ xcsFreeMemFn
11:18:04.444340 7954.4 : component:23 pointer:43f558
11:18:04.444348 7954.4 : cbmindex:-1
11:18:04.444353 7954.4 : --} xcsFreeMemFn rc=OK
11:18:04.444359 7954.4 : -} xcsEmptyCBM rc=OK
11:18:04.444373 7954.4 : } xihCSThreadClear rc=OK
11:18:04.444378 7954.4 : { xcsRelinquishThreadMutexSems
11:18:04.444388 7954.4 : hmtx is 43f5a0
===========================================================2
bash-3.00$ |
and part of the second file up to where I see a potential shared memory issue:
[code:1:1ad8d52d66]bash-3.00$ cat AMQ7955.0.FMT | more
Timestamp Process.Thread Trace Ident Trace Data
===========================================================
11:18:03.462392 7955.1 : Header.v02:7.0:SunOS 5.10:64:4:1:EST
11:18:03.462412 7955.1 : Version : 7.0.1.5 Level : p701-105-110419
11:18:03.462425 7955.1 : UTC Date : 08/27/12 Time : 15:18:03.462383
11:18:03.462432 7955.1 : Local Date : 08/27/12 Time : 11:18:03.462383 EST
11:18:03.462465 7955.1 : PID : 7955 Process : amqzxma0_nd (64-bit)
11:18:03.462472 7955.1 : Host : stmpbdr1
11:18:03.462477 7955.1 : Operating System : SunOS 5.10
11:18:03.462482 7955.1 : Product Long Name : WebSphere MQ for Solaris (x86-64 platform)
11:18:03.462501 7955.1 : --------------------------------
11:18:03.462506 7955.1 : xtrNullFd: 4, xihTraceFileNum: 5
11:18:03.462518 7955.1 : Thread stack
11:18:03.462565 7955.1 : -> ExecCtrlrMain
11:18:03.462604 7955.1 : -> xcsInitialize
11:18:03.462611 7955.1 : -> xcsGetEnvironmentString
11:18:03.462618 7955.1 : --{ xcsGetEnvironmentString
11:18:03.462627 7955.1 : xcsGetEnvironmentString[AMQ_SERVICE_MODULE] = NULL
11:18:03.462636 7955.1 : --}! xcsGetEnvironmentString rc=xecE_E_ENV_VAR_NOT_FOUND
11:18:03.462641 7955.1 : --{ xcsReleaseThreadMutexSem
11:18:03.462645 7955.1 : hmtx is 495370
11:18:03.462650 7955.1 : --} xcsReleaseThreadMutexSem rc=OK
11:18:03.462654 7955.1 : -} xcsInitialize rc=OK
11:18:03.462664 7955.1 : -{ xcsRestoreSubpoolManagedSetList
11:18:03.462674 7955.1 : --{ xcsIsMultiInstanceSubpool
11:18:03.462679 7955.1 : ---{ xcsGetQMDetails
11:18:03.462684 7955.1 : ----{ xcsGetIniFilename
11:18:03.462688 7955.1 : -----{ xcsGetEnvironmentString
11:18:03.462693 7955.1 : xcsGetEnvironmentString[AMQ_MQS_INI_LOCATION] = NULL
11:18:03.462700 7955.1 : -----}! xcsGetEnvironmentString rc=xecE_E_ENV_VAR_NOT_FOUND
11:18:03.462705 7955.1 : MQS.INI full name : /var/mqm/mqs.ini
11:18:03.462709 7955.1 : ----} xcsGetIniFilename rc=OK
11:18:03.462714 7955.1 : ----{ xcsGetMemFn
11:18:03.462734 7955.1 : component:23 function:380 length:68992 options:0 cbmindex:-1 *pointer:4b1ad8
11:18:03.462740 7955.1 : ----} xcsGetMemFn rc=OK
11:18:03.462745 7955.1 : ----{ xcsBrowseIniCallback
11:18:03.462749 7955.1 : -----{ xufOpenIniRead
11:18:03.462763 7955.1 : ------{ xufHasAccessToStanzaFile
11:18:03.462769 7955.1 : Path(/var/mqm/mqs.ini) ReqPerms(1)
11:18:03.462788 7955.1 : Has access to stanza file? (YES)
11:18:03.462793 7955.1 : ------} xufHasAccessToStanzaFile rc=OK
11:18:03.462806 7955.1 : Start locking(/var/mqm/mqs.ini)
11:18:03.462812 7955.1 : -----} xufOpenIniRead rc=OK
11:18:03.462818 7955.1 : -----{ xusReadSingleStanza
11:18:03.462829 7955.1 : ------{ xusGetLineType
11:18:03.462834 7955.1 : RetCode (xusLINE_IS_COMMENT)
11:18:03.462838 7955.1 : ------} xusGetLineType rc=OK
11:18:03.462843 7955.1 : ------{ xusAddStanzaLineList
11:18:03.462847 7955.1 : -------{ xcsGetMemFn
11:18:03.462855 7955.1 : component:24 function:15 length:2096 options:0 cbmindex:0 *pointer:4a8dd8
11:18:03.462859 7955.1 : -------} xcsGetMemFn rc=OK
11:18:03.462864 7955.1 : RetCode (OK)
11:18:03.462869 7955.1 : ------} xusAddStanzaLineList rc=OK
11:18:03.462874 7955.1 : ------{ xusGetLineType
11:18:03.462878 7955.1 : RetCode (xusLINE_IS_COMMENT)
11:18:03.462883 7955.1 : ------} xusGetLineType rc=OK
11:18:03.462887 7955.1 : ------{ xusAddStanzaLineList
11:18:03.462891 7955.1 : -------{ xcsGetMemFn
11:18:03.462896 7955.1 : component:24 function:15 length:2096 options:0 cbmindex:1 *pointer:4a9648
11:18:03.462903 7955.1 : -------} xcsGetMemFn rc=OK
11:18:03.462908 7955.1 : RetCode (OK)
11:18:03.462912 7955.1 : ------} xusAddStanzaLineList rc=OK
11:18:03.462917 7955.1 : ------{ xusGetLineType
11:18:03.462921 7955.1 : RetCode (xusLINE_IS_COMMENT)
11:18:03.462925 7955.1 : ------} xusGetLineType rc=OK
11:18:03.462930 7955.1 : ------{ xusAddStanzaLineList
11:18:03.462934 7955.1 : -------{ xcsGetMemFn
11:18:03.462939 7955.1 : component:24 function:15 length:2096 options:0 cbmindex:2 *pointer:4a9eb8
11:18:03.462950 7955.1 : -------} xcsGetMemFn rc=OK
11:18:03.462956 7955.1 : RetCode (OK)
11:18:03.462960 7955.1 : ------} xusAddStanzaLineList rc=OK
11:18:03.462965 7955.1 : ------{ xusGetLineType
11:18:03.462970 7955.1 : RetCode (xusLINE_IS_COMMENT)
11:18:03.462974 7955.1 : ------} xusGetLineType rc=OK
11:18:03.462978 7955.1 : ------{ xusAddStanzaLineList
11:18:03.462982 7955.1 : -------{ xcsGetMemFn
11:18:03.462987 7955.1 : component:24 function:15 length:2096 options:0 cbmindex:3 *pointer:4aa728
11:18:03.462991 7955.1 : -------} xcsGetMemFn rc=OK
11:18:03.462996 7955.1 : RetCode (OK)
11:18:03.463004 7955.1 : ------} xusAddStanzaLineList rc=OK
11:18:03.463009 7955.1 : ------{ xusGetLineType
11:18:03.463013 7955.1 : RetCode (xusLINE_IS_COMMENT)
11:18:03.463018 7955.1 : ------} xusGetLineType rc=OK
11:18:03.463022 7955.1 : ------{ xusAddStanzaLineList
11:18:03.463026 7955.1 : -------{ xcsGetMemFn
11:18:03.463031 7955.1 : component:24 function:15 length:2096 options:0 cbmindex:4 *pointer:4aaf98
11:18:03.463035 7955.1 : -------} xcsGetMemFn rc=OK
11:18:03.463040 7955.1 : RetCode (OK)
11:18:03.463043 7955.1 : ------} xusAddStanzaLineList rc=OK
11:18:03.463050 7955.1 : ------{ xusGetLineType
11:18:03.463054 7955.1 : RetCode (xusLINE_IS_COMMENT)
11:18:03.463058 7955.1 : ------} xusGetLineType rc=OK
11:18:03.463062 7955.1 : ------{ xusAddStanzaLineList
11:18:03.463066 7955.1 : -------{ xcsGetMemFn
11:18:03.463071 7955.1 : component:24 function:15 length:2096 options:0 cbmindex:5 *pointer:4a1808
11:18:03.463075 7955.1 : -------} xcsGetMemFn rc=OK
11:18:03.463080 7955.1 : RetCode (OK)
11:18:03.463084 7955.1 : ------} xusAddStanzaLineList rc=OK
11:18:03.463089 7955.1 : ------{ xusGetLineType
11:18:03.463103 7955.1 : RetCode (xusLINE_IS_COMMENT)
11:18:03.463108 7955.1 : ------} xusGetLineType rc=OK
11:18:03.463112 7955.1 : ------{ xusAddStanzaLineList
11:18:03.463116 7955.1 : -------{ xcsGetMemFn
11:18:03.463121 7955.1 : component:24 function:15 length:2096 options:0 cbmindex:6 *pointer:4a2078
11:18:03.463126 7955.1 : -------} xcsGetMemFn rc=OK
11:18:03.463130 7955.1 : RetCode (OK)
11:18:03.463135 7955.1 : ------} xusAddStanzaLineList rc=OK
11:18:03.463140 7955.1 : ------{ xusGetLineType
11:18:03.463144 7955.1 : RetCode (xusLINE_IS_COMMENT)
11:18:03.463151 7955.1 : ------} xusGetLineType rc=OK
11:18:03.463155 7955.1 : ------{ xusAddStanzaLineList
11:18:03.463159 7955.1 : -------{ xcsGetMemFn
11:18:03.463164 7955.1 : component:24 function:15 length:2096 options:0 cbmindex:7 *pointer:4a7488
11:18:03.463168 7955.1 : -------} xcsGetMemFn rc=OK
11:18:03.463173 7955.1 : RetCode (OK)
11:18:03.463177 7955.1 : ------} xusAddStanzaLineList rc=OK
11:18:03.463182 7955.1 : ------{ xusGetLineType
11:18:03.463187 7955.1 : RetCode (xusLINE_IS_COMMENT)
11:18:03.463191 7955.1 : ------} xusGetLineType rc=OK
11:18:03.463198 7955.1 : ------{ xusAddStanzaLineList
11:18:03.463203 7955.1 : -------{ xcsGetMemFn
11:18:03.463208 7955.1 : component:24 function:15 length:2096 options:0 cbmindex:8 *pointer:4a5938
11:18:03.463212 7955.1 : -------} xcsGetMemFn rc=OK
11:18:03.463216 7955.1 : RetCode (OK)
11:18:03.463221 7955.1 : ------} xusAddStanzaLineList rc=OK
11:18:03.463225 7955.1 : ------{ xusGetLineType
11:18:03.463230 7955.1 : RetCode (xusLINE_IS_COMMENT)
11:18:03.463234 7955.1 : ------} xusGetLineType rc=OK
11:18:03.463238 7955.1 : ------{ xusAddStanzaLineList
11:18:03.463245 7955.1 : -------{ xcsGetMemFn
11:18:03.463250 7955.1 : component:24 function:15 length:2096 options:0 cbmindex:9 *pointer:4a61a8
11:18:03.463254 7955.1 : -------} xcsGetMemFn rc=OK
11:18:03.463259 7955.1 : RetCode (OK)
11:18:03.463263 7955.1 : ------} xusAddStanzaLineList rc=OK
11:18:03.463268 7955.1 : ------{ xusGetLineType
11:18:03.463272 7955.1 : RetCode (xusLINE_IS_COMMENT)
11:18:03.463277 7955.1 : ------} xusGetLineType rc=OK
11:18:03.463281 7955.1 : ------{ xusAddStanzaLineList
11:18:03.463285 7955.1 : -------{ xcsGetMemFn
11:18:03.463290 7955.1 : component:24 function:15 length:2096 options:0 cbmindex:-1 *pointer:4a7cf8
11:18:03.463302 7955.1 : -------} xcsGetMemFn rc=OK
11:18:03.463307 7955.1 : RetCode (OK)
11:18:03.463312 7955.1 : ------} xusAddStanzaLineList rc=OK
11:18:03.463317 7955.1 : ------{ xusGetLineType
11:18:03.463322 7955.1 : RetCode (xusLINE_IS_COMMENT)
11:18:03.463326 7955.1 : ------} xusGetLineType rc=OK
11:18:03.463330 7955.1 : ------{ xusAddStanzaLineList
11:18:03.463334 7955.1 : -------{ xcsGetMemFn
11:18:03.463340 7955.1 : component:24 function:15 length:2096 options:0 cbmindex:-1 *pointer:4a8568
11:18:03.463351 7955.1 : -------} xcsGetMemFn rc=OK
11:18:03.463356 7955.1 : RetCode (OK)
11:18:03.463360 7955.1 : ------} xusAddStanzaLineList rc=OK
11:18:03.463365 7955.1 : ------{ xusGetLineType
11:18:03.463370 7955.1 : RetCode (xusLINE_IS_COMMENT)
11:18:03.463374 7955.1 : ------} xusGetLineType rc=OK
11:18:03.463378 7955.1 : ------{ xusAddStanzaLineList
11:18:03.463383 7955.1 : -------{ xcsGetMemFn
11:18:03.463388 7955.1 : component:24 function:15 length:2096 options:0 cbmindex:-1 *pointer:49e7d8
11:18:03.463392 7955.1 : -------} xcsGetMemFn rc=OK
11:18:03.463409 7955.1 : RetCode (OK)
11:18:03.463413 7955.1 : ------} xusAddStanzaLineList rc=OK
11:18:03.463418 7955.1 : ------{ xusGetLineType
11:18:03.463423 7955.1 : RetCode (xusLINE_IS_COMMENT)
11:18:03.463427 7955.1 : ------} xusGetLineType rc=OK
11:18:03.463431 7955.1 : ------{ xusAddStanzaLineList
11:18:03.463436 7955.1 : -------{ xcsGetMemFn
11:18:03.463441 7955.1 : component:24 function:15 length:2096 options:0 cbmindex:-1 *pointer:49f048
11:18:03.463445 7955.1 : -------} xcsGetMemFn rc=OK
11:18:03.463450 7955.1 : RetCode (OK)
11:18:03.463456 7955.1 : ------} xusAddStanzaLineList rc=OK
11:18:03.463461 7955.1 : ------{ xusGetLineType
11:18:03.463466 7955.1 : RetCode (xusLINE_IS_COMMENT)
11:18:03.463470 7955.1 : ------} xusGetLineType rc=OK
11:18:03.463475 7955.1 : ------{ xusAddStanzaLineList
11:18:03.463479 7955.1 : -------{ xcsGetMemFn
11:18:03.463484 7955.1 : component:24 function:15 length:2096 options:0 cbmindex:-1 *pointer:4a28e8
11:18:03.463489 7955.1 : -------} xcsGetMemFn rc=OK
11:18:03.463493 7955.1 : RetCode (OK)
11:18:03.463497 7955.1 : ------} xusAddStanzaLineList rc=OK
11:18:03.463504 7955.1 : ------{ xusGetLineType
11:18:03.463508 7955.1 : RetCode (xusLINE_IS_COMMENT)
11:18:03.463512 7955.1 : ------} xusGetLineType rc=OK
11:18:03.463516 7955.1 : ------{ xusAddStanzaLineList
11:18:03.463519 7955.1 : -------{ xcsGetMemFn
11:18:03.463524 7955.1 : component:24 function:15 length:2096 options:0 cbmindex:-1 *pointer:4a3158
11:18:03.463528 7955.1 : -------} xcsGetMemFn rc=OK
11:18:03.463533 7955.1 : RetCode (OK)
11:18:03.463537 7955.1 : ------} xusAddStanzaLineList rc=OK
11:18:03.463541 7955.1 : ------{ xusGetLineType
11:18:03.463548 7955.1 : RetCode (xusLINE_IS_COMMENT)
11:18:03.463552 7955.1 : ------} xusGetLineType rc=OK
11:18:03.463556 7955.1 : ------{ xusAddStanzaLineList
11:18:03.463560 7955.1 : -------{ xcsGetMemFn
11:18:03.463565 7955.1 : component:24 function:15 length:2096 options:0 cbmindex:-1 *pointer:4a39c8
11:18:03.463570 7955.1 : -------} xcsGetMemFn rc=OK
11:18:03.463575 7955.1 : RetCode (OK)
11:18:03.463579 7955.1 : ------} xusAddStanzaLineList rc=OK
11:18:03.463584 7955.1 : ------{ xusGetLineType
11:18:03.463588 7955.1 : RetCode (xusLINE_IS_COMMENT)
11:18:03.463602 7955.1 : ------} xusGetLineType rc=OK
11:18:03.463606 7955.1 : ------{ xusAddStanzaLineList
11:18:03.463611 7955.1 : -------{ xcsGetMemFn
11:18:03.463616 7955.1 : component:24 function:15 length:2096 options:0 cbmindex:-1 *pointer:4a4238
11:18:03.463620 7955.1 : -------} xcsGetMemFn rc=OK
11:18:03.463625 7955.1 : RetCode (OK)
11:18:03.463630 7955.1 : ------} xusAddStanzaLineList rc=OK
11:18:03.463635 7955.1 : ------{ xusGetLineType
11:18:03.463639 7955.1 : RetCode (xusLINE_IS_COMMENT)
11:18:03.463643 7955.1 : ------} xusGetLineType rc=OK
11:18:03.463656 7955.1 : ------{ xusAddStanzaLineList
11:18:03.463661 7955.1 : -------{ xcsGetMemFn
11:18:03.463666 7955.1 : component:24 function:15 length:2096 options:0 cbmindex:-1 *pointer:4a4aa8
11:18:03.463671 7955.1 : -------} xcsGetMemFn rc=OK
11:18:03.463676 7955.1 : RetCode (OK)
11:18:03.463680 7955.1 : ------} xusAddStanzaLineList rc=OK
11:18:03.463685 7955.1 : ------{ xusGetLineType
11:18:03.463690 7955.1 : RetCode (xusLINE_IS_COMMENT)
11:18:03.463694 7955.1 : ------} xusGetLineType rc=OK
11:18:03.463698 7955.1 : ------{ xusAddStanzaLineList
11:18:03.463702 7955.1 : -------{ xcsGetMemFn
11:18:03.463716 7955.1 : component:24 function:15 length:2096 options:0 cbmindex:-1 *pointer:4c2898
11:18:03.463722 7955.1 : -------} xcsGetMemFn rc=OK
11:18:03.463727 7955.1 : RetCode (OK)
11:18:03.463731 7955.1 : ------} xusAddStanzaLineList rc=OK
11:18:03.463736 7955.1 : ------{ xusGetLineType
11:18:03.463741 7955.1 : RetCode (xusLINE_IS_COMMENT)
11:18:03.463745 7955.1 : ------} xusGetLineType rc=OK
11:18:03.463750 7955.1 : ------{ xusAddStanzaLineList
11:18:03.463754 7955.1 : -------{ xcsGetMemFn
11:18:03.463759 7955.1 : component:24 function:15 length:2096 options:0 cbmindex:-1 *pointer:4c3108
11:18:03.463766 7955.1 : -------} xcsGetMemFn rc=OK
11:18:03.463771 7955.1 : RetCode (OK)
11:18:03.463775 7955.1 : ------} xusAddStanzaLineList rc=OK
11:18:03.463780 7955.1 : ------{ xusGetLineType
11:18:03.463785 7955.1 : RetCode (xusLINE_IS_KEY)
11:18:03.463789 7955.1 : ------} xusGetLineType rc=OK
11:18:03.463793 7955.1 : ------{ xusAddStanzaLineList
11:18:03.463797 7955.1 : -------{ xcsGetMemFn
11:18:03.463808 7955.1 : component:24 function:15 length:2096 options:0 cbmindex:-1 *pointer:4c3978
11:18:03.463818 7955.1 : -------} xcsGetMemFn rc=OK
11:18:03.463823 7955.1 :
_________________
Yeah, well, you know, that's just, like, your opinion, man. - The Dude |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Aug 27, 2012 8:22 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Does it show where it starts to return the mangled error code - the 545284129 ? |
|
| Back to top |
|
|
| LouML |
| Posted: Tue Aug 28, 2012 3:35 am Post subject: |
|
|
Partisan
Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY
|
Solved!!!
I was able to create a new queue manager and start it, so that meant the problem was specific to the existing queue manager and not MQ in general or the server.
I grepped the trace files for the 545284129 and found it. Errors in this area pointed to permissions on /var/mqm/sockets/QM\!PBDR1\!01/qmgrlocl/stmpbdr1/subpool.lck. I did a find for all files owned by root in /var/mqm. The Unix admin removed the files in the qmgrs directory and I was able to start the queue manager.
| Code: |
bash-3.00$ strmqm QM.PBDR1.01
WebSphere MQ queue manager 'QM.PBDR1.01' starting.
The Queue Manager ended for reason 545284129 ''
bash-3.00$ find . -user root -ls
27 2 drwxr-sr-x 2 root mqm 3 Jun 29 2011 ./tivoli
30 1 -rw------- 1 root mqm 0 Aug 26 01:06 ./qmgrs/QM!PBDR1!01/@qmpersist/shm
143 1 -rw------- 1 root mqm 0 Aug 26 01:06 ./qmgrs/QM!PBDR1!01/@qmpersist/shm
114 1 -rw-r--r-- 1 root mqm 0 Aug 26 01:06 ./qmgrs/QM!PBDR1!01/@app/shmem/stm
2142 1 -rw-rw-r-- 1 root mqm 0 Jun 29 2011 ./qmgrs/@SYSTEM/nodelock
bash-3.00$ pwd
/var/mqm
bash-3.00$ find . -user root -ls
27 2 drwxr-sr-x 2 root mqm 3 Jun 29 2011 ./tivoli
2142 1 -rw-rw-r-- 1 root mqm 0 Jun 29 2011 ./qmgrs/@SYSTEM/nodelock
bash-3.00$ strmqm QM.PBDR1.01
WebSphere MQ queue manager 'QM.PBDR1.01' starting.
50 log records accessed on queue manager 'QM.PBDR1.01' during the log replay phase.
Log replay for queue manager 'QM.PBDR1.01' complete.
Transaction manager state recovered for queue manager 'QM.PBDR1.01'.
WebSphere MQ queue manager 'QM.PBDR1.01' started.
bash-3.00$ dspmq
QMNAME(QM.PBDR1.01) STATUS(Running)
QMNAME(QM.TEST.01) STATUS(Ended normally)
bash-3.00$
|
I don't know why this happened or why this prevented any FDC's from being created or errors to be logged.
Thanks to all for the help.
_________________
Yeah, well, you know, that's just, like, your opinion, man. - The Dude |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Aug 28, 2012 4:30 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I'm sure it happened because someone ran strmqm as the root user instead of as the mqm user.
I can't speculate on why it prevented FDCs from being written, except to suggest that a failure to acquire a semaphore shouldn't cause an FDC. It *should* cause an error to be written to the AMQERR logs, however.
But potentially the process didn't get far enough to open those logs. |
|
| Back to top |
|
|
| LouML |
| Posted: Tue Aug 28, 2012 4:57 am Post subject: |
|
|
Partisan
Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY
|
| mqjeff wrote: |
| I'm sure it happened because someone ran strmqm as the root user instead of as the mqm user. |
I can't imagine that anyone ran strmqm as root. First, the last messages in the error log are in line with the weekly reboot at 01:00 AM Sunday morning. They show the user as 'mqm'. MQ is started upon reboot from /etc/init.d/mqm. Besides, no one in our firm would even be awake at this time  And as I am the only MQ admin, no one would even know the command (unless they looked in the /etc/init.d/mqm file I suppose)
As I look at the log file more closely now, it does not appear that the startup was correct. The log below shows the shutdown at 01:00 AM, the partial startup at 01:06 and the next message is when we corrected the problem yesterday.
The startup from yesterday has the correct sequence of messages in the log compared to the few we received at 01:06 AM Sunday.
Anyway, it's working now and I guess it will remain a mystery.
| Code: |
-------------------------------------------------------------------------------
08/26/12 01:00:32 - Process(1214.1) User(mqm) Program(amqzxma0_nd)
Host(stmpbdr1)
AMQ8004: WebSphere MQ queue manager 'QM.PBDR1.01' ended.
EXPLANATION:
WebSphere MQ queue manager 'QM.PBDR1.01' ended.
ACTION:
None.
-------------------------------------------------------------------------------
08/26/12 01:06:21 - Process(1212.1) User(mqm) Program(amqzxma0_nd)
Host(stmpbdr1)
AMQ7229: 5 log records accessed on queue manager 'QM.PBDR1.01' during the log
replay phase.
EXPLANATION:
5 log records have been accessed so far on queue manager QM.PBDR1.01 during the
log replay phase in order to bring the queue manager back to a previously known
state.
ACTION:
None.
-------------------------------------------------------------------------------
08/26/12 01:06:21 - Process(1212.1) User(mqm) Program(amqzxma0_nd)
Host(stmpbdr1)
AMQ7230: Log replay for queue manager 'QM.PBDR1.01' complete.
EXPLANATION:
The log replay phase of the queue manager restart process has been completed
for queue manager QM.PBDR1.01.
ACTION:
None.
-------------------------------------------------------------------------------
08/26/12 01:06:21 - Process(1212.1) User(mqm) Program(amqzxma0_nd)
Host(stmpbdr1)
AMQ7231: 0 log records accessed on queue manager 'QM.PBDR1.01' during the
recovery phase.
EXPLANATION:
0 log records have been accessed so far on queue manager QM.PBDR1.01 during the
recovery phase of the transactions manager state.
ACTION:
None.
-------------------------------------------------------------------------------
08/26/12 01:06:21 - Process(1212.1) User(mqm) Program(amqzxma0_nd)
Host(stmpbdr1)
AMQ7232: Transaction manager state recovered for queue manager 'QM.PBDR1.01'.
EXPLANATION:
The state of transactions at the time the queue manager ended has been
recovered for queue manager QM.PBDR1.01.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:48 - Process(15190.4) User(mqm) Program(amqzmuc0)
Host(stmpbdr1)
AMQ5037: The Queue Manager task 'LOGGER-IO' has started.
EXPLANATION:
The Critical Utility Task Manager, processId(15190), has started the LOGGER-IO
task. This task has now started 1 times.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:49 - Process(15188.1) User(mqm) Program(amqzxma0_nd)
Host(stmpbdr1)
AMQ7229: 50 log records accessed on queue manager 'QM.PBDR1.01' during the log
replay phase.
EXPLANATION:
50 log records have been accessed so far on queue manager QM.PBDR1.01 during
the log replay phase in order to bring the queue manager back to a previously
known state.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:49 - Process(15188.1) User(mqm) Program(amqzxma0_nd)
Host(stmpbdr1)
AMQ7230: Log replay for queue manager 'QM.PBDR1.01' complete.
EXPLANATION:
The log replay phase of the queue manager restart process has been completed
for queue manager QM.PBDR1.01.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:49 - Process(15188.1) User(mqm) Program(amqzxma0_nd)
Host(stmpbdr1)
AMQ7231: 0 log records accessed on queue manager 'QM.PBDR1.01' during the
recovery phase.
EXPLANATION:
0 log records have been accessed so far on queue manager QM.PBDR1.01 during the
recovery phase of the transactions manager state.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:49 - Process(15188.1) User(mqm) Program(amqzxma0_nd)
Host(stmpbdr1)
AMQ7232: Transaction manager state recovered for queue manager 'QM.PBDR1.01'.
EXPLANATION:
The state of transactions at the time the queue manager ended has been
recovered for queue manager QM.PBDR1.01.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:49 - Process(15188.1) User(mqm) Program(amqzxma0_nd)
Host(stmpbdr1)
AMQ7233: 0 out of 0 in-flight transactions resolved for queue manager
'QM.PBDR1.01'.
EXPLANATION:
0 transactions out of 0 in-flight at the time queue manager QM.PBDR1.01 ended
have been resolved.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:49 - Process(15190.5) User(mqm) Program(amqzmuc0)
Host(stmpbdr1)
AMQ5037: The Queue Manager task 'CHECKPOINT' has started.
EXPLANATION:
The Critical Utility Task Manager, processId(15190), has started the CHECKPOINT
task. This task has now started 1 times.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:49 - Process(15191.5) User(mqm) Program(amqzmur0)
Host(stmpbdr1)
AMQ5037: The Queue Manager task 'APP-SIGNAL' has started.
EXPLANATION:
The Restartable Utility Task Manager, processId(15191), has started the
APP-SIGNAL task. This task has now started 1 times.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:49 - Process(15191.4) User(mqm) Program(amqzmur0)
Host(stmpbdr1)
AMQ5037: The Queue Manager task 'ERROR-LOG' has started.
EXPLANATION:
The Restartable Utility Task Manager, processId(15191), has started the
ERROR-LOG task. This task has now started 1 times.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:49 - Process(15188.1) User(mqm) Program(amqzxma0_nd)
Host(stmpbdr1)
AMQ8003: WebSphere MQ queue manager 'QM.PBDR1.01' started.
EXPLANATION:
WebSphere MQ queue manager 'QM.PBDR1.01' started.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:49 - Process(15191.6) User(mqm) Program(amqzmur0)
Host(stmpbdr1)
AMQ5037: The Queue Manager task 'STATISTICS' has started.
EXPLANATION:
The Restartable Utility Task Manager, processId(15191), has started the
STATISTICS task. This task has now started 1 times.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:49 - Process(15190.16) User(mqm) Program(amqzmuc0)
Host(stmpbdr1)
AMQ5037: The Queue Manager task 'ASYNCQ' has started.
EXPLANATION:
The Critical Utility Task Manager, processId(15190), has started the ASYNCQ
task. This task has now started 1 times.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:49 - Process(15190.17) User(mqm) Program(amqzmuc0)
Host(stmpbdr1)
AMQ5037: The Queue Manager task 'EXPIRER' has started.
EXPLANATION:
The Critical Utility Task Manager, processId(15190), has started the EXPIRER
task. This task has now started 1 times.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:49 - Process(15190.18) User(mqm) Program(amqzmuc0)
Host(stmpbdr1)
AMQ5037: The Queue Manager task 'DUR-SUBS-MGR' has started.
EXPLANATION:
The Critical Utility Task Manager, processId(15190), has started the
DUR-SUBS-MGR task. This task has now started 1 times.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:49 - Process(15193.1) User(mqm) Program(amqrrmfa)
Host(stmpbdr1)
AMQ9410: Repository manager started
EXPLANATION:
The repository manager started successfully.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:50 - Process(15191.7) User(mqm) Program(amqzmur0)
Host(stmpbdr1)
AMQ5037: The Queue Manager task 'MARKINTSCAN' has started.
EXPLANATION:
The Restartable Utility Task Manager, processId(15191), has started the
MARKINTSCAN task. This task has now started 1 times.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:50 - Process(15190.19) User(mqm) Program(amqzmuc0)
Host(stmpbdr1)
AMQ5037: The Queue Manager task 'TOPIC-TREE' has started.
EXPLANATION:
The Critical Utility Task Manager, processId(15190), has started the TOPIC-TREE
task. This task has now started 1 times.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:50 - Process(15192.5) User(mqm) Program(amqzmuf0)
Host(stmpbdr1)
AMQ5037: The Queue Manager task 'QPUBSUB-QUEUE-NLCACHE' has started.
EXPLANATION:
The Publish/Subscribe Utility Task Manager, processId(15192), has started the
QPUBSUB-QUEUE-NLCACHE task. This task has now started 1 times.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:50 - Process(15190.21) User(mqm) Program(amqzmuc0)
Host(stmpbdr1)
AMQ5037: The Queue Manager task 'PRESERVED-Q' has started.
EXPLANATION:
The Critical Utility Task Manager, processId(15190), has started the
PRESERVED-Q task. This task has now started 1 times.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:50 - Process(15192.6) User(mqm) Program(amqzmuf0)
Host(stmpbdr1)
AMQ5037: The Queue Manager task 'QPUBSUB-SUBPT-NLCACHE' has started.
EXPLANATION:
The Publish/Subscribe Utility Task Manager, processId(15192), has started the
QPUBSUB-SUBPT-NLCACHE task. This task has now started 1 times.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:50 - Process(15192.4) User(mqm) Program(amqzmuf0)
Host(stmpbdr1)
AMQ5037: The Queue Manager task 'QPUBSUB-CTRLR' has started.
EXPLANATION:
The Publish/Subscribe Utility Task Manager, processId(15192), has started the
QPUBSUB-CTRLR task. This task has now started 1 times.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:50 - Process(15191.8) User(mqm) Program(amqzmur0)
Host(stmpbdr1)
AMQ5037: The Queue Manager task 'PUBSUB-DAEMON' has started.
EXPLANATION:
The Restartable Utility Task Manager, processId(15191), has started the
PUBSUB-DAEMON task. This task has now started 1 times.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:50 - Process(15190.20) User(mqm) Program(amqzmuc0)
Host(stmpbdr1)
AMQ5037: The Queue Manager task 'Q-DELETION' has started.
EXPLANATION:
The Critical Utility Task Manager, processId(15190), has started the Q-DELETION
task. This task has now started 1 times.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:50 - Process(15195.1) User(mqm) Program(amqzmgr0)
Host(stmpbdr1)
AMQ5022: The Channel Initiator has started. ProcessId(15197).
EXPLANATION:
The Channel Initiator process has started.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:50 - Process(15195.1) User(mqm) Program(amqzmgr0)
Host(stmpbdr1)
AMQ5024: The Command Server has started. ProcessId(15198).
EXPLANATION:
The Command Server process has started.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:50 - Process(15195.1) User(mqm) Program(amqzmgr0)
Host(stmpbdr1)
AMQ5026: The Listener 'LISTENER1' has started. ProcessId(15199).
EXPLANATION:
The Listener process has started.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:50 - Process(15197.1) User(mqm) Program(runmqchi)
Host(stmpbdr1)
AMQ8024: WebSphere MQ channel initiator started.
EXPLANATION:
The channel initiator for queue has been started.
ACTION:
None.
-------------------------------------------------------------------------------
08/27/12 12:35:51 - Process(15200.1) User(mqm) Program(amqfqpub)
Host(stmpbdr1)
AMQ5806: WebSphere MQ Publish/Subscribe broker started for queue manager
QM.PBDR1.01.
EXPLANATION:
WebSphere MQ Publish/Subscribe broker started for queue manager QM.PBDR1.01.
ACTION:
None.
-------------------------------------------------------------------------------
|
_________________
Yeah, well, you know, that's just, like, your opinion, man. - The Dude |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Aug 28, 2012 5:20 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
If nobody was awake, how did the queue manager attempt to start twice?
I see "./tivoli" in there. Is it possible that an operations console user did something to cause tivoli to start the queue manager? Or that the tivoli monitoring agent was accidentally started by the root user rather than the mqm user, and this caused it to acquire a root-owned lock on MQ shared memory? |
|
| Back to top |
|
|
| LouML |
| Posted: Tue Aug 28, 2012 5:33 am Post subject: |
|
|
Partisan
Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY
|
| mqjeff wrote: |
If nobody was awake, how did the queue manager attempt to start twice?
I see "./tivoli" in there. Is it possible that an operations console user did something to cause tivoli to start the queue manager? Or that the tivoli monitoring agent was accidentally started by the root user rather than the mqm user, and this caused it to acquire a root-owned lock on MQ shared memory? |
It didn't attempt it twice. It only attempted to start the queue manager at 01:06 AM Sunday.
The Tivoli monitoring agent is no longer installed. The only file in that directory is
| Code: |
bash-3.00$ ls -la /var/mqm/tivoli/
total 8
drwxr-sr-x 2 root mqm 3 Jun 29 2011 .
drwxrwsr-x 12 mqm mqm 15 Aug 27 12:17 ..
-r--r--r-- 1 mqm mqm 100 Aug 13 2009 WMQBaseS070000.sys
bash-3.00$
|
_________________
Yeah, well, you know, that's just, like, your opinion, man. - The Dude |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Aug 28, 2012 7:52 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Next time and just for caution I would not delete the root owned files but just have them change ownership. Then if there still is a problem delete them... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| LouML |
| Posted: Wed Aug 29, 2012 2:07 am Post subject: |
|
|
Partisan
Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY
|
| fjb_saper wrote: |
| Next time and just for caution I would not delete the root owned files but just have them change ownership. Then if there still is a problem delete them... |
Actually, we tar'd them first, so we still have them
_________________
Yeah, well, you know, that's just, like, your opinion, man. - The Dude |
|
| Back to top |
|
|
| Andyh |
| Posted: Wed Aug 29, 2012 6:01 am Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 239
|
545284129 indicates some kind of access control issue.
Try starting the queue manager with trace active and then grep for xecP_E_ACCESS_DENIED in the formatted trace.
Have you made any changes to the mqm userid on your system ?
Do the ownerships of the executables in the mq bin directory look correct ?
Do the file permissions of the files in the queue manager directory tree look correct ? |
|
| Back to top |
|
|
| LouML |
| Posted: Wed Aug 29, 2012 6:34 am Post subject: |
|
|
Partisan
Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY
|
| Andyh wrote: |
545284129 indicates some kind of access control issue.
Try starting the queue manager with trace active and then grep for xecP_E_ACCESS_DENIED in the formatted trace.
Have you made any changes to the mqm userid on your system ?
Do the ownerships of the executables in the mq bin directory look correct ?
Do the file permissions of the files in the queue manager directory tree look correct ? |
We've solved the problem (see a few posts up)
We ran the trace and found the 545284129 using grep
There were no changes to the mqm userid
Ownerships and permissions of executables were all correct.
The problem was some leftover shared memory files in the qmgrs tree were owned by root
| Code: |
bash-3.00$ strmqm QM.PBDR1.01
WebSphere MQ queue manager 'QM.PBDR1.01' starting.
The Queue Manager ended for reason 545284129 ''
bash-3.00$ find . -user root -ls
27 2 drwxr-sr-x 2 root mqm 3 Jun 29 2011 ./tivoli
30 1 -rw------- 1 root mqm 0 Aug 26 01:06 ./qmgrs/QM!PBDR1!01/@qmpersist/shm
143 1 -rw------- 1 root mqm 0 Aug 26 01:06 ./qmgrs/QM!PBDR1!01/@qmpersist/shm
114 1 -rw-r--r-- 1 root mqm 0 Aug 26 01:06 ./qmgrs/QM!PBDR1!01/@app/shmem/stm
2142 1 -rw-rw-r-- 1 root mqm 0 Jun 29 2011 ./qmgrs/@SYSTEM/nodelock
bash-3.00$ |
_________________
Yeah, well, you know, that's just, like, your opinion, man. - The Dude |
|
| Back to top |
|
|
| rcp_mq |
| Posted: Wed Aug 29, 2012 10:08 pm Post subject: |
|
|
Centurion
Joined: 13 Dec 2011
Posts: 133
|
I know this is absolutely not related to your specific issue but is relevant to the subject of your query.
In a desperate situation where my queue manager just won't start (the dirty impatient developer killed the queue manager because it was taking a long time to end) this is what i did,
I copied all the particular data and log files, deleted the queue manager. Recreated the queue manager with the same properties (used the MQSC script file) copied back the data and log files.
At this point if i do not delete the older queue manager's folder, I'll have a NEWQMGR.0000 and that is what will be in qm.ini. (and that's a bad thing)
Refresh the objects (strmqm -c qmgr) and start the Qmgr.
This is a desperate solution. I apologize for stealing a bit of your thunder. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
