| Author |
Message
|
| hellosummer2013 |
 Posted: Wed May 23, 2012 8:59 pm Post subject: Unable to end the queue manager occasionally |
 |
|
Newbie
Joined: 23 May 2012
Posts: 4
|
May I know any people can help to analyze this issue ? Thanks!!!
WAS : Version 5.1.1.19
MQ : Version 530.7 CSD07
The WAS services and MQ queue manager are scheduled to stop at 3AM and start at 4AM every day. But the MQ queue failed to be ended occasionally.
WAS Log:
[5/22/12 3:01:14:999 JST] 502f9a67 ApplicationMg A WSVR0220I: Application stopped: RSS-PROD-EAR
[5/22/12 3:01:15:015 JST] 502f9a67 ApplicationMg A WSVR0217I: Stopping application: adminconsole
[5/22/12 3:01:15:028 JST] 502f9a67 WebContainer A SRVE0170I: Stopping Web Module: adminconsole.
[5/22/12 3:01:15:043 JST] 502f9a67 WebGroup I SRVE0180I: [adminconsole] [/admin] [Servlet.LOG]: InvokerServlet: destroy
[5/22/12 3:01:15:045 JST] 502f9a67 WebGroup I SRVE0180I: [adminconsole] [/admin] [Servlet.LOG]: SimpleFileServlet: destroy
[5/22/12 3:01:15:046 JST] 502f9a67 WebGroup I SRVE0180I: [adminconsole] [/admin] [Servlet.LOG]: JSP 1.2 Processor: destroy
[5/22/12 3:01:15:047 JST] 502f9a67 WebGroup I SRVE0180I: [adminconsole] [/admin] [Servlet.LOG]: SecureCleanup: destroy
[5/22/12 3:01:15:052 JST] 502f9a67 ApplicationMg A WSVR0220I: Application stopped: adminconsole
[5/22/12 3:01:17:254 JST] 502f9a67 JMSEmbeddedPr A MSGS0054I: Stopping the Broker
[5/22/12 3:01:23:129 JST] 502f9a67 JMSEmbeddedPr A MSGS0055I: Broker is stopped
[5/22/12 3:01:23:143 JST] 502f9a67 JMSEmbeddedPr A MSGS0056I: Stopping the Queue Manager
MQ queue manager Log:
05/22/12 03:01:33
AMQ9542: Queue manager is ending.
Explanation:
The program will end because the queue manager is quiescing.
ACTION:
Done.
----amgrmssa.c : 2098 --------------------------------------------------
05/22/12 03:01:35
AMQ9542: Queue manager is ending.
Explanation:
The program will end because the queue manager is quiescing.
ACTION:
Done.
The Queue Manager remain "Ending Immediately" status.
We could not access Queue manager working directory folder: zsocketEC. Failed to show the folder properties and execute "cd zsocketEC" command.
[/img] |
|
| Back to top |
|
 |
| exerk |
| Posted: Wed May 23, 2012 11:55 pm Post subject: Re: Unable to end the queue manager occasionally |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| hellosummer2013 wrote: |
WAS : Version 5.1.1.19
MQ : Version 530.7 CSD07 |
These are seriously out of date, and I mean years out of date. Why have you not upgraded? Why are you not at the latest CSD/FixPack for that version of WMQ?
| hellosummer2013 wrote: |
| But the MQ queue failed to be ended occasionally. |
Queues don't end, queue managers end.
| hellosummer2013 wrote: |
| [5/22/12 3:01:23:143 JST] 502f9a67 JMSEmbeddedPr A MSGS0056I: Stopping the Queue Manager |
So WAS is issuing the stop command to the queue manager?
| hellosummer2013 wrote: |
| We could not access Queue manager working directory folder: zsocketEC. Failed to show the folder properties and execute "cd zsocketEC" command. |
Any FDC files cut? Have you researched any known problems with accessing that directory?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| hellosummer2013 |
| Posted: Thu May 24, 2012 12:23 pm Post subject: Thanks for your reply |
|
|
Newbie
Joined: 23 May 2012
Posts: 4
|
(Q1) Why I don't upgrade it
For some reasons, the upgrading project has been pending since the application on WAS failed to be migarated so far.
(Q2) Queues don't end, queue managers end.
I missed the "manager", I mean queue managers unable to be ended.
(Q3) So WAS is issuing the stop command to the queue manager?
Yes.
we have three WAS instance and three queue managers.
devWASInstance --> devQueueManager
uatWASInstance --> uatWASInstance
prdWASInstnace --> prdWASInstance
Each WAS instance issue the stop command to the relative queue manager when it self be stopped.
There is a schedule job (3AM) to stop the WAS instances one by one.
We have the issue 4 time this year (it's stable before).
sometime the devQueuemanager was unable to be ended.
But sometimes the prdQueueManager was unable to be ended.
The 'zsocketEC' folder couln't be "ls-l zsocketEC" when the QM failed to be ended. ("ls zsocketEC" command is ok)
(Q4) There is no FDC file generated when the QM(prdQueueManager) failed to be ended (5/22/2012 3AM )
But at 7AM, when the server be restarted to resume the services. All the MQ QMs were started , and then some FDC files generated for QM(prdQueueManager). |
|
| Back to top |
|
|
| exerk |
| Posted: Thu May 24, 2012 12:31 pm Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
And what's in the head of the FDC files? And do you have all your eggs in one basket, i.e all three environments on the same server? Have you Googled zsocketEC to see what hits you get that are WMQ related?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| hellosummer2013 |
| Posted: Thu May 24, 2012 5:46 pm Post subject: |
|
|
Newbie
Joined: 23 May 2012
Posts: 4
|
Yes, all the environment are on same server (Not me build that server  )
Below is the part of the FDC files:
+-----------------------------------------------------------------------------+
| |
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- Tuesday May 22 05:55:55 JST 2012 |
| Host Name :- ***** (AIX 5.2) |
| LVLS :- 530.7 CSD07 |
| Product Long Name :- WebSphere MQ for AIX |
| Vendor :- IBM |
| Probe Id :- XC338001 |
| Application Name :- MQM |
| Component :- xehAsySignalHandler |
| Build Date :- May 27 2004 |
| CMVC level :- p530-07-L040527 |
| Build Type :- IKAP - (Production) |
| UserID :- 00000202 (mqm) |
| Program Name :- endmqm |
| Process :- 00573616 |
| Thread :- 00000002 |
| Major Errorcode :- xecE_W_UNEXPECTED_ASYNC_SIGNAL |
| Minor Errorcode :- OK |
| Probe Type :- MSGAMQ6209 |
| Probe Severity :- 3 |
| Probe Description :- AMQ6109: An internal WebSphere MQ error has occurred. |
| FDCSequenceNumber :- 0 |
| Arith1 :- 2 2 |
| Arith2 :- 573616 8c0b0 |
| Comment1 :- SIGINT |
| |
| |
+-----------------------------------------------------------------------------+
MQM Function Stack
xehAsySignalMonitor
xehHandleAsySignal
xcsFFST
MQM Trace History
{ xppInitialiseDestructorRegistrations
} xppInitialiseDestructorRegistrations rc=OK
{ xehAsySignalMonitor
-{ xppPostAsySigMonThread
-} xppPostAsySigMonThread rc=OK
-{ xehHandleAsySignal
--{ xcsRequestThreadMutexSem
--} xcsRequestThreadMutexSem rc=OK
--{ xcsFFST
Component Dumps (Thread 00000002)
+-----------------------------------------------------------------------------+
| |
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- Tuesday May 22 07:07:11 JST 2012 |
| Host Name :- **** (AIX 5.2) |
| LVLS :- 530.7 CSD07 |
| Product Long Name :- WebSphere MQ for AIX |
| Vendor :- IBM |
| Probe Id :- XC338001 |
| Application Name :- MQM |
| Component :- xehAsySignalHandler |
| Build Date :- May 27 2004 |
| CMVC level :- p530-07-L040527 |
| Build Type :- IKAP - (Production) |
| UserID :- 00000202 (root) |
| Program Name :- amqzllp0 |
| Process :- 00385248 |
| Thread :- 00000002 |
| Major Errorcode :- xecE_W_UNEXPECTED_ASYNC_SIGNAL |
| Minor Errorcode :- OK |
| Probe Type :- MSGAMQ6209 |
| Probe Severity :- 3 |
| Probe Description :- AMQ6109: An internal WebSphere MQ error has occurred. |
| FDCSequenceNumber :- 0 |
| Arith1 :- 15 f |
| Arith2 :- 385248 5e0e0 |
| Comment1 :- SIGTERM |
| |
| |
+-----------------------------------------------------------------------------+
Below is the information from internet for zsocketEC:
The folder beneath /var/mqm/qmgrs/<qmname> that contain UNIX-domain socket files. To provide robust server bindings and to shield queue managers from abrupt application termination. |
|
| Back to top |
|
|
| exerk |
| Posted: Thu May 24, 2012 10:09 pm Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| hellosummer2013 wrote: |
Below is the information from internet for zsocketEC:
The folder beneath /var/mqm/qmgrs/<qmname> that contain UNIX-domain socket files. To provide robust server bindings and to shield queue managers from abrupt application termination. |
And you found nothing in regard to permissions on that directory?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
|
|
