Author |
Message
|
kevinf2349 |
Posted: Thu Oct 12, 2006 6:11 am Post subject: Probe id XC371020 |
|
|
 Grand Master
Joined: 28 Feb 2003 Posts: 1311 Location: USA
|
OK.
We are already talking to IBM about this but I thought I would ask for ideas here too.
Yesterday afternoon one of our development Windows queue managers threw a FDC file and died.
The queue manager is showing 'Unavailable' on MQ windows explorer and all the services under the queue manager are showing 'Stopped'
The header for the FDC follows:
Code: |
+-----------------------------------------------------------------------------+
| |
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- Wed October 11 15:51:51 Central Daylight Time 2006 |
| Host Name :- XXXXXXK1 (Windows Ver 5.2 Build 3790: Service Pack 1) |
| PIDS :- 5724B4100 |
| LVLS :- 530.11 CSD11 |
| Product Long Name :- WebSphere MQ for Windows |
| Vendor :- IBM |
| Probe Id :- XC371020 |
| Application Name :- MQM |
| Component :- xstServerRequest |
| Build Date :- Aug 2 2005 |
| CMVC level :- p530-11-L050802 |
| Build Type :- IKAP - (Production) |
| UserID :- MUSR_MQADMIN |
| Process Name :- C:\Program Files\IBM\WebSphere MQ\bin\AMQPCSEA.EXE |
| Process :- 00002172 |
| Thread :- 00000001 |
| QueueManager :- WINFPLDV |
| Major Errorcode :- xecF_E_UNEXPECTED_SYSTEM_RC |
| Minor Errorcode :- OK |
| Probe Type :- MSGAMQ6119 |
| Probe Severity :- 2 |
| Probe Description :- AMQ6119: An internal WebSphere MQ error has occurred |
| (Subpool Server 1 pid[2092] has terminated while application connected.) |
| FDCSequenceNumber :- 0 |
| Arith2 :- 2092 82c |
| Comment1 :- Subpool Server 1 pid[2092] has terminated while |
| application con |
| |
+-----------------------------------------------------------------------------+ |
This implies that the command server 'bought the farm'. The only hit I get on the probe id is muttering about over 300,000 persistent messages. Firstly I would be mildly surprised if this were the case, but secondly we can't get the queue manager up to find out!
Short of a reboot, has anyone got any ideas of how to get out of this?
We could reboot that that almost requires an act of congress and would mean dropping 2 working queue managers.
As I say, we are still working with IBM on this but I thought I would throw it out there.
(We are still investigating what (if anything) actually happened on the box at the time of failure) ....not really anything in the error log for the queue manager other than the command server getting a 2009 at the time of failure.
Update
We now have the queue manager started. We had to kill the process AMQRMPPA. Once this was done the queue manager started. As expected there are no where near 300,000 persistent messages on any queue nor indeed on the queue manager itself in total.
Investigations continue at a more leisurely pace  |
|
Back to top |
|
 |
fjb_saper |
Posted: Thu Oct 12, 2006 3:37 pm Post subject: |
|
|
 Grand High Poobah
Joined: 18 Nov 2003 Posts: 20756 Location: LI,NY
|
Don't know how to handle shared memory under windows. Under Unix I would use amqiclen to clean up all shared memory associated with this qmgr.
Enjoy  _________________ MQ & Broker admin |
|
Back to top |
|
 |
Nick Lethbridge |
Posted: Wed Oct 18, 2006 2:57 am Post subject: |
|
|
 Voyager
Joined: 13 Aug 2001 Posts: 88 Location: Santander, UK
|
Hi Kevin,
I'll see your "XC371020", and raise you...
....we have started getting this as well.
+-----------------------------------------------------------------------------+
| |
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- Tue October 17 16:45:49 GMT Daylight Time 2006 |
| Host Name :- UKPWG174WIN (Windows Ver 5.2 Build 3790: Service Pack |
| 1) |
| PIDS :- 5724B4100 |
| LVLS :- 530.11 CSD11 |
| Product Long Name :- WebSphere MQ for Windows |
| Vendor :- IBM |
| Probe Id :- XC371020 |
| Application Name :- MQM |
| Component :- xstServerRequest |
| Build Date :- Aug 2 2005 |
| CMVC level :- p530-11-L050802 |
| Build Type :- IKAP - (Production) |
| UserID :- mqsiuat |
| Process Name :- D:\Program Files\IBM\WebSphere MQ\bin\RUNMQCHI.exe |
| Process :- 00000752 |
| Thread :- 00000001 |
| QueueManager :- WMBCMA1 |
| Major Errorcode :- xecF_E_UNEXPECTED_SYSTEM_RC |
| Minor Errorcode :- OK |
| Probe Type :- MSGAMQ6119 |
| Probe Severity :- 2 |
| Probe Description :- AMQ6119: An internal WebSphere MQ error has occurred |
| (Subpool Server 1 pid[6040] has terminated while application connected.) |
| FDCSequenceNumber :- 0 |
| Arith2 :- 6040 1798 |
| Comment1 :- Subpool Server 1 pid[6040] has terminated while |
| application con |
| |
+-----------------------------------------------------------------------------+
I tried rebooting the server, but it didn't make any difference.
Did IBM have any idea of what might be causing it ?
Perhaps it's just an "issue" with Fix Pack 11 ?
Regards,
Nick. |
|
Back to top |
|
 |
Ross |
Posted: Tue Nov 21, 2006 6:14 am Post subject: |
|
|
Centurion
Joined: 15 Jun 2005 Posts: 127 Location: Ireland
|
I am getting this too.
Did anyone figure out a reason for it?
Thanks,
Ross. |
|
Back to top |
|
 |
jefflowrey |
Posted: Tue Nov 21, 2006 6:19 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002 Posts: 19981
|
Are you also at MQ v5.3, FixPack/CSD 11? _________________ I am *not* the model of the modern major general. |
|
Back to top |
|
 |
Ross |
Posted: Tue Nov 21, 2006 6:38 am Post subject: |
|
|
Centurion
Joined: 15 Jun 2005 Posts: 127 Location: Ireland
|
It has happened at CSD10, and I upgraded to CSD12 and it still happens. |
|
Back to top |
|
 |
Gaya3 |
Posted: Tue Nov 21, 2006 6:43 am Post subject: |
|
|
 Jedi
Joined: 12 Sep 2006 Posts: 2493 Location: Boston, US
|
Hi
This is due to large number of persistent messages in the application queue.
Please refer the below site for more information
http://www-1.ibm.com/support/docview.wss?uid=swg21178058
Thanks and Regards
Gayathri _________________ Regards
Gayathri
-----------------------------------------------
Do Something Before you Die |
|
Back to top |
|
 |
jefflowrey |
Posted: Tue Nov 21, 2006 6:45 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002 Posts: 19981
|
Gaya3 wrote: |
This is due to large number of persistent messages in the application queue. |
It can be caused by that, yes.
It seems clear from the other posters that it doesn't have to be caused by that.
Ross - can you post the header block of your FDC - also the stack trace. _________________ I am *not* the model of the modern major general. |
|
Back to top |
|
 |
Ross |
Posted: Tue Nov 21, 2006 7:03 am Post subject: |
|
|
Centurion
Joined: 15 Jun 2005 Posts: 127 Location: Ireland
|
Here is the FDC below.
I have no messages in any queue, so that isn't the problem.
How do I get the Stack Trace?
I suspect it may be to do with shared memory server or security access, but not 110% sure how to verify this.
Thanks.
+-----------------------------------------------------------------------------+
| |
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- Tue November 21 08:00:11 Eastern Standard Time 2006 |
| Host Name :- CBPPAP13 (Windows 2000 Build 2195: Service Pack 4) |
| PIDS :- 5724B4100 |
| LVLS :- 530.12 CSD12 |
| Product Long Name :- WebSphere MQ for Windows |
| Vendor :- IBM |
| Probe Id :- XC371020 |
| Application Name :- MQM |
| Component :- xstServerRequest |
| Build Date :- Dec 8 2005 |
| CMVC level :- p530-12-L051208 |
| Build Type :- IKAP - (Production) |
| UserID :- MUSR_MQADMIN |
| Process Name :- C:\Program Files\IBM\WebSphere MQ\bin\AMQPCSEA.EXE |
| Process :- 00001928 |
| Thread :- 00000001 |
| QueueManager :- CBPPAP13 |
| Major Errorcode :- xecF_E_UNEXPECTED_SYSTEM_RC |
| Minor Errorcode :- OK |
| Probe Type :- MSGAMQ6119 |
| Probe Severity :- 2 |
| Probe Description :- AMQ6119: An internal WebSphere MQ error has occurred |
| (Subpool Server 1 pid[1584] has terminated while application connected.) |
| FDCSequenceNumber :- 0 |
| Arith2 :- 1584 630 |
| Comment1 :- Subpool Server 1 pid[1584] has terminated while |
| application con |
| |
+-----------------------------------------------------------------------------+ |
|
Back to top |
|
 |
jefflowrey |
Posted: Tue Nov 21, 2006 7:06 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002 Posts: 19981
|
The stack trace would be the next section of the FDC. _________________ I am *not* the model of the modern major general. |
|
Back to top |
|
 |
Ross |
Posted: Tue Nov 21, 2006 7:10 am Post subject: |
|
|
Centurion
Joined: 15 Jun 2005 Posts: 127 Location: Ireland
|
MQM Function Stack
pcmMain
MQGET
zstMQGET
ziiBreakConnection
zcpDetachPipe
xcsDisconnectSharedMemSet
xstDisconnectExtent
xstStorageRequest
xstServerRequest
xcsFFST |
|
Back to top |
|
 |
jpeela |
Posted: Tue Dec 05, 2006 1:17 am Post subject: |
|
|
 Centurion
Joined: 23 Jan 2006 Posts: 139
|
Hi Kevin,
we too are under the similar situation.Did you solve yours.
If so could you share with us.
awaiting your reply.
Thanks in advance. _________________ Jogi |
|
Back to top |
|
 |
kevinf2349 |
Posted: Tue Dec 05, 2006 5:31 am Post subject: |
|
|
 Grand Master
Joined: 28 Feb 2003 Posts: 1311 Location: USA
|
jpeela wrote: |
Hi Kevin,
we too are under the similar situation.Did you solve yours.
If so could you share with us.
awaiting your reply.
Thanks in advance. |
Quote: |
Update
We now have the queue manager started. We had to kill the process AMQRMPPA. Once this was done the queue manager started. As expected there are no where near 300,000 persistent messages on any queue nor indeed on the queue manager itself in total.
|
|
|
Back to top |
|
 |
Ross |
Posted: Tue Dec 05, 2006 5:57 am Post subject: |
|
|
Centurion
Joined: 15 Jun 2005 Posts: 127 Location: Ireland
|
I was able to get this error by starting the qmgr using strmqm command. It works fine if started with MQ Explorer.
How are you starting it?
Are you using BMC Patrol?
What MQ Version/CSD? |
|
Back to top |
|
 |
jpeela |
Posted: Tue Dec 05, 2006 9:47 pm Post subject: |
|
|
 Centurion
Joined: 23 Jan 2006 Posts: 139
|
Hi all,
We too were abble to start the quemanager successfully and the qmgr is running fine presently, but the quemanager is ending unexpectedly frequently for the last few days.We dont know the exact reason , why this is happening.And we doubt that this would happen again.Any remedy against this.
We are using MQ5.3 on windows with CSD7.BUt CSD 12 also does not solve this problem As for the previous posts.Any solution or rsuggesions
Thanx in advance _________________ Jogi |
|
Back to top |
|
 |
|