| Author |
Message
|
| kun.leeing |
 Posted: Thu Apr 02, 2009 6:43 am Post subject: Errors occurs after restarting qmgr |
 |
|
Disciple
Joined: 27 Sep 2008
Posts: 171
|
Hi, guys.
For restarting qmgr, I run 'kill -9 some processes that can not be cleaned (amqiclen)' and then the qmgr got some problems . I restarted my server and qmgr, also have the problems. symptoms:
I） clients could not create sessions,got error
| Quote: |
| javax.jms.JMSException: MQJMS3005: pub/sub “com.ibm.mq.MQException: MQJE001: completion code“2”，reason code“2009” |
II) On server, some channels are abnormal. Sender and Receiver used normally before can not run now, always in retrying or inactive status(throwing error 'AMQ9604').
III）The port that clients connected at all times can not be opened. When I start it manually, it go 'stopped' right away.
Here is the FDC files(Server and Client).
Client:
| Quote: |
FDCTitle
Product :- IBM WebSphere MQ classes for JMS
Date/Time :- Thu Apr 02 15:58:30 CST 2009
System time :- 1238659110140
Operating System :- Windows XP
UserID :- Administrator
Java Vendor :- Sun Microsystems Inc.
Java Version :- 10.0-b22
Source Class :- com.ibm.msg.client.wmq.v6.jms.internal.MQConnection
Source Method :- createUniqueID(MQQueueManager)
ProbeID :- XO008005
Thread :- name=main priority=5 group=main ccl=sun.misc.Launcher$AppClassLoader@18d107f
Data
----
| Comment :- failed to backout conID message from queue
| Exception :- ExceptionDepth is 2
| Exception :-
| | Cause:1 :- com.ibm.mq.MQException: MQJE001: completion code “2”，reason code“2009”。
| | Message:1 :- MQJE001: completion code “2” , reason code“2009”。
| | StackTrace:1 :- com.ibm.mq.MQException: MQJE001: completion code “2”，reason code“2009”。
| | at com.ibm.msg.client.wmq.v6.base.internal.MQQueueManager.backout(MQQueueManager.java:520)
| | at com.ibm.msg.client.wmq.v6.jms.internal.MQConnection.createUniqueID(MQConnection.java:1833)
| | at com.ibm.msg.client.wmq.v6.jms.internal.MQConnection.setSubscriptionStores(MQConnection.java:2923)
| | at com.ibm.msg.client.wmq.v6.jms.internal.MQConnection.getMQPubSubServices(MQConnection.java:2211)
| | at com.ibm.msg.client.wmq.v6.jms.internal.MQSession.addPubSubServices(MQSession.java:1002)
| | at com.ibm.msg.client.wmq.v6.jms.internal.MQSession.createTProducer(MQSession.java:3590)
| | at com.ibm.msg.client.wmq.v6.jms.internal.MQSession.createProducer(MQSession.java:2882)
| | at com.ibm.msg.client.wmq.v6.jms.internal.MQSession.createProducer(MQSession.java:2929)
| | at com.ibm.msg.client.jms.internal.JmsSessionImpl.createProducer(JmsSessionImpl.java:1123)
| | at com.ibm.mq.jms.MQSession.createProducer(MQSession.java:593)
| | at com.zh.mqpack.ZHMQOperator.send(ZHMQOperator.java:266)
| | at com.zh.mqpack.ZHMQOperator.main(ZHMQOperator.java:422)
| |
| Message :- MQJMS1016 |
Server:
| Quote: |
AMQ9604: Channel 'SUB.TO.SAC' terminated unexpectedly
EXPLANATION:
The process or thread executing channel 'SUB.TO.SAC' is no longer running.
The check process system call returned 545284357 for process 7922.
ACTION:
No immediate action is required because the channel entry has been removed from the list of running channels. Inform the system administrator who should
examine the operating system procedures to determine why the channel process has terminated. |
And FDC file is :
| Quote: |
+-----------------------------------------------------------------------------+
| |
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- 02 17:07:31 CST 2009 |
| Host Name :- localhost.localdomain (Linux 2.6.9-55.ELsmp) |
| PIDS :- 5724H7204 |
| LVLS :- 6.0.2.5 |
| Product Long Name :- WebSphere MQ for Linux (x86 platform) |
| Vendor :- IBM |
| Probe Id :- RM027001 |
| Application Name :- MQM |
| Component :- rriAddStatusEntry |
| SCCS Info :- lib/remote/amqrcsia.c, 1.128.1.9 |
| Line Number :- 1047 |
| Build Date :- Sep 24 2008 |
| CMVC level :- p600-205-080922 |
| Build Type :- IKAP - (Production) |
| UserID :- 00000501 (mqm) |
| Program Name :- amqrmppa |
| Addressing mode :- 32-bit |
| Process :- 9659 |
| Thread-Process :- 9659 |
| Thread :- 4 |
| ThreadingModel :- PosixThreads |
| QueueManager :- ZH110!SACRIFICED!QMANAGER |
| ConnId(1) IPCC :- 902 |
| Last HQC :- 1.0.0-29940 |
| Last HSHMEMB :- 0.0.0-0 |
| Major Errorcode :- rrcE_STATUS_TABLE_CORRUPT |
| Minor Errorcode :- OK |
| Probe Type :- MSGAMQ9543 |
| Probe Severity :- 2 |
| Probe Description :- AMQ9543: |
| FDCSequenceNumber :- 0 |
| |
+-----------------------------------------------------------------------------+
|
Now could you give me some hints and what should I do for resuming my qmgr? |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu Apr 02, 2009 6:48 am Post subject: Re: Errors occurs after restarting qmgr |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| kun.leeing wrote: |
| For restarting qmgr, I run 'kill -9 some processes that can not be cleaned (amqiclen)' |
Why did you have to do this? What was preventing the queue manager coming down on an endmqm -i?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Thu Apr 02, 2009 6:58 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
Raise a PMR? QM looks damaged.
| Quote: |
| rrcE_STATUS_TABLE_CORRUPT |
Failing that, you could try and recreate the objects that aint working like what they oughta
Last edited by WMBDEV1 on Thu Apr 02, 2009 7:04 am; edited 1 time in total |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Apr 02, 2009 7:03 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| WMBDEV1 wrote: |
| Raise a PMR? QM looks damaged. |
Response to the PMR is "You issued a kill -9 without due care and attention. If you stop the queue manager by hitting it with a hammer, you're likely to damage it". Hence my question about why it had to be stopped that way. I suppose I should also have asked, rather than assumed, if the kill commands performed in the sequence given in the documentation for a manual shutdown.
Though I do agree with the diagnosis. Given the queue manager's apparent name, I suspect the solution will be to delete and redefine it.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Thu Apr 02, 2009 4:28 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
Make sure all IPC resources owned by mqm are removed before starting the queue manager, otherwise it may think that some processes are still running that you killed using -9.
This includes shared memory, semaphores and message queues (not to be confused with MQ's queues). amqiclen should be able to do the job, or failing that, the ipcs and ipcrm commands.
_________________
Glenn |
|
| Back to top |
|
|
| kun.leeing |
| Posted: Thu Apr 02, 2009 4:59 pm Post subject: |
|
|
Disciple
Joined: 27 Sep 2008
Posts: 171
|
Thanks for reponses above.
I used 'endmqm -i' firstly, it hold long time and no response. Finally give me a list of processes still running.
Then, I used 'amqiclen' to clean and run 'endmqm' again, same processes existed.
And then I used ipcs and ipcrm to free semaphores and shared memory. After this, mostly be deleted but some still there. Trying 'endmqm -i' again, there're several processes yet.
So I can not help but 'kill -9' finally.  |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Apr 02, 2009 10:56 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
So you used ipcs, ipcrm & amqiclen and then started using kill -9?
No wonder the queue manager thinks it's been mugged. 
As I've said before, you shouldn't remove semaphore unless you're certain of what you're doing and never while there's a running process using them! Bad things will always result (not just in a WMQ context).
Make sure all the running process are stopped, using kill -9 if necessary, as documented and when the dust settles that is the time to clean up still held resources.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Apr 03, 2009 3:40 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
kun.leeing,
There's a section in the MQ System Admin Guide that tells you how to forcefully put a QM out of its misery if endmqm doesn't work.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| kun.leeing |
| Posted: Mon Apr 06, 2009 12:31 am Post subject: |
|
|
Disciple
Joined: 27 Sep 2008
Posts: 171
|
Thanks,guys.
This time I have no choice but to recreate objects.
Hope this problem will not befall me again and maybe next time I'll use the correct method. |
|
| Back to top |
|
|
|
|
