| Author |
Message
|
| JWJ |
 Posted: Fri Oct 01, 2004 9:28 am Post subject: Hung Qmgr on LINUX for Intel |
 |
|
Novice
Joined: 13 Jun 2002
Posts: 18
Location: Farmers Insurance Group
|
Qmgr hung problem.
WMQ 5.3 on LINUX for Intel.
Several test Qmgrs on same LINUX system.
One of these seems to have become non-responsive. Other Qmgr on same system is OK.
Display QMGR says its running. Display Command sever says it running.
RUNMQSC hangs. Attempt to access Qmgr info from M071 support pack hangs, but Qmgr shows green on M071.
I trying to make this a learning experiance. I have looked at the error log and nothing jumps out there. Can anyone make suggestions for futher DX moves I can make? Before I recycle this Qmgr, I'd like to learn more about diagnosing this type of problem... Thanks very much.... jerry
_________________
JWJ |
|
| Back to top |
|
 |
| csmith28 |
| Posted: Fri Oct 01, 2004 9:48 am Post subject: |
|
|
Grand Master
Joined: 15 Jul 2003
Posts: 1196
Location: Arizona
|
You are sure there are no entries in
/var/mqm/qmgrs/QMGRNAME/errors/AMQERR0*.LOG
or
/var/mqm/errors/AMQERR0*.LOG
or .FDC files in /var/mqm/errors?
Is this a server that you can play around on or is it in use by a customer?
_________________
Yes, I am an agent of Satan but my duties are largely ceremonial. |
|
| Back to top |
|
|
| JWJ |
| Posted: Fri Oct 01, 2004 10:12 am Post subject: |
|
|
Novice
Joined: 13 Jun 2002
Posts: 18
Location: Farmers Insurance Group
|
Thanks very much.... I have some new places to look!
I did review these and saw some errors but it sounded like these were not related to the MQgr going to sleep. They happened the day before the last activity that was recorded in the /var/mqm/qmgrs/QMGRNAME/errors/ AMQ files. which showed a Chnl start up, followed a disconnect due to a time expiration, followed by a normal end. This all happen on 9/29, the last error messages in the other logs were on 9/28...
Thanks again... your suggestion was very helpful in my MQ education!!
_________________
JWJ |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Oct 01, 2004 10:52 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Is your qmgr by any chance quiescing ?
Suppose you qmgr is shutting down while some process still has a handle on it.
As long as the process has not released its handle the qmgr appears to be running but will be non responsive to any app trying to connect (including MO71). If the offending process shuts down or gets killed the qmgr will stop.
If you try to restart the qmgr any offending process still holding a handle on the qmgr will show up in the error message. Before you get on you will have to stop / kill them.
Enjoy  |
|
| Back to top |
|
|
| JWJ |
| Posted: Fri Oct 01, 2004 11:00 am Post subject: |
|
|
Novice
Joined: 13 Jun 2002
Posts: 18
Location: Farmers Insurance Group
|
Thanks for information!!!!
If I use the dspmq -m Qmgrname -s is show --
STATUS(running).
If it were trying to quiesce, and couldn't would it show "running" or something else?
Jerry
_________________
JWJ |
|
| Back to top |
|
|
| JWJ |
| Posted: Fri Oct 01, 2004 12:38 pm Post subject: |
|
|
Novice
Joined: 13 Jun 2002
Posts: 18
Location: Farmers Insurance Group
|
It gets weirder--- I can send data to this Qmgr from a OS390 Qmgr. Also in the same Linux environment we have WAS running with a Message Driven bean that can get the data sent into a local queue pointed to by a remote Queue on OS390.
But I still can't access it thru M071 (Client connection) nor can I get RUNMQSC runnning on the same LINUX system to do anything other than start and then hang up...... Must be related to being Friday!!!!!
_________________
JWJ |
|
| Back to top |
|
|
| csmith28 |
| Posted: Fri Oct 01, 2004 12:44 pm Post subject: |
|
|
Grand Master
Joined: 15 Jul 2003
Posts: 1196
Location: Arizona
|
Is this the only MQManager on the server?
If so can you try stopping it endmqm -i QMGR
the restarting it strmqm QMGR and see how it behaves
Are there any defunct processes owned by mqm?
Are there any filesystems full?
Are there any limits on the system resources the mqm user is allowed to use.
Also if the endmqm -i QMGR fails you can try killing all the processes that belong to the mqm user then remove any lingering shared memory segment/semiphores and start the MQManager again.
_________________
Yes, I am an agent of Satan but my duties are largely ceremonial. |
|
| Back to top |
|
|
| JWJ |
| Posted: Fri Oct 01, 2004 12:52 pm Post subject: |
|
|
Novice
Joined: 13 Jun 2002
Posts: 18
Location: Farmers Insurance Group
|
There are three Qmgs running this LINUX system, one of which is not mine, so I don't know its status. Of the two I run there, one is working normally.
Eventually I will stop this bad boy and restart, but I'm trying to determine why its in this state and what diagnostic procedures I can use to determine what's going on here, so we can deal with this problem, if it happens again.
Thanks Jerry
_________________
JWJ |
|
| Back to top |
|
|
| csmith28 |
| Posted: Fri Oct 01, 2004 1:22 pm Post subject: |
|
|
Grand Master
Joined: 15 Jul 2003
Posts: 1196
Location: Arizona
|
| JWJ wrote: |
There are three Qmgs running this LINUX system, one of which is not mine, so I don't know its status. Of the two I run there, one is working normally.
Eventually I will stop this bad boy and restart, but I'm trying to determine why its in this state and what diagnostic procedures I can use to determine what's going on here, so we can deal with this problem, if it happens again.
Thanks Jerry |
Yea that is why I asked earlier if this was a server you could "play around with". How many processes are running that are owned my mqm
ps -ef | grep mqm | grep -v grep | wc -l
Is there a limit to the number of processes mqm can run on that server?
You may also want to make sure that all the mqseries processes are indeed owned by mqm and not orphaned.
I have had this happen to me on rare occasions but there was usually an .FDC file created that pointed to why.
Also I know there is a way to turn on Tracing for AIX and Solaris but I have never ran MQSeries on a Linux Box. Check the Command reference.
The instructions for running a Trace in AIX and Solaris are:
| Code: |
*************************************************************
Solaris MQTrace
strmqtrc -m <qmgrname> -t all -t detail // to start trace
//when problem occurs //
endmqtrc -m <qmgrname> //to end trace
endmqtrc -a
dspmqtrc /var/mqm/trace/*.TRC // to format traces
*************************************************************
AIX MQTrace
Here is a command to start a trace:
trace -a -j30D,30E -o /path/trace.trc -L 50000000 -T detail
to stop a trace:
trcstop
to format:
trcrpt -t /usr/mqm/lib/amqtrc.fmt trace.trc > report.out |
Once you get the trace running try accessing the MQManager with runmqsc again. Note you may have to send these files to IBM MQSupport after opeing a Problem Ticket with them. They can at times be rather cryptic.
_________________
Yes, I am an agent of Satan but my duties are largely ceremonial. |
|
| Back to top |
|
|
| JWJ |
| Posted: Fri Oct 01, 2004 2:11 pm Post subject: |
|
|
Novice
Joined: 13 Jun 2002
Posts: 18
Location: Farmers Insurance Group
|
Great information!!! I got some things to do that I wasn't aware of.
Thanks very much for sharing your knowledge.
Jerry
_________________
JWJ |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Oct 02, 2004 9:15 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Quote: |
| I got a WAS running and the MDB's are able to put and get to the queues |
This will prevent your qmgr from shutting down. You may have requested a shutdown and this is why runmqsc and all other processes on the box don't work right any more with this qmgr. But as long as the WAS has a hold on the qmgr (MDB accessing the queue even if there is no message (listener process) ...) you will not be able to shut down the qmgr.
Enjoy |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Sun Oct 03, 2004 5:16 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
|
| Back to top |
|
|
| csmith28 |
| Posted: Sun Oct 03, 2004 5:48 am Post subject: |
|
|
Grand Master
Joined: 15 Jul 2003
Posts: 1196
Location: Arizona
|
| JWJ wrote: |
Great information!!! I got some things to do that I wasn't aware of.
Thanks very much for sharing your knowledge.
Jerry |
Correction, syntax for running an MQTrace is in the WMQ Systems Administration Guide not the Command Reference guide as I said previously.
_________________
Yes, I am an agent of Satan but my duties are largely ceremonial. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Oct 03, 2004 11:24 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Peter,
The only clean way I know about shutting down a JMS MessageListener is to add a JMS ExceptionListener and shutdown the MessageListener and MDB if an exception is caught by the exception listener.
To be honest I don't know of many MDB implementations that do this. IBM WAS 5.x doesn't seem to do it as a standard => that's why the WAS still has a connection to the qmgr after qmgr shutdown. The WAS has still the MessageListner on the queue....
Enjoy  |
|
| Back to top |
|
|
| JWJ |
| Posted: Mon Oct 04, 2004 1:56 pm Post subject: |
|
|
Novice
Joined: 13 Jun 2002
Posts: 18
Location: Farmers Insurance Group
|
Thanks very much to all of you. I've passed this info on to the app developer. He'll probably be joining this group...
JJ
_________________
JWJ |
|
| Back to top |
|
|
|
|
