| Author |
Message
|
| sheepdog316 |
 Posted: Mon Aug 09, 2004 6:33 am Post subject: monitor to detect a 'hung queue manager' |
 |
|
Newbie
Joined: 20 Jul 2004
Posts: 5
Location: St. Louis
|
Does anybody know how to monitor for a 'hung' queue manager. I know checking for the processes won't work, and I don't really want to just connect to a queue and do something trivial for security reasons.
I was trying to write a korn shell script to do the test:
exec `echo "" | runmqsc ${QMGR}` &
sleep 30
HUNGPID=`ps -ef | grep runmqsc | grep $! | awk -F" " '{print $2}'`
#echo $HUNGPID
if [ $HUNGPID ]
then
kill -9 ${HUNBPID}
echo 'exit 2'
fi
When I run this script from a script, I get:
qmgr_up2.ksh[27]: 5724-B41: not found
running each line individually from the command line operates as I expect it to. |
|
| Back to top |
|
 |
| jsware |
| Posted: Tue Aug 10, 2004 11:57 pm Post subject: |
|
|
Chevalier
Joined: 17 May 2001
Posts: 455
|
You could try:
Check the command server process is running. If not start it.
echo "PING QMGR" | runmqsc $QMGR_NAME
rc=$?
If the rc is not 0 then the ping qmgr is not responding it means the queue manager is in a bad way. It might take 30 seconds for the PING QMGR to fail.
Stop the command server.
You only need to start/stop the command server if you don't like it running all the time.
_________________
Regards
John
The pain of low quaility far outlasts the joy of low price. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Aug 11, 2004 10:40 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
As you are not setting the parms for remote admin I assume you are running on the same box.
There really should be no need for having the command server running if your qmgr is local to execute runmqsc commands.
Now for remote admin or programmed admin (pcf commands put to the SYSTEM.ADMIN.COMMAND.QUEUE) this is a different matter.
Enjoy |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Aug 11, 2004 1:37 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
You also might want to just use the dspmq command to get a status of all the queue managers on a system. If they are happy, you will get RUNNING for their status.
Of course, that is no proof that the listener is up, or that the channel initiator is up, or that the trigger monitors are up, etc.......
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| sheepdog316 |
| Posted: Thu Aug 12, 2004 7:45 am Post subject: |
|
|
Newbie
Joined: 20 Jul 2004
Posts: 5
Location: St. Louis
|
These are all good ideas. However, I don't think they will find the exact problem I want to detect.
I want to detect when mqseries mqseries startup thinks it needs to recover, the recovery fails, and in continues on indefinately.
If you'd like to see what causes this, here is what we did: 1) don't have large file support turned on for your unix queue manager. 2) put more than 2 gigs of data in a queue. 3) recycle your queue manager.
MQSeries is smart enough to realize there is a problem with the queue, but not not smart enough to realize what it is.
While it was continually trying to recover the QMGR, all mqseries commands (runmqsc, dspxxxx, etc) hung. That is why I came up with the idea of shadow process, and if that shadow is still around in x number of seconds, then I will have a flag for a 'hung' qmgr.
Also, does anybody know why I can run a file that contains a runmqsc command in it in the background from an interactive session, but I can't call it from a script? |
|
| Back to top |
|
|
| vennela |
| Posted: Thu Aug 12, 2004 12:06 pm Post subject: |
|
|
Jedi Knight
Joined: 11 Aug 2002
Posts: 4055
Location: Hyderabad, India
|
| Quote: |
| but I can't call it from a script? |
You sure can. |
|
| Back to top |
|
|
| sheepdog316 |
| Posted: Thu Aug 12, 2004 12:38 pm Post subject: |
|
|
Newbie
Joined: 20 Jul 2004
Posts: 5
Location: St. Louis
|
Let me repeat:
| Quote: |
When I run this script from a script, I get:
qmgr_up2.ksh[27]: 5724-B41: not found |
is what I get when I run runmqsc in the background from a script.
Also let me repeat that the whole reason for running in the background is (not being a system programmer) I don't know of a way to 'Time Out' the main script if I don't get a return from runmqsc. If there is an alternate way that one script can be used to do the monitoring, please let me know. |
|
| Back to top |
|
|
| vennela |
| Posted: Thu Aug 12, 2004 1:13 pm Post subject: |
|
|
Jedi Knight
Joined: 11 Aug 2002
Posts: 4055
Location: Hyderabad, India
|
| What does your line look line in your shell script? |
|
| Back to top |
|
|
| sheepdog316 |
| Posted: Fri Aug 13, 2004 5:20 am Post subject: |
|
|
Newbie
Joined: 20 Jul 2004
Posts: 5
Location: St. Louis
|
command from my monitor script is:
pingT &
where contents of file pingT is simply:
echo 'end' | runmqsc QMGR_NAME
(or echo 'ping QMGR' | runmqsc QMGR_NAME)
I was also previously trying to do:
`echo 'end' | runmqsc QMGR_NAME` &
but abandoned that when I found documentation that says commands with pipes in them can't be run in the background, unless they are in a file. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Aug 13, 2004 10:38 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
why don't you try and put your command into a file.
On top of it I would like instant info ex:
file info:
| Code: |
dis ql(*) curdepth
end |
now the command to run:
| Code: |
| runmqsc QMGR_NAME < info | grep CURD | grep -v "(0)" > stat.txt |
in file stat.txt you now have an instant snapshot of the queues with their current depth > 0.
Enjoy |
|
| Back to top |
|
|
| sheepdog316 |
| Posted: Fri Aug 13, 2004 12:00 pm Post subject: |
|
|
Newbie
Joined: 20 Jul 2004
Posts: 5
Location: St. Louis
|
Thank you, that worked.
Summary:
Main script is
| Code: |
#!/bin/ksh
export QMGR="$1"
export DIR="/home/b1064192/mq_scripts"
#####################################
# test if hanging (background runmqsc won't complete) #
#####################################
.qmgr_up.ksh &
sleep 10
#find pid of background command called by this shell
HUNGPID=`ps -ef | grep runmqsc | grep $! | awk -F" " '{print $2}'`
#if background process is still operating, we have a 'hung' qmgr
if [ $HUNGPID ]
then
kill -9 ${HUNGPID}
RC=2
exit ${RC}
fi
#########################################
# not hung, test if accepting commands #
#########################################
.qmgr_up.ksh
# 0 = ran successfully
# 20 = QMGR NOT AVAILABLE / invalid qmgr
#130 = Killed (Ctrl-C)
#137 = Killed (kill -9)
RC=$?
exit ${RC}
|
support file .qmgr_up.ksh is:
| Code: |
runmqsc ${QMGR}< .qmgr_up >> /dev/null
#runmqsc ${QMGR}< .qmgr_up >> ${QMGR}_up.log
|
and file .qmgr_up is:
|
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Aug 13, 2004 5:02 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
but don't you have to END the runmqsc command? If I runmqsc, then issue the ping command, or anything else, it just sits there after returning my output until I enter END. Looking at this code, I am guessing it will always still be running, waiting for your END command, even after sleeping 10 seconds?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Aug 14, 2004 5:28 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Peter
The End command is not mandatory when you redirect a file as input.
The EOF will be considered as end command for runmqsc.
I usually still add it but then that's just me.
Enjoy |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Sat Aug 14, 2004 6:16 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
duh, thats right. The MS03 scripts never have an END at the end of them, and I always use those without worrying about runmqsc ending.
thanks
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| btjo |
| Posted: Thu Sep 09, 2004 12:01 pm Post subject: |
|
|
Novice
Joined: 07 Jul 2004
Posts: 19
|
| The MS03 scripts pipe the output to /dev/null which is where it has its EOF. |
|
| Back to top |
|
|
|
|
