| Author |
Message
|
| sfari |
 Posted: Tue Mar 25, 2008 5:23 am Post subject: runmqlsr does not end with endmqm |
 |
|
Centurion
Joined: 15 Apr 2003
Posts: 144
|
Hello,
We experienced problems that runmqlsr seems not always to end properly when we use endmqm. On one of our queue managers I see now the following situation (see the old processes from Mar 21), even though we stop the queue manager every evening at 20:00. endmqm was executing without problems and the queue manager is now running normally.
We are using 6.0.1.1 on Solaris 8.
| Code: |
ps -ef | grep mqm
...
mqm 14789 22656 0 10:54:57 ? 0:08 amqzlaa0 -mC004502A.SUN.P -fip16
mqm 2637 2634 0 Mar 21 ? 0:56 /opt/mqm/bin/runmqlsr -r -m C004502A.SUN.P -t TCP -p 1414
mqm 22659 22656 0 20:01:50 ? 0:00 amqzmur0 -m C004502A.SUN.P
mqm 22662 22656 0 20:01:51 ? 0:14 amqzlaa0 -mC004502A.SUN.P -fip0
mqm 3046 22663 0 09:15:45 ? 0:01 /opt/mqm/bin/runmqchl -c C004502A.CSQPH1 -m C004502A.SUN.P
mqm 26407 2637 0 20:30:01 ? 0:35 /opt/mqm/bin/amqrmppa -m C004502A.SUN.P
mqm 22697 2637 0 20:02:01 ? 0:49 /opt/mqm/bin/amqrmppa -m C004502A.SUN.P
mqm 22658 22656 0 20:01:50 ? 0:37 amqzmuc0 -m C004502A.SUN.P
mqm 22668 1 0 20:01:51 ? 0:00 runmqtrm -m C004502A.SUN.P
mqm 26532 22656 0 20:30:04 ? 0:05 amqzlaa0 -mC004502A.SUN.P -fip5
mqm 18252 2637 0 22:57:10 ? 1:01 /opt/mqm/bin/amqrmppa -m C004502A.SUN.P
mqm 22693 2637 0 20:01:56 ? 0:36 /opt/mqm/bin/amqrmppa -m C004502A.SUN.P
mqm 22661 22656 0 20:01:50 ? 0:00 /opt/mqm/bin/amqzdmaa -m C004502A.SUN.P
mqm 22656 1 0 20:01:50 ? 0:02 amqzxma0 -m C004502A.SUN.P
mqm 2634 1 0 Mar 21 ? 0:00 /opt/mqm/bin/amqzmgr0 -m C004502A.SUN.P
mqm 10907 22656 0 12:02:47 ? 0:08 amqzlaa0 -mC004502A.SUN.P -fip21
...
|
I don't understand this situation since we have included the listener as object using runmqsc.
| Code: |
dis LISTENER(TCP.1414)
2 : dis LISTENER(TCP.1414)
AMQ8630: Display listener information details.
LISTENER(TCP.1414) CONTROL(QMGR)
TRPTYPE(TCP) PORT(1414)
IPADDR( ) BACKLOG(0)
DESCR(TCP listener on port 1414) ALTDATE(2006-09-17)
ALTTIME(15.43.25)
|
It's not the first time we have such a situation. Has somebody experienced a similar behaviour? Any idea? |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Tue Mar 25, 2008 5:26 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Are you sure that the listener was started by the queue manager?
Are you sure that the listener object in the queue manager is in a started state?
Are there absolutely no errors at all in the AMQERR logs for the queue manager itself?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| sfari |
| Posted: Tue Mar 25, 2008 5:40 am Post subject: |
|
|
Centurion
Joined: 15 Apr 2003
Posts: 144
|
Yes the listener has been started by the QM (see control flag).
Here the absolutely strange state of the listener:
| Code: |
dis lsstatus(*) all
3 : dis lsstatus(*) all
AMQ8631: Display listener status details.
LISTENER(TCP.1414) STATUS(STOPPING)
PID(2637) STARTDA(2008-03-21)
STARTTI(20.01.56) DESCR(TCP listener on port 1414)
TRPTYPE(TCP) CONTROL(QMGR)
IPADDR(*) PORT(1414)
BACKLOG(100)
|
In the AMQERR logs I can't find anything. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Mar 25, 2008 5:45 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
That says that the listener object is not the object that started that instance of runmqlsr.
There should be errors in the AMQERR files for the queue manager.
Make sure, when the queue manager is next stopped, that there are no existing runmqlsr processes. If there are, kill them before you start the queue manager again.
Confirm that you don't have any scripts that will start runmqlsr outside qmgr control - particularly in whatever scripts you have for stopping and starting the queue manager.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Mar 25, 2008 5:45 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| sfari wrote: |
| Yes the listener has been started by the QM (see control flag). |
Yes, but is the listener that's actually running the one started by the QM, or one that's been started from a command prompt by a cron job (or similar) issuing a runmqslsr command?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| sfari |
| Posted: Tue Mar 25, 2008 8:02 am Post subject: |
|
|
Centurion
Joined: 15 Apr 2003
Posts: 144
|
Thanks, for your answers. The Listener which is running is the one you see here in stopped state. You can see that in the STARTTI and STARTDA attribute.
I checked the start stop script again and I can assure, that we did not start anything using the runmqlsr command.
This is the second time I have seen this behaviour on our productive machine. If there would be a problem with our start stop logic we would see it always. I solved the problem last time exactly how Jeff has described. I rather would like that such things would not happen.
I have found a troubleshooting note, and as far as I understand they suggest exactly the same troubleshooting. But they emphasize that if the control is on QMGR such a workaround should not be needed:
http://www-1.ibm.com/support/docview.wss?uid=swg21259556 |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Mar 25, 2008 8:16 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Right, so the problem seems to be that the listener isn't stopping under qmgr control.
But we don't know that from the information you've presented.
How are you stopping the queue manager, by the way? with plain "endmqm"? Or with options on endmqm?
Again, you should verify that the listener process does not stop at the next shutdown. Then you should look for errors.
Then you should end the listener manually. And then start the queue manager, and look for errors.
And then monitor the listener status to see when/if it goes into stopping state.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| sfari |
| Posted: Wed Mar 26, 2008 1:09 am Post subject: |
|
|
Centurion
Joined: 15 Apr 2003
Posts: 144
|
I just let the automated stop-backup-start process running yesterday 25. March. The listener has been stopped this time.
These are logs from the 24. March when the qm tried to start the listener but it couldn't.
| Code: |
-------------------------------------------------------------------------------
03/24/08 20:01:51 - Process(2634.1) User(root) Program(amqzmgr0)
AMQ5026: The Listener 'TCP.1414' has started. ProcessId(22664).
EXPLANATION:
The Listener process has started.
ACTION:
None.
-------------------------------------------------------------------------------
...
-------------------------------------------------------------------------------
03/24/08 20:01:51 - Process(22664.1) User(root) Program(runmqlsr_nd)
AMQ9255: Listener already running.
EXPLANATION:
The request to start the WebSphere MQ listener failed because there is already
a listener running against the specified network resources.
ACTION:
None.
----- amqcclxa.c : 594 --------------------------------------------------------
03/24/08 20:01:51 - Process(2634.1) User(root) Program(amqzmgr0)
AMQ5027: The Listener 'TCP.1414' has ended. ProcessId(22664).
EXPLANATION:
The Listener process has ended.
ACTION:
None.
-------------------------------------------------------------------------------
|
This is the log from yesterday when the long running listener has stopped and new one started.
| Code: |
----- amqrmrsa.c : 468 --------------------------------------------------------
03/25/08 20:00:45 - Process(2634.1) User(root) Program(amqzmgr0)
AMQ5027: The Listener 'TCP.1414' has ended. ProcessId(2637).
EXPLANATION:
The Listener process has ended.
ACTION:
None.
-------------------------------------------------------------------------------
...
-------------------------------------------------------------------------------
03/25/08 20:02:18 - Process(19875.1) User(root) Program(amqzmgr0)
AMQ5026: The Listener 'TCP.1414' has started. ProcessId(19878).
EXPLANATION:
The Listener process has started.
ACTION:
None.
-------------------------------------------------------------------------------
|
We are stopping with endmqm -i.[/code] |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Mar 26, 2008 1:36 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Firstly (and unrelatedly) are you sure these processes should be running as root rather than mqm?
Secondly, it seems clear from the log that the listener didn't start because there was another one running. Hence the listener wasn't under queue manager control & doesn't stop when the queue manager does. But when you cancel the errant listener the queue manager can start one (and I bet stop one) without issue. So there's your problem.
Thirdly, why endmqm -i? Do you have badly coded applications that don't respond to a quiesce?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Mar 26, 2008 1:36 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Firstly (and unrelatedly) are you sure these processes should be running as root rather than mqm?
Secondly, it seems clear from the log that the listener didn't start because there was another one running. Hence the listener wasn't under queue manager control & doesn't stop when the queue manager does. But when you cancel the errant listener the queue manager can start one (and I bet stop one) without issue. So there's your problem.
Thirdly, why endmqm -i? Do you have badly coded applications that don't respond to a quiesce?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| sfari |
| Posted: Wed Mar 26, 2008 5:09 am Post subject: |
|
|
Centurion
Joined: 15 Apr 2003
Posts: 144
|
Hi Vitor, what you are explaining under secondly is completely true. But looking at the whole thread you would have seen that the listener is under qmgr congrol. This also answers your question under firstly the qmgr itself had started this processes under root. But it's a good point I haven't recognized yet that the process is running under root. Maybe that's the problem?
I know endmqm -i is may a little too hard, but it should not cause this kind of problems. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Mar 26, 2008 5:22 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| sfari wrote: |
| Hi Vitor, what you are explaining under secondly is completely true. But looking at the whole thread you would have seen that the listener is under qmgr congrol. |
A listener is under queue manager control - I have doubts it's the listener. I think you have 2, one as a queue manager object, one a runmqlsr command being started. I've not seen anywhere in this post (which I have read from start to finish) where you've proved the queue manager object is the only listener process. Indeed, the fact that the queue manager listener can't start because something else is holding the port is evidence of a second listener.
As always, I stand to be corrected on this.
| sfari wrote: |
| This also answers your question under firstly the qmgr itself had started this processes under root. But it's a good point I haven't recognized yet that the process is running under root. Maybe that's the problem? |
It shouldn't cause this effect, but it's not a good idea for a number of reasons. Very few processes should run as root and WMQ is not one of them, for reasons a search will turn up.
| sfari wrote: |
| I know endmqm -i is may a little too hard, but it should not cause this kind of problems. |
No, it shouldn't.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| LouML |
| Posted: Wed Mar 26, 2008 5:36 am Post subject: |
|
|
Partisan
Joined: 10 Nov 2005
Posts: 305
Location: Jersey City, NJ / Bethpage, NY
|
| sfari wrote: |
Hi Vitor, what you are explaining under secondly is completely true. But looking at the whole thread you would have seen that the listener is under qmgr congrol. This also answers your question under firstly the qmgr itself had started this processes under root. But it's a good point I haven't recognized yet that the process is running under root. Maybe that's the problem?
I know endmqm -i is may a little too hard, but it should not cause this kind of problems. |
| jefflowrey wrote: |
Right, so the problem seems to be that the listener isn't stopping under qmgr control.
But we don't know that from the information you've presented.
How are you stopping the queue manager, by the way? with plain "endmqm"? Or with options on endmqm?
Again, you should verify that the listener process does not stop at the next shutdown. Then you should look for errors.
Then you should end the listener manually. And then start the queue manager, and look for errors.
And then monitor the listener status to see when/if it goes into stopping state. |
As Jeff stated above, you should make sure that once the Queue Manager has stopped, there are no 'mqm' processes running at all. Particularly runmqlsr. If after the Queue Manager has stopped, runmqlsr is still running, kill it manually.
Then, when the Queue Manager starts, check that the listener has started properly. Check the log and do the ps -ef | grep mqm again.
_________________
Yeah, well, you know, that's just, like, your opinion, man. - The Dude |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Mar 26, 2008 3:17 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| louML wrote: |
| If after the Queue Manager has stopped, runmqlsr is still running, kill it manually. |
I would not kill it manually but first try to stop it the normal way:
endmqlsr -m QMGR
or endmqlsr -f -m QMGR
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| sfari |
| Posted: Fri Mar 28, 2008 12:26 am Post subject: |
|
|
Centurion
Joined: 15 Apr 2003
Posts: 144
|
Thanks all for your answers, after the mentioned restart it works fine without changing anything on my scripts. The problem did not occure anymore. I have still no explanation where the runmqlsr processes did come from.
As action we are planning to add checks to our start/stop scripts, which ensures that no processes will remain after a stop.
Thanks! |
|
| Back to top |
|
|
|
|
