| Author |
Message
|
| gbaddeley |
 Posted: Thu Jul 09, 2015 5:23 pm Post subject: How to delete a local queue that has opprocs from MCA |
 |
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
As part of app decommissioning tasks, I have a requirement to remove messages and delete a local queue. It currently has IPPROCS 0 and OPPROCS 1 and no uncommitted messages.
I can't use CLEAR QLOCAL because the object is in use (this is a well known fact). So I can use MQ Explorer (for example) to destructively MQGET the messages.
The next challenge is get rid of OPPROCS 1 so that the queue can be deleted. This is straight forward if it is an app program that has the queue open. Stopping the app or put disabling the queue are the usual methods. If the app is using a Client channel it can be feasible to stop the channel or stop the connection, though you need to be careful not to affect other users of the SVRCONN channel.
In my case, the OPPROC is a MCA thread in amqrmppa. It it is not possible to stop the channel or bounce the queue manager because it is a cluster receiver that is used by lots of production apps and has lots of queues open and lots of message traffic. It never goes into INACTIVE state.
Is there a method to force the MCA to close the queue immediately, without affecting the operation of the running channel?
Does the MCA have a timeout on holding the queue open?
_________________
Glenn |
|
| Back to top |
|
 |
| PeterPotkay |
| Posted: Thu Jul 09, 2015 5:49 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
There is a post somewhere here from me on the "timeout"...I think its maybe related to the Heartbeat Interval or something like that which tells the incoming MCA channel how long to keep the handle open even if no new messages. This was a long, long time ago, so that may no longer be true, if ever.
If this was non prod, I would be inclined to experiment a bit. Try Put Inhibiting the queue....maybe even have to then send a message down that channel to that Put Inhibited queue. Perhaps that would get that channel to realize its relationship to that queue is over, time to let it go, time to move on, there are plenty of other queues in the sea......
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| PaulClarke |
| Posted: Thu Jul 09, 2015 6:57 pm Post subject: |
|
|
Grand Master
Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand
|
This is testing my memory a bit Peter is correct that a channel will release all queues if it receives a Heartbeat. This means that the channel must not have moved any messages for an entire heartbeat interval so that may not happen in a production channel. So, the queue is also closed if the queue is unused for some time period. I think it's about 15 minutes but I don't have the code to check. The last reason is that the cache itself becomes full and the queue is pushed out of the cache for being the oldest used. The cache size is normally 200 so you could just put to 200 'other' queues. 
However, I can't remember a command or method to make the channel let go of one queue. The main thing is time. Just wait half an hour and the channel should have closed the queue providing no one else is putting to the queue.
Cheers,
Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jul 10, 2015 5:42 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| You could also try the expendience of put disabling the queue... |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Jul 10, 2015 8:48 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Worst case scenario
Find the conn holding this queue and stop the connection.
If the channel gets interrupted, it should reestablish itself right away.
If no more messages are headed for that queue you should be good.
Best case scenario.
Have you checked and removed the queue from the cluster?
You might have to use alter queue ... FORCE
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Sun Jul 12, 2015 5:05 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
Thanks for the ideas guys.
Heartbeat on a busy channel is unlikely to happen. Forcing the channel to open 200 new queues to push to queue in question out of the cache is not very practical.
The queue is not in a cluster, but the messages are coming in over a cluster receiver channel. This is because the originating qmgr does not have sdr/rcvr channels to this qmgr, but they are both in a MQ cluster, so MQ routes msgs via the cluster chls.
If I put(disable) the queue, the originating app does not fail when it puts msgs. The msgs go the the dead letter queue on the dest qmgr, and there are appropriate "AMQ9544: Messages not put to destination queue." in the error logs for both qmgrs. The good news is that the OPPROCS goes to 0 on the dest queue, allowing me to delete the queue. The bad news is that there could be ongoing stream of msgs going to the DLQ that will need to be cleaned up (or handled by runmqdlh rules). The only recourse is the locate the producer app(s) and ensure they no longer use the queue.
_________________
Glenn
Last edited by gbaddeley on Sun Jul 12, 2015 6:27 pm; edited 1 time in total |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Sun Jul 12, 2015 5:22 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
I also tried another scenario, where the queue is in a cluster. I started an app (amqsput) to successfully put messages to the queue via another qmgr that is in the cluster.
If I put(disable) the queue, the app fails to put msgs with ReasonCode 2268 MQRC_CLUSTER_PUT_INHIBITED. On the dest qmgr, the cluster receiver MCA keeps the queue open (OPPROCS=1). I can dynamically put(enable), put(disable) and amqsput dutifully succeeds or fails to put each msg.
This is clearly different behavior compared to putting messages via the non-clustered queue delivery method. But, it is as expected.
It appears that I can getting rid of the opprocs using put(disabled) and delete it, if the queue is not clustered, and its OK for the messages to go to DLQ and the producer app is unware that its dest queue is no longer a valid destination.
If the queue is clustered, the cluster receiver MCA is not going to let go of the OPPROC if the queue is altered to put(disabled).
_________________
Glenn |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Jul 13, 2015 5:47 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| gbaddeley wrote: |
I also tried another scenario, where the queue is in a cluster. I started an app (amqsput) to successfully put messages to the queue via another qmgr that is in the cluster.
If I put(disable) the queue, the app fails to put msgs with ReasonCode 2268 MQRC_CLUSTER_PUT_INHIBITED. On the dest qmgr, the cluster receiver MCA keeps the queue open (OPPROCS=1). I can dynamically put(enable), put(disable) and amqsput dutifully succeeds or fails to put each msg.
This is clearly different behavior compared to putting messages via the non-clustered queue delivery method. But, it is as expected.
It appears that I can getting rid of the opprocs using put(disabled) and delete it, if the queue is not clustered, and its OK for the messages to go to DLQ and the producer app is unware that its dest queue is no longer a valid destination.
If the queue is clustered, the cluster receiver MCA is not going to let go of the OPPROC if the queue is altered to put(disabled). |
If you are going to delete the queue anyway, first step is to uncluster it. At that point the CLUSRCVR MCA may release it. If not, you move to the put inhibit 'trick'.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Tue Jul 14, 2015 5:24 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| PeterPotkay wrote: |
| If you are going to delete the queue anyway, first step is to uncluster it. At that point the CLUSRCVR MCA may release it. If not, you move to the put inhibit 'trick'. |
Deleting a clustered queue is OK. It is implicitly removed from the cluster.
I tested amqsput to a clustered local queue on another qmgr in the cluster.
Putting a msg results in OPPROCS 1 (amqrmppa) on the queue.
I then altered the queue to remove it from the cluster.
Putting another msg results in 2189 MQRC_CLUSTER_RESOLUTION_ERROR.
There is still an OPPROC on the queue. (amqrmppa MCA for the cluster receiver channel did not close the queue, even though it is no longer in the cluster. This could be justified because the queue may be a valid destination for non-clustered msg transmission that may be flowing over the cluster channel)
Restarting amqsput and putting a message results in 2085 MQRC_UNKNOWN_OBJECT_NAME. (as expected)
This doesn't really help the situation, I still can't delete the queue.
_________________
Glenn |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jul 15, 2015 1:02 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Try put inhibiting it after you unclustered it.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Thu Jul 23, 2015 5:04 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
I'm considering an RFE idea that will allow a specified app handle to be forced closed, and return a ReasonCode to the app. Eg. Immediately on waiting MQGET or on next MQPUT/MQGET/etc or on MQCB thread.
It would need to support handles that are held by MCAs.
Forced close of handles held by other MQ internal processes may sometimes be useful, but could compromise the integrity of MQ.
Any comments before I proceed?
_________________
Glenn |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Sun Aug 09, 2015 4:36 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
|
| Back to top |
|
|
|
|
