| Author |
Message
|
| cr |
 Posted: Tue Jan 16, 2007 12:38 pm Post subject: best way to delete messages where ReplyToQ no longer exists |
 |
|
Novice
Joined: 04 Oct 2006
Posts: 12
|
Hi
An application which puts messages onto a queue suddenly dies. Its dynamic reply-to queue gets cleaned up by MQ. However, all the messages the application has put on a request queue do not.
To clean up we currently get each message consumer to check that a message's ReplyToQ still exists and, if not, we do a destructive get on that message to get rid of it.
Surely this can't be the best method as it involves transmitting all the useless data from the queue manager to the consumer just to have it thrown away!
Is there a lightweight way of clearing the queue? I can't see anything so far but I'm sure I must have missed it. Can't use message expiry because there is no useful value in this instance.
Thanks for any advice/pointers!
cr
PS On a related note, I need to find a way to cancel a set of messages put on a queue by a unit of work, but after that work has been committed. i.e. the user wants to Cancel a batch, rather than the above case where the application has just disappeared. |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Tue Jan 16, 2007 1:04 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Request/Reply server programs should always do destructive gets on the request - possibly in syncpoint. In fact, almost all MQ applications should always do destructive gets, possibly in syncpoint.
Use syncpoint for retries, not Browse. Only use Browse for monitoring applications.
If the ReplyTo Queue doesn't exist, then the PUT will fail, and the server program can assume that the request is invalid and discard it.
Or the Put will succeed but the reply message will go to a DLQ, where the administrator can discard it.
In terms of canceling a batch... once messages are committed, only a destructive GET can remove them from the destination queue. And there's no guarantee that the destination queue is local to the sender and thus no guarantee that a Destructive GET can be performed. Nor is there any guarantee that the committed messages aren't ALREADY BEING PROCESSED by the time the user wants to cancel.
If your users don't realize they've made a mistake until after they've comitted their messages... then your users need to execute a business level transaction cancel or compensation.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| cr |
| Posted: Tue Jan 16, 2007 1:29 pm Post subject: |
|
|
Novice
Joined: 04 Oct 2006
Posts: 12
|
Thanks for quick response!
Thanks for all the general tips. I take your point about trying to cancel a batch when it's already started to be processed!
OK, so I think what you're saying is that our current method of checking for dead-client messages (i.e. destructive get, check reply-to still exists, if not don't process the message contents) is the best we can do.
Was just trying to save the cost in time and network traffic of e.g. 1000 GETters one-by-one getting some of 10000 messages with potentially large data (500Kb) in some, only to throw them away! This process could quite easily annoy other users of the 'grid' (who won't be hassling the true sources of the problem i.e. the author of the crashing application or the user who now wants to cancel a big batch)!
Cheers
cr |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Jan 16, 2007 1:44 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
No.
I'm saying you shouldn't need to specifically do a destructive get, nor should you check to see if the ReplyTo queue exists.
You should always do a destructive Get on your server application.
Your server application should check if the PUT to the ReplyTo succeeds.
Your server application should never check if the replyTo queue exists - in the case of dynamic queues on remote queue managers it's not actually POSSIBLE to check if the remote queue exists - without jumping through a lot of hoops (like creating a client connection to the remote qmgr or sending pcf messages to the remote qmgr).
If you're concerned about the load on your server application, then you should scale your server application to meet the load. You shouldn't try to hack the standard design patterns for request/reply in order to provide performance optimizations for what should be a pathological case (a crashed requster).
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| cr |
| Posted: Tue Jan 16, 2007 2:15 pm Post subject: |
|
|
Novice
Joined: 04 Oct 2006
Posts: 12
|
Totally agree about hacking about for the pathological case.
I'm a bit confused about why a server should not check if the ReplyTo queue exists for a particular message - is this only because of the dynamic-queue-on-a-remote-queue problem (which is a pretty big problem!)? Or would it be bad form anyway?
If the processing of the message data takes one hour, surely it would be sensible to do a quick check first rather than tieing up a machine for that amount of time, only to have the reply put fail or DLQd?
Thanks for your help,
cr. |
|
| Back to top |
|
|
| KevinF23492 |
| Posted: Tue Jan 16, 2007 2:28 pm Post subject: |
|
|
Novice
Joined: 26 Dec 2006
Posts: 22
|
| cr wrote: |
I'm a bit confused about why a server should not check if the ReplyTo queue exists for a particular message - is this only because of the dynamic-queue-on-a-remote-queue problem (which is a pretty big problem!)? Or would it be bad form anyway?
|
Because if it doesn't exist the Queue manager will throw it into the DLQ for you and you can clean them up from there. There should be no need for the application to go through all that checking ahead of time.  |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Jan 16, 2007 2:41 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
If the processing of the request data takes an hour, then it's gonna take an hour if the requester is still there, and so the system should be scaled and performance tuned such that this won't affect other users.
If the processing of the request takes an hour, it's not clear that the requestor can't have restarted properly and recreated the dynamic queue by the time the request is ready to be sent... so it may be better to design the requesting applications to be able to do that than to optimize the sender side...
Or the requesters could not use dynamic queues...
Another possible option is to code the requesters to send a "panic" message - at a higher priority than the normal requests - when they crash. This would include some useful information about what messages should be discarded. The server application would then receive these messages before any pending requests (same problem as the cancel request, though, that some requests are already in process), and could use the information in the panic message to skip processing certain requests.
In all of these cases, the logical separation between the sender and receivers is preserved.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
|
|
