| Author |
Message
|
| simon.starkie |
 Posted: Tue Aug 06, 2002 4:38 am Post subject: Do I really need a Dead Letter Queue? |
 |
|
Disciple
Joined: 24 Mar 2002
Posts: 180
|
Hi,
Message Affinity and the sequence of messages is a requirement for us.
But sometimes we see messages being sent to the DLQ.
Obviously, this complicates our remediation process. For example, we cannot just take the DLQ's and resubmit them because this could result in those messages being processed out of sequence with respect to other messages that successfully processed after the DLQ's were written.
We always use PERSISTENT messages.
The MQ Intercommunication manual on page 5 states:
"Each queue manager should have a dead-letter queue (also known as the undelivered message queue). Messages are put on this queue if they cannot be delivered to their destination."
The MQ Intercommunication manual on page 13 also states:
"The dead-letter queue (or undelivered-message queue) is the queue to which messages are sent if they cannot be routed to their correct destination. Messages are put on this queue when they cannot be put on the destination queue for some reason (for example, because the queue does not exist, or because it is full). Dead-letter queues are also used at the sending end of a channel, for data-conversion errors.
We recommend that you define a dead-letter queue for each queue manager. If you do not, and the MCA is unable to put a message, it is left on the transmission queue and the channel is stopped.
Also, if fast, non-persistent messages (see “Fast, nonpersistent messages” on page 22) cannot be delivered and no DLQ exists on the target system, these messages are discarded.
However, using dead-letter queues can affect the sequence in which messages are delivered, and so you may choose not to use them."
This last sentence is of particular interst to us because of the aforementioned Message Affinity and sequence requirements.
Does anyone have specific experience of running without a DLQ?
Is it safe to run without a DLQ?
Are there any other issues that would result in DLQ's other than the queue full, non-existent queue or data conversions issues noted in the above quote?
I am particularly interested in finding out about the "gotcha's" of running without a DLQ. Does anyone have a cookbook or laundry list?
I am also partularly interested in what needs to be done to preserve the robustness of the MQ layer when a DLQ is not used. What additional monitoring needs to be done for channels, queue depths, etc?
Cheers!
 |
|
| Back to top |
|
 |
| mrlinux |
| Posted: Tue Aug 06, 2002 4:45 am Post subject: |
|
|
Grand Master
Joined: 14 Feb 2002
Posts: 1261
Location: Detroit,MI USA
|
Well the biggest issue is the channel stopping when the message cannot be delievered, beyond that there isnt much more.
_________________
Jeff
IBM Certified Developer MQSeries
IBM Certified Specialist MQSeries
IBM Certified Solutions Expert MQSeries |
|
| Back to top |
|
|
| bduncan |
| Posted: Tue Aug 06, 2002 5:17 am Post subject: |
|
|
Padawan
Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley
|
My experience says that it is a dangerous path to go down, neglecting to have a dead letter queue. Yes, the channels stop, but isn't that bad enough? Especially if you are using clustering, and everything uses the same SYSTEM.CLUSTER.TRANSMIT.QUEUE.
Keep in mind also that when the channel stops and messages begin to pile up on the transmission queue, you can potentially fill it up. Once this happens, all your MQPUTs will simply start failing. Now you are in even deeper trouble.
The point is, you don't just "resubmit" messages from the dead letter queue if your particular architecture requires that messages are processed in sequence, etc. You can write a custom dead letter handler, and add as much processing logic as you wish in order to maintain your particular business requirements...
_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator |
|
| Back to top |
|
|
| nimconsult |
| Posted: Tue Aug 06, 2002 10:33 pm Post subject: |
|
|
Master
Joined: 22 May 2002
Posts: 268
Location: NIMCONSULT - Belgium
|
I agree with Brandon that this should be the last resort solution.
However there are cases where you absolutely need it (a dead-letter queue handler does not help because the whole architecture is asynchronous and you cannot be sure that the handler will resubmit message n before the application processes message n+1).
If you go that way:
- you have to run on a dedicated queue manager, because removing the dead-letter queue is for the entire queue manager and not for a single channel.
- you have to use dedicated channels all the way from the client to server applications. To make sure that stopping the channel only affects this specific application.
Before reaching the last resort solution, consider alternatives:
- Challenge the pure FIFO nature of the application.
- Produce sequence numbers (1, 2, 3...) on the client applications. This gives the server application the opportunity to detect when messages are out of sequence and either raise an alert, or wait until a message with correct sequence reaches the queue (re-submitted from dead-letter queue). You can even use the correlation id to store the sequence number and make sure that the server application never performs a MQGET of a message out of sequence.
_________________
Nicolas Maréchal
Senior Architect - Partner
NIMCONSULT Software Architecture Services (Belgium)
http://www.nimconsult.be |
|
| Back to top |
|
|
| mrlinux |
| Posted: Wed Aug 07, 2002 3:31 am Post subject: |
|
|
Grand Master
Joined: 14 Feb 2002
Posts: 1261
Location: Detroit,MI USA
|
You could also use MQSeries's Message Segmentation feature.
_________________
Jeff
IBM Certified Developer MQSeries
IBM Certified Specialist MQSeries
IBM Certified Solutions Expert MQSeries |
|
| Back to top |
|
|
| bduncan |
| Posted: Wed Aug 07, 2002 9:34 am Post subject: |
|
|
Padawan
Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley
|
Yes, if message sequence is absolutely essential, I agree with Nicolas - your application should stop processing if it can't find message x after processing x-1. In the meantime, message x may have landed on a dead letter queue, and yes, your entire system has come to a stop, but you can use a dead letter handler to retrieve the message, determine if it can be reprocessed, and then do so.
_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator |
|
| Back to top |
|
|
| simon.starkie |
| Posted: Fri Aug 09, 2002 10:39 pm Post subject: Thanks. We will go without a DLQ. |
|
|
Disciple
Joined: 24 Mar 2002
Posts: 180
|
Thanks everyone who participated.
Basically, we have decided to go with no DLQ and monitor the dickens out of the channels and transmit queues.
The deciding factor, for us, is message affinity. We would rather stop the application dead in it's tracks (as is the case when no DLQ is used) rather than risk having one or more messages go to a DLQ which in turn introduces an out of sequence condition (assuming some messages before or after the DLQ event get processed normally).
This is exactly what the documentation states.
We don's use MQ CLustering very much although I fully inderstand the issue is multiple apps backup on the SYSTEM.CLUSTER.TRANSMIT.QUEUE (since they are all in one queue, how do you differentiate which message belongs to a particular app, assuming you are not intimate with the data...). We will continue using DLQ's for these Cluster qmgrs.
Thanks again.
Cheers!  |
|
| Back to top |
|
|
| simon.starkie |
| Posted: Mon Aug 26, 2002 5:19 am Post subject: Non-Persistent messages discarded when no DLQ available... |
|
|
Disciple
Joined: 24 Mar 2002
Posts: 180
|
So far so good.
I ran into an issue when running without a DLQ. Basically, non-persistent messages were discarded if they could not be delivered and there was no DLQ available to the local Queue Manager. (But we use persistent exclusively for all our application components). However.
Research found the following statement on page 71 of the Intercommunication reference manual:
"On a fast channel, nonpersistent messages that cannot be written to a dead-letter queue are lost."
This suggested to me that for nonpersistent messages, the channel speed, fast versus normal, also matters as well. "Fast" is the default, but the above statement implies the "Normal" option may avoid the discard.
I ran a little test and confirmed the above conclusion. If no DLQ was available to the local Queue Manager, undeliverable non-persistent messages were indeed discarded as expected if the Sender and Receiver ends of the chanel specified the default "Fast" in the "Non-Persistent Message Speed" attribute.
But if the Channel was altered such that the "Non-Persistent Message Speed" was set to "Normal" at both ends, then undeliverable non-persistent messages were NOT discarded. Instead they were queued on the Transmit Queue just like persistent messages are.
If I want the performance benefits of nonpersistent messages and also want to run without a DLQ and you also want to avoid losing any of the non-persistent messages that are undeliverable, then setting the channel (both sender and receiver) to "normal" instead of "fast" allows me to doo all of this.
Cheers. |
|
| Back to top |
|
|
| bduncan |
| Posted: Mon Aug 26, 2002 8:09 am Post subject: |
|
|
Padawan
Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley
|
Interesting findings!
Though I would make the case that if you made them non-persistent in the first place, then you can afford to lose them! 
Of course, I wonder what can be done once the messages pile up on the transmission queue. Does a dead letter header get affixed?
_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator |
|
| Back to top |
|
|
| simon.starkie |
| Posted: Mon Aug 26, 2002 8:30 am Post subject: No Dead Head on XmitQ... |
|
|
Disciple
Joined: 24 Mar 2002
Posts: 180
|
Thanks.
I only see an XQH and an MD in the messages that backup on the XmitQ.
I don't see a Dead Head on the XmitQ. I would be surprised to to see one on any queue other that the Dead Queue. Or am I missng something  |
|
| Back to top |
|
|
| simon.starkie |
| Posted: Mon Aug 26, 2002 9:31 am Post subject: ps |
|
|
Disciple
Joined: 24 Mar 2002
Posts: 180
|
When they pile up on the XmitQ, we can remediate the root cause, re-enable GET on the XmitQ and stop/start the paused channel to get things going again.
But if we use FAST and MQ discards them, we can't remediate and we have no choice to re-run the app that puts the messages back on the queue. Since we have no knowledge (bad app design) of which particular message was the one that started the discard process, we have to figure out where to restart the sending app which involves app support folks. A very good case for persistent. But, in this particular case, the messages happen to be 100MB (max) and we didn't want the overhead of logging.
Using "NORMAL" with nonpersistent seems like a winner for this articular case. And the fact that a DLQ may or may not be assigned does not seem to affect this particular configuration.
Do nonpersistent messages go to the DLQ if one is available? They do not seem to ... they either back-up on the XmitQ if the Channel is NORMAL or the are discarded if it is FAST. |
|
| Back to top |
|
|
| mrlinux |
| Posted: Mon Aug 26, 2002 11:14 am Post subject: |
|
|
Grand Master
Joined: 14 Feb 2002
Posts: 1261
Location: Detroit,MI USA
|
With few exceptions, the messages that go on the DLQ are place there by
the rcvr channel not senders.
Exceptions that I know of are:
1) OTMA IMS Bridge.
2) runmqdlq.
_________________
Jeff
IBM Certified Developer MQSeries
IBM Certified Specialist MQSeries
IBM Certified Solutions Expert MQSeries |
|
| Back to top |
|
|
| simon.starkie |
| Posted: Mon Aug 26, 2002 12:01 pm Post subject: Agreed. But our Qmgrs don't have a DLQ at either end. |
|
|
Disciple
Joined: 24 Mar 2002
Posts: 180
|
| Agreed. But our Qmgrs don't have a DLQ at either end. |
|
| Back to top |
|
|
| kanwisch |
| Posted: Mon Sep 22, 2003 10:46 am Post subject: Follow-up |
|
|
Newbie
Joined: 16 Jul 2003
Posts: 2
|
Our organization just had this very same discussion, for the same reasons. simon.starkie, if you're still around, can you give an update on your experience?
I've been lucky enough to have very good designers working with us, and they've come up with a reliable batch/sequence system that will be used to maintain the message order. As with you, the apps are not necessarily designed very well, but with an ERP looming, overhauling interest is very low.
Unfortunately, anything going to the DLQ requires many other people resources to deal with it, and I still have to write a complex DLQ handler.
Thanks for any advice.
_________________
I can't wait until the meek inherit the earth. The stupid currently have it. |
|
| Back to top |
|
|
| simon.starkie |
| Posted: Mon Sep 22, 2003 11:30 am Post subject: |
|
|
Disciple
Joined: 24 Mar 2002
Posts: 180
|
kanwisch, our experiences with running without a DLQ have been positive. We did have to ramp-up the monitoring via Tivoli of various MQ entities, such as the Transmit Queue and User Queue Depths are properly monitored. We also ramped up the channel monitoring to catch when a Channel is STOPPED (a problem) versus INACTIVE (normal).
But the more serious problem, that of preserving message affinity (i.e. not allowing some messages to go to DLQ while later ones process successfully) has been solved. It's better, in our case, to stop everything cold rather than letting some messages flow to a DLQ and having the rest process normally.
A DL Handler to re-inject DLQ's and maintain message affinity would have involved some very complicated business logic and may not have been possible here.
Last edited by simon.starkie on Mon Sep 22, 2003 11:57 am; edited 1 time in total |
|
| Back to top |
|
|
|
|
