Author |
Message
|
Sam Uppu |
Posted: Wed Oct 14, 2009 10:17 am Post subject: How to avoid msgs not landing onto DLQ? |
|
|
 Yatiri
Joined: 11 Nov 2008 Posts: 610
|
Hi Guys,
We have a typical requirement from one of our application.
We are on MQ version 7 on Unix platform.
We have 2 qmgrs lets say QM1 and QM2 on seperate servers which are in a cluster. We have a cluster queue - CLUSQ defined on both QM1 and QM2 with attributes being CLWLUSEQ(ANY), DEFBIND(NOTFIXED), CLWLPRTY(0) on both CLUSQs.
Now a client application on a remote machine is trying to access and put msgs on the CLUSQ on both QM1 and QM2 by connecting to one of the QMgr, say QM1.
The msgs will round robin between QM1 and QM2. If the CLUSQ is full on QM1, the app is getting 2053 and if the CLUSQ on QM2 is full, then the msgs are going to the dead letter queue. I think this is expected as MQ is designed in such a way.
The application doesn't want the msgs goto the dead letter queue for whatever queue it is whether defined locally or remotely. Is it feasible?.
Any thoughts would be appreciated.
Thanks.
Last edited by Sam Uppu on Wed Oct 14, 2009 10:22 am; edited 1 time in total |
|
Back to top |
|
 |
fjb_saper |
Posted: Wed Oct 14, 2009 10:21 am Post subject: |
|
|
 Grand High Poobah
Joined: 18 Nov 2003 Posts: 20756 Location: LI,NY
|
To avoid the 2053 and or messages going to the DLQ you need to have monitoring in place that alerts you to start more instances of the consumer to avoid the consumer input queue getting full...  _________________ MQ & Broker admin |
|
Back to top |
|
 |
Sam Uppu |
Posted: Wed Oct 14, 2009 10:27 am Post subject: |
|
|
 Yatiri
Joined: 11 Nov 2008 Posts: 610
|
This is used by the Batch application. The sending app is faster than consumer app. The sending app will stop sending the batch when it encouters a 2053 error. After some time the sending app will inquire the current depth of the queue and when there is some free space, the sending Batch will again start sending the transactions. The sending Batch program is expecting 2053 for a remote queue(cluster queue - CLUSQ) on QM2.
Thanks for your suggestions/ thoughts. |
|
Back to top |
|
 |
PeterPotkay |
Posted: Wed Oct 14, 2009 10:50 am Post subject: |
|
|
 Poobah
Joined: 15 May 2001 Posts: 7722
|
Sam Uppu wrote: |
After some time the sending app will inquire the current depth of the queue and when there is some free space, the sending Batch will again start sending the transactions. |
Bad design.
Spend the few bucks to get more disk space and make the queues deep enough to hold the batch. Or add a 3rd QM on a 3rd server that has additional desk space. _________________ Peter Potkay
Keep Calm and MQ On |
|
Back to top |
|
 |
Sam Uppu |
Posted: Wed Oct 14, 2009 11:06 am Post subject: |
|
|
 Yatiri
Joined: 11 Nov 2008 Posts: 610
|
Peter,
I appreciate your suggestion and I can convey the same msg to my app team.
Now from the MQ admin perspective can we do something for msgs not going to DLQ or the app gets 2053 when the remote queue(cluster queue on QM2) gets filled?. I know the sending MCA will handover the msg to receiving MCA on QM2. The receiving MCA on QM2 will try to deliver the msg onto CLUSQ on QM2. When the receiving MCA on QM2 finds queue full event, it will route the msg onto DLQ on QM2. Correct me if I am wrong.
Thanks. |
|
Back to top |
|
 |
PeterPotkay |
Posted: Wed Oct 14, 2009 11:42 am Post subject: |
|
|
 Poobah
Joined: 15 May 2001 Posts: 7722
|
Write a custome Cluster Workload Exit that checks the q depth of every destination q before it decides where to send a message...and watch your throughput rate drop like a rock.
-or-
Get more disk; make the q bigger; move on to real problems. _________________ Peter Potkay
Keep Calm and MQ On |
|
Back to top |
|
 |
bruce2359 |
Posted: Wed Oct 14, 2009 11:49 am Post subject: |
|
|
 Poobah
Joined: 05 Jan 2008 Posts: 9469 Location: US: west coast, almost. Otherwise, enroute.
|
Quote: |
Now from the MQ admin perspective can we do something for msgs not going to DLQ |
Not really. If you make maxdepth somewhat bigger, it is possible that an application can put enough messages to fill the queue.
The ultimate resolution (presuming that a full queue will only be a temporary problem) is to create a dead-letter queue handler script that tries n times to put the message to the original destination queue.
Disk space is sooooo cheap, there's no excuse to run short.
Set maxdepth to 999999999 - the WMQ maximum queue depth. Only when messages are actually put to the queue will disk space be consumed. _________________ I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
Back to top |
|
 |
mqjeff |
Posted: Wed Oct 14, 2009 12:33 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 17447
|
Write a monitoring app that watches the qdepth and alters the clwlprty or clwlwght? |
|
Back to top |
|
 |
Sam Uppu |
Posted: Wed Oct 14, 2009 1:09 pm Post subject: |
|
|
 Yatiri
Joined: 11 Nov 2008 Posts: 610
|
Jesus... I dont want to write any code or any monitoring tool. I was looking for anything we can configure in MQ to do this. If nothing can be done from MQ side to stop the msgs going to DLQ, then I will pull the app team why they went with this stupid approach and ask them to rebuild their app.
Thanks. |
|
Back to top |
|
 |
fjb_saper |
Posted: Wed Oct 14, 2009 1:35 pm Post subject: |
|
|
 Grand High Poobah
Joined: 18 Nov 2003 Posts: 20756 Location: LI,NY
|
Sam Uppu wrote: |
Jesus... I dont want to write any code or any monitoring tool. I was looking for anything we can configure in MQ to do this. If nothing can be done from MQ side to stop the msgs going to DLQ, then I will pull the app team why they went with this stupid approach and ask them to rebuild their app.
Thanks. |
The only thing you can do is increase max qdepth... Be prepared to have enough disk space (and log space) to hold the full batch if need be...
The right thing to do is scale the receiving app so that it can handle the peak loads. You may combine both qdepth and scaling of the processing app...  _________________ MQ & Broker admin |
|
Back to top |
|
 |
gbaddeley |
Posted: Wed Oct 14, 2009 2:10 pm Post subject: |
|
|
 Jedi Knight
Joined: 25 Mar 2003 Posts: 2538 Location: Melbourne, Australia
|
Sam Uppu wrote: |
I was looking for anything we can configure in MQ to do this. |
There is! Increase the maximum queue depths to at least cope with the backlog of message queuing that is seen in normal operations. That's the whole point of MQ store-and-forward queuing. MQ deals with the production and consumption of asynchronous messages so that the apps can operate without knowing about each other or what MQ is doing.
You may also need to increase the disk space allocation to MQ.
This is a simple capacity planning exercise. _________________ Glenn |
|
Back to top |
|
 |
Vitor |
Posted: Wed Oct 14, 2009 2:42 pm Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
Sam Uppu wrote: |
I was looking for anything we can configure in MQ to do this. |
Set up the supplied dead letter handler to dump anything with a 2053 back in the input queue, and accept this is a quick kludge with a few downsides.
Sam Uppu wrote: |
I will pull the app team why they went with this stupid approach and ask them to rebuild their app. |
Do that anyway. While buying more disc and setting max depth correctly for your needs. _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
bruce2359 |
Posted: Wed Oct 14, 2009 2:56 pm Post subject: |
|
|
 Poobah
Joined: 05 Jan 2008 Posts: 9469 Location: US: west coast, almost. Otherwise, enroute.
|
There are circumstances beyond your (any syasdmin) control that will cause a msg to end up in the DLQ. Most of the reasons msgs go to the DLQ are out of your control.
You are asking if there is anything you can do in this one circumstance to keep msgs from going to the DLQ. So, what if the msgs end up in a DLQ for any of the other reasons?
Application developers need to understand that messages MAY end up in the DLQ - despite their wishful thinking; and they need to design something to do when this happens. _________________ I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
Back to top |
|
 |
gbaddeley |
Posted: Thu Oct 15, 2009 4:23 pm Post subject: |
|
|
 Jedi Knight
Joined: 25 Mar 2003 Posts: 2538 Location: Melbourne, Australia
|
bruce2359 wrote: |
...Application developers need to understand that messages MAY end up in the DLQ - despite their wishful thinking; and they need to design something to do when this happens. |
Besides the MQ admins implementing the standard DLQ handler to try redelivering the messages to the app queue, the app designers need to be aware that messages may come from the DLQ handler in bursts, out of order, or a long time after they originally expected them. _________________ Glenn |
|
Back to top |
|
 |
Mr Butcher |
Posted: Thu Oct 15, 2009 10:13 pm Post subject: |
|
|
 Padawan
Joined: 23 May 2005 Posts: 1716
|
if you do not want messages to be routed to the DLQ, rmove it from the qmgr DEADQ attribute. if your local queue on remote qmgr becomes full, channels will be stopped / retry as messages can not be delivered (assuming persistence and npmspeed stuff as discussed many times before), and they will pile up on the SCTQ. if that one is full too, the sending application will get the 2053 again. (if this is fine for you and is handled by the application, as it already gets the 2053 for the "local" target queue).
there are pros aond cons about running without dlq, but i do not want to start this discussion here. _________________ Regards, Butcher |
|
Back to top |
|
 |
|