ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexClusteringShould the DLQ on a Queue Manager *must* be a local queue on

Post new topicReply to topic Goto page Previous  1, 2, 3, 4, 5  Next
Should the DLQ on a Queue Manager *must* be a local queue on View previous topic :: View next topic
Author Message
fjb_saper
PostPosted: Wed Mar 14, 2012 8:40 pm Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

And let's face it, compared to the queue remote setup, the queue alias pointing to a remote cluster queue is weak.
Depending on the communications and what not, you may get an error return code, letting you know that at the moment of your request the cluster could not resolve the destination of your alias queue.... bummer if you have it as DLQ and rightly it should not be accepted !!!

The queue remote however should always resolve, at least to the SCTQ where the message may wait to be delivered until communications allow it so... And remind me what happens when the only queue manager hosting the Enterprise DLQ goes down for maintenance and the queue has been declared as and alias? Suddenly the DLQ is no longer reachable!! and errors would happen all over your landscape?? This is certainly not the desired effect... One more thing that the QR covers nicely compared to the QA...

Have fun
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
mqdev
PostPosted: Thu Mar 15, 2012 6:10 am Post subject: Reply with quote

Centurion

Joined: 21 Jan 2003
Posts: 136

fjb_saper: I agree with your point and I did raise this possibility. The response was that we will house the ENTERPRISE_DLQ on a highly-reliable, multi-instance QM which is up 365x24x7. You prolly guessed it by now - am not a big fan of the idea - but in view of the history and local reasons, am forced down this path! The risks in this path are acknowledged and are mitigated (or atleast attempted to mitigate) to certain extent.

-mqdev
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu Mar 15, 2012 7:17 am Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

mqdev wrote:
The risks in this path are acknowledged and are mitigated (or atleast attempted to mitigate) to certain extent.

This design presumes that all dead-letter messages will successfully end up on a single enterprise-wide DLQ, on a single qmgr (MI or not), and will be successfully processed there.

Any single failure of a lesser-available qmgr (or channel, xmitq, etc.) will orphan (a verb) dead-letter messages on the failed qmgr. How will orphaned dead-letter messages be mitigated?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Thu Mar 15, 2012 7:29 am Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

bruce2359 wrote:
mqdev wrote:
The risks in this path are acknowledged and are mitigated (or atleast attempted to mitigate) to certain extent.

This design presumes that all dead-letter messages will successfully end up on a single enterprise-wide DLQ, on a single qmgr (MI or not), and will be successfully processed there.

Any single failure of a lesser-available qmgr (or channel, xmitq, etc.) will orphan (a verb) dead-letter messages on the failed qmgr. How will orphaned dead-letter messages be mitigated?


I think this horse has flown the coop.

The design mentioned here is not a recommended practice. Mqdev understands and knows this.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu Mar 15, 2012 7:41 am Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

mqjeff wrote:
I think this horse has flown the coop.

I like the way you mix metaphors.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
mqdev
PostPosted: Thu Mar 15, 2012 9:20 am Post subject: Reply with quote

Centurion

Joined: 21 Jan 2003
Posts: 136

bruce2359 wrote:
mqdev wrote:
The risks in this path are acknowledged and are mitigated (or atleast attempted to mitigate) to certain extent.

This design presumes that all dead-letter messages will successfully end up on a single enterprise-wide DLQ, on a single qmgr (MI or not), and will be successfully processed there.

Any single failure of a lesser-available qmgr (or channel, xmitq, etc.) will orphan (a verb) dead-letter messages on the failed qmgr. How will orphaned dead-letter messages be mitigated?



First off, not all QMs in the Enterprise join this Cluster. There is a checklist being worked out which will decide if a given QM will be part of the Cluster. "lesser available" QM or QMs which can potentially leave orphaned msgs on the ENTERPRISE_DLQ will not be part of the Cluster. Assuming we do run into that scenario, the orphaned msgs are just ignored.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu Mar 15, 2012 9:40 am Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

Orphaned messages are those dead-letter messages that are stranded on an unavailable (dead, network connection fails, etc.) qmgr, AND that do not end up on the ENTERPRISE_DLQ.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
mqdev
PostPosted: Thu Mar 15, 2012 10:58 am Post subject: Reply with quote

Centurion

Joined: 21 Jan 2003
Posts: 136

All,
We have been spending quality time on this topic and have seen some very infomative posts on this topic. I have tried to consolidate all criticisms and tried to provide my inputs (in italics). Let us for a few mins, set aside our egos and pretend we are part of a class and have a nice technical debate. I have a lot of respect for the technical capabilities of all you folks and at the same time do not believe myself to be a novice either. Here we go...

Thanks
-mqdev

• If a DLQ is a local queue then you can be assured it's quickly and easily available.
Assuming a highly-available Q in a low latency Network (we believe all these are true in our environment), is this really a concern? Not for us!

• What's going to happen in your estate if a high volume process is trying to dead letter and there's a serious network delay?
We are aware of this. The bosses are fine with it. In fact the Network delay/outage (even in a small busy segment if not Enterprise-wide) could lead to a vicious circle that could impact the MQ Network (for the precise definition freaks – this is the logical view of your MQ infrastructure) eventually crippling the MQ Network (due to the non-availability of DLQ). The bosses are aware and are ok with it with the belief that we will never run into that scenario – something I do not subscribe to (I am a fan of Murphys law - having seen it strike at most in-opportune moments!).

• Or someone accidently unshares the enterprise DLQ from the cluster?
Not possible without the right set of folks involved as the access rights are tightly controlled.

• Or changes the permissions on it?
Not possible without the right set of folks involved as the access rights are tightly controlled.

• Consider the case where a receiver channel can't deliver a message. The receiver channel will then attempt to write the message to the DLQ. In the case where the DLQ is in fact not a qlocal, this means it will attempt to write the message to an XMITQ.

Now suppose that the reason the receiver was unable to deliver the message was because the message was being multi-hopped, and the destination xmitq was full... But the destination qmgr is the same qmgr that hosts the remote DLQ.... who's xmitq is full...

We are talking 2 xmitQs here – the xmitQ on the QM (QM_FR in my example) hosting ENTERPRISE_DLQ and xmitq on a peer QM – say QM1 in my example scenario.

If the xmitQ on QM hosting ENTERPRISE_DLQ is full, the corresponding CHL will halt. This does not mean a channel to the QM_FR cannot be started! Off course if the DLQ on QM_FR is full, it will hose the QM and making ENTERPRISE_DLQ unavailable.

If the xmitq from QM1 to QM_FR is full, it will hose QM1 (as the msg tries to go the DLQ on QM1 and hence to the xmitQ – this becomes a endless loop)
This however can happen even without centralizing the DLQ. Consider an xmitQ on the QM gets full – the next msg that needs to go through the CHL will end-up on the DLQ (assuming persistent msg – QM is not allowed to “silently” discard the msg). Once the DLQ fills up too – the QM is essentially hosed.
While even a single msg can cause the QM to go into a endless loop (and result in a hosed QM) in the former case, we need a lot more msgs (fillup xmitq & DLQ completely) in the latter case.

We are aware of the all possibilities here….


• It is a useful and meaningful notion to centralize your management of DLQs. It is a recommended practice to do this by centralizing your MANAGEMENT of DLQs, not by centralizing your DLQs themselves.
Really? How exactly do you “Centralize the MANAGEMENT of DLQs”
Aha – here is a way….


Nothing says you can't modify the default dlq handler to be a client app and then read from remote DLQs and run several instances on a central machine.
This means multiple moving parts (each Client App is a moving part) – any single app down means the corresponding DLQ isn’t being serviced!

• It is also a recommended practice to force applications to rely on backout queues rather than on DLQs, and reserve DLQs for system level issues.
Agreed. What about the scenarios where all the backout attempts are exhausted (say BOQ is also full) and the msg still needs to go to DLQ?

• Again, the recommended practice is you should NOT move messages to a central DLQ, either manually or through MQ channels. The recommended practice is that you should set up jobs to process each DLQ remotely, and have those jobs deliver messages to queues relevant to the connected queue manager
I haven’t seen this recommendation.
The whole thing boils down to this: Reduce the number of moving parts and hence the breakage points. Like the Client Apps servicing the individual DLQs – any single job down means the corresponding DLQ is not being serviced.
ENTERPRISE_DLQ is an attempt to reduce the number of moving parts. In the process, we are putting all eggs in one basket and making the basket a potential single-point-of-failure. But with enough redundancy & monitoring built into the system, we should be better off (hopefully!) then having multiple, distributed points of failure
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Mar 15, 2012 11:28 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

mqdev wrote:
• It is a useful and meaningful notion to centralize your management of DLQs. It is a recommended practice to do this by centralizing your MANAGEMENT of DLQs, not by centralizing your DLQs themselves.
Really? How exactly do you “Centralize the MANAGEMENT of DLQs”
Aha – here is a way….



Here's another, which I mentioned earlier; have whatever's monitoring the individual queue managers and reporting to a central management point also monitor and report the arrival of messages on the local DLQ. This is almost certainly the same tool as will be used to monitor the centralized DLQ.

All the usual suspects in the monitoring space do this.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu Mar 15, 2012 1:47 pm Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

mqdev wrote:
This however can happen even without centralizing the DLQ. Consider an xmitQ on the QM gets full – the next msg that needs to go through the CHL will end-up on the DLQ.

Actually, no. The default behavior is... if an xmitq gets full, the app (trying to MQPUT the message) will get a ReasonCode indicating that the queue is full - the put will fail, and NO attempt will be made to next try the DLQ.

mqdev wrote:
If the xmitQ on QM hosting ENTERPRISE_DLQ is full, the corresponding CHL will halt.

No. xmitq full will not stop the channel. The xmitq on the QM hosting the ENTERPRISE_DLQ is for outbound traffic only.

mqdev wrote:
It is also a recommended practice to force applications to rely on backout queues rather than on DLQs, and reserve DLQs for system level issues.
Agreed. What about the scenarios where all the backout attempts are exhausted (say BOQ is also full) and the msg still needs to go to DLQ?

The qmgr provides no automation/magic to do anything with the BOQ. It is the responsibility of the application developer to interrogate the value of the BOQ queue name, then MQPUT the offending message to the BOQ queue. If, as you state, the BOQ is full, then the app needs to decide what to do next with the offending message. Adequate BOQ MAXDEPTH, and monitoring software to increase boq MAXDEPTH, should the need arise, seems to be a simpler solution.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Thu Mar 15, 2012 1:53 pm Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

mqdev wrote:

• Again, the recommended practice is you should NOT move messages to a central DLQ, either manually or through MQ channels. The recommended practice is that you should set up jobs to process each DLQ remotely, and have those jobs deliver messages to queues relevant to the connected queue manager
[i]I haven’t seen this recommendation.


PeterPotkay wrote:

If the manuals say it has to be a local queue, if there are reasons articulated why it should be a local queue, it doesn't really matter if you do get it working somehow. The next fix pack may close the hole that you exploited that allowed it to work not being a local queue.


Open the PMR. Get the official answer from IBM.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
mqdev
PostPosted: Tue Apr 03, 2012 6:12 am Post subject: PMR response.... Reply with quote

Centurion

Joined: 21 Jan 2003
Posts: 136

Hello Folks,
Here is the response of IBM on this topic:

As you noted, the requirements for the DLQ in the infocenter clearly
document that is must be a local queue.

While I have not been able to find the reason for this requirement
originally being imposed, I suspect it is because one of the main uses
of the DLQ is to give channels a place to put messages which cannot be
transmitted. This allows a channel to continue to run when it encounters
a problem message.

Defining a qremote for the DLQ would risk creating a loop where a
problem message can't be sent over a channel, and gets put to the
DLQ. Since this is a qremote the message would end up back on the xmit
queue and the channel would try to process it again.
This is likely to have a severe impact on the channels involved.

Even if the qmgr does not prevent the use of a qremote as the DLQ, this
configuration will not have been tested, and would not be considered a
supported configuration.


Depending on exactly what the customer's requirement is, an alternative
may be to configure a local DLQ with a dead letter handler application
to reroute messages. Using their own application to send problem
messages back to another destination would mean that you could include
logic to handle cases where messages can't be sent and repeatedly return
to the DLQ.


The so-called loop scenario can occur even if the DLQ is a local Q on the QM. So we are left with "mama knows the best - just do as I say" kind of response without any concrete reason as to why it has to be that way. We are trying to see if we can approach the design team in Hursley to get this answered. In the meantime, ex-IBMers on the forum - can you please chip in?

Thanks
-mqdev
Back to top
View user's profile Send private message
Vitor
PostPosted: Tue Apr 03, 2012 6:23 am Post subject: Re: PMR response.... Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

mqdev wrote:
The so-called loop scenario can occur even if the DLQ is a local Q on the QM.


How? An example scenario please, where the application causing the dead lettering of a message doesn't get a failure reason code on the put.

mqdev wrote:
In the meantime, ex-IBMers on the forum - can you please chip in?


I have.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
mqdev
PostPosted: Tue Apr 03, 2012 6:39 am Post subject: Re: PMR response.... Reply with quote

Centurion

Joined: 21 Jan 2003
Posts: 136

Vitor wrote:
mqdev wrote:
The so-called loop scenario can occur even if the DLQ is a local Q on the QM.


How? An example scenario please, where the application causing the dead lettering of a message doesn't get a failure reason code on the put.


What happens in the following scenario?
Lets say there is a problem with a Channel. The result would be that the QM accepts the msgs and stacks them up on the corresponding xmitq. The MQPUTs themselves are successful here. Now once the xmitq itself gets full, the QM would start dumping the msgs on the DLQ. Once the DLQ also gets full, the next msg would go into a infinite loop...
Back to top
View user's profile Send private message
mqjeff
PostPosted: Tue Apr 03, 2012 6:48 am Post subject: Re: PMR response.... Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

mqdev wrote:
Now once the xmitq itself gets full, the QM would start dumping the msgs on the DLQ. Once the DLQ also gets full, the next msg would go into a infinite loop...

No.

Once teh DLQ also gets full, the MQPUT fails with an MQCC!=0.
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Goto page Previous  1, 2, 3, 4, 5  Next Page 4 of 5

MQSeries.net Forum IndexClusteringShould the DLQ on a Queue Manager *must* be a local queue on
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.