| Author |
Message
|
| banjar1 |
 Posted: Mon Oct 15, 2007 4:21 am Post subject: Publish and DEAD.LETTER.QUEUE |
 |
|
Acolyte
Joined: 29 Nov 2006
Posts: 54
Location: FRA
|
Hi.
I'm testing Publish & Subscribe mechanism as this looks like the only way to get MQ to multiply messages. I use it in a rather simple way: I registered two queues, one Local and one Remote (pointing to another test QM) and started publishing. For test purpose I shrinked the transmission queue (connected with subscriber) to accomodate only 5 messages, stopped the channel and published 10 messages. 5 were processed (of course), the rest was not - they were kept on the stream queue. So far so good, that was expected. But the problems started when I restarted the channel: broker kept reporting that it couldn't put a message on DEAD.LETTER.QUEUE (!!) instead of starting processing the messages normally.
Is there any way to restart/reset a stream so _all_ messages kept in it (for any reasons) would be delivered normally to subscribers after all obstacles are removed? We can't afford loosing a single message and if even only one is put on DLQ it is lost for us. We don't use DLQ and we don't have any DLQ manager.
It is MQ6 with built-in broker.
Regards. |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Mon Oct 15, 2007 4:48 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
There are a whole bunch of other ways to create more than one copy of a message. An application level distribution list is one of them.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Oct 15, 2007 4:50 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
You should always have a DLQ, even if your system has no use for it, to cater for exceptional circumstances. DLQ management in this instance is very simplistic - there's a message in it, there's a problem someone needs to look at.
Consider using a broker collective to handle the remote subscribers rather than a remote queue.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| banjar1 |
| Posted: Mon Oct 15, 2007 5:15 am Post subject: |
|
|
Acolyte
Joined: 29 Nov 2006
Posts: 54
Location: FRA
|
I would rather stick to my question then start a general discussion on our approach, but here are answers:
1. I'm working with MQ, I have no influence on application being used with it. I want to control the distribution list dynamically and MQ is best for that.
2. One can use DLQ only if there is a proper handler available. We have no handler - and no resources to obtain one. Also, we have other ways to determine if troubles are ahead then an extreme situation, when a message is already put somewhere else than it's destination.
I have read about broker meanwhiel and it looks like there is no way around this...
What is a "broker collective"? |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Mon Oct 15, 2007 5:24 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
This is not an MB specific question - in fact, it's almost an MB non-specific question. Hopefully an interested moderator will move this to some other, more appropriate, forum.
You don't need to use Message Broker. A "broker collective" is an MB specific term, for a specific kind of pub/sub topology. You can configure more than one queue manager to run a regular MQ pub/sub broker, and connect the two of them. Then the movement of messages from the qmgr of publication to the qmgr of subscription is handled "automatically"
If you can't control the applications at all, you can't tell them to use Pub/Sub. They will have to change their code to use it.
There is a Dead Letter Handler that comes with the MQ. It's fairly comprehensive in function, and well documented. There are lots of good reasons for having a DLQ in the first place, even if you don't use any kind of Handler. For one, it ensures that you don't LOSE messages.
There are still several different ways to produce more than one message. One of those ways may actually be *more* suited for your actual requirements and *easier to manage* than pub/sub.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| banjar1 |
| Posted: Mon Oct 15, 2007 5:35 am Post subject: |
|
|
Acolyte
Joined: 29 Nov 2006
Posts: 54
Location: FRA
|
Ok, more explanation then:
To make the dev people to enhance our application with publish command took me 3 years.
I have the control over only one QM: the HUB. It delivers data to our customers - and I don't want them to even notice (much less do anything about that) that we started to use P&S. Therefore I publish and subscribe inside HUB QM only.
Exactly by NOT using DLQ we don't lose any messages. Not to mention, that our data flows are both time and order critical. Please believe me, we are fully aware of benefits and drawbacks of DLQ, this discussion has been going on for years here. So far it's minuses are too many and too big.
I think my question is straight and very about MB: how to make it working without DLQ?
We have had IBM people here to present us the options for us and they only suggested P&S, they have not mentioned any other way to multiply messages. Would you please give me some hints so I can google for more? |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Mon Oct 15, 2007 5:47 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
What are the requirements to create more than one message?
If you can change the HUB area to use Pub/Sub, then you can change the HUB area to use a distribution list. This distribution list can be populated from a lot of different sources, depending on how you code the Hub area.
You can also use an API exit (perhaps based on the sample mirrorq) to duplicate any messages PUT to one queue to be PUT to more than one queue.
Those are probably the most straight forward choices.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Oct 15, 2007 5:51 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| jefflowrey wrote: |
This is not an MB specific question - in fact, it's almost an MB non-specific question. Hopefully an interested moderator will move this to some other, more appropriate, forum.
|
Moved to General MQ as this is regarding base MQ broker, not WMB.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| banjar1 |
| Posted: Mon Oct 15, 2007 5:54 am Post subject: |
|
|
Acolyte
Joined: 29 Nov 2006
Posts: 54
Location: FRA
|
Ech.... We have almost 40 customers, each gets at least 10 types of data, each data type means about 3k messages per hour. I can't have na exit to cope with that.
We were lookin gat distribution lists years ago and discarded it as not suitable for us. Don't remevber why now.
Please, does it mean there is NO WAY to make broker to re-try putting on subcriber queue after it is available again instead of trying to put on DLQ till the end of the wrold? |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Oct 15, 2007 6:01 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| banjar1 wrote: |
| Please, does it mean there is NO WAY to make broker to re-try putting on subcriber queue after it is available again instead of trying to put on DLQ till the end of the wrold? |
I would doubt it because the previous subscriber is unavailable - it's now a new instance. But I've limited experience with the base MQ pub/sub.
It does sound like the problem you're experiencing though.
If it was me I'd define a DLQ and set the handler to reprocess the message, but that's ruled out by your site standards.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Mon Oct 15, 2007 6:07 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Okay, those are good requirements for pub/sub.
The broker should have decided it was done with the messages (i.e. "had delivered them") when it put them on the transmission queue - either that or it should have decided that the remote subscriber was not "active" and not tried to send anything.
I don't quite follow your explanation of what you've done to "break" things.
Messages are only ever sent to the active subscribers for a given topic at the time of the publication. When a subscriber registers, it will only receive NEW publications - unless you're using Message Broker and Retained Publications.
What happens if you endmqbrk/strmqbrk?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| vmcgloin |
| Posted: Mon Oct 15, 2007 6:23 am Post subject: |
|
|
Knight
Joined: 04 Apr 2002
Posts: 560
Location: Scotland
|
| Quote: |
| It is MQ6 with built-in broker. |
The point about MB is that the built in broker is not message broker.
Chapter 8 in the "WebSphere MQ Publish/Subscribe User's Guide" gives more information about error handling in the MQ broker and provides the retry type behaviour that you might be looking for (e.g. DLQNonPersistentPublication=no).
In particular:
| Quote: |
If the message is a Publish command message, and there is a problem sending an outgoing message on to a subscriber, the processing is as follows:
v The outgoing message is put to the dead-letter queue, if this is permitted by the broker and queue manager configuration. If the outgoing message cannot be put to the dead-letter queue .. it is discarded if this is permitted by the broker and queue manager configuration.
v If the outgoing message cannot be put to the dead-letter queue or discarded, the input message is restored. The input message is retried after suitable time interval, and (less frequently) a message is written to the log to alert the administrator.
Note: If the broker cannot put a publication message onto a destination queue or the dead-letter queue, and cannot discard the message, the broker continues trying to put the publication message onto the destination queue (at suitable intervals) and does not continue processing subsequent messages. The dead-letter queue and discard options for nonpersistent messages are specified in queue manager configuration file (qm.ini or equivalent). These options are described in Chapter 10, “Setting up a broker,” on page 99. |
Cheers,
Vicky |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Oct 15, 2007 8:11 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
| banjar1 wrote: |
Ech.... We have almost 40 customers, each gets at least 10 types of data, each data type means about 3k messages per hour. I can't have na exit to cope with that.
We were lookin gat distribution lists years ago and discarded it as not suitable for us. Don't remevber why now.
Please, does it mean there is NO WAY to make broker to re-try putting on subcriber queue after it is available again instead of trying to put on DLQ till the end of the wrold? |
The way out of your dilemma is quite easy. Add another qmgr to your setup. That qmgr has a DLQ and does pub/sub.
Whenever you have a message in the DLQ (alerts) you need to research the reason it hit the DLQ (hint RC in DLQH).
When you are ready you can then reprocess the message from the DLQ using the DLQ handler... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| banjar1 |
| Posted: Mon Oct 15, 2007 9:15 am Post subject: |
|
|
Acolyte
Joined: 29 Nov 2006
Posts: 54
Location: FRA
|
jefflowrey:
I simply caused the subscriber queue to be full.
vmcgloin:
that is exactly what I expected from it. Unfortunately even after I restarted the broker it still tries to put a message into DLQ...
ALL our messages are persistant. There are situations when we get hundreds of them queued (read: they would be put into DLQ if we had one). No man can hadle such situations, especially at 4:00 a.m. And we deliver data and offer a TechHelpdesk 24/7.
I remember now why we didn't go for DL. Our application can only re-read config file if restarted - and that means an outage for all our customers. That's why I'm giving P&S a try. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Oct 15, 2007 9:21 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7716
|
banjar, that's odd beahviour. Over the weekend we had a subscriber q fill up. The broker started putting messages to the DLQ. As soon as we upped the max depth of the subscriber queue new messages stopped going to the DLQ and instead started going to the bigger subscriber q.
I'm puzzled as to why the broker in your case wouldn't resume putting to the remote q def once the underlying XMITQ was able to take more messages.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
|
|
