| Author |
Message
|
| mktgurutsm |
 Posted: Wed Aug 03, 2005 6:52 am Post subject: MQ Pub / Sub Issue |
 |
|
Novice
Joined: 08 Jan 2004
Posts: 21
Location: New York
|
Hi
We have a new application that wants to use the MQ Event Broker on z/OS for Pub/Sub. The Event broker will PUT the messages to the subscription queues in native MQ. A difficult question has came up from the applications. We asked IBM and they do not have the answer. Here is the scenario.
Say the event broker publisher sprays a message to 50 subscription queues. The native MQ queues will be listened to (polled) by MQClient connections. Say some of the MQClients (subscribers) are not active, so the message will sit on the queue. Now the application on the front end decides that the message should be retracted and for any subscribers that have not yet consumed the message, want to get the message from the queue. Are you with me on this? Is there a good way for the front end application or event broker to go back to the 50 queues and GET the message off any of the queues where it has not yet been consumed by the client?
This seems very tricky and difficult. Hope someone out there knows a good way to do this. Some things we are kicking around are using a Correlid (the eb supports Correlid). So if the front end app sends the msg with Correlid and the Correlid gets carried through the eb flow through to the delivery to the subscriber queue, and the front end app then decides to retract the message, couldn't the app instruct the eb (through a rule) to hit the subscriber queues and GET the message with that Correlid? |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Wed Aug 03, 2005 7:23 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
This really depends.
You can (and should, really) set the subscriptions up so that they are not retained. Then, if the subscriber goes away, no publications will go to their subscriptions - because they will be automatically unsubscribed.
You can (and again, should) also make sure that the publications are not retained. Then when a subscriber registers, it will not get the "backlog" of publications on a topic, it will only get any new publications.
You can also configure the publications to expire.
In other words, you have to go out of your way to set up the kind of problem you're trying to go out of your way to solve.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| wschutz |
| Posted: Wed Aug 03, 2005 7:43 am Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
So you're saying that the client applications do not cancel their subscriptions before going away...
_________________
-wayne |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Aug 03, 2005 8:03 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| wschutz wrote: |
| So you're saying that the client applications do not cancel their subscriptions before going away... |
It didn't sound like they'd written the client apps yet.
If they have, then setting the publications to expire is the quickest solution.
Followed by ensuring that the app always cancels their subscription before it goes away... followed by not using durable subscriptions and retained publications.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| wschutz |
| Posted: Wed Aug 03, 2005 8:26 am Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
So, if they are still writing the subscriber applications, why not just take the approach of ignoring all but the LAST message received on any topic, and if the publisher want to retract a publication on that topic, publish an empty message (which would be treated by the subscriber appplication as a non-message).
_________________
-wayne |
|
| Back to top |
|
|
| mktgurutsm |
| Posted: Wed Aug 03, 2005 9:35 am Post subject: |
|
|
Novice
Joined: 08 Jan 2004
Posts: 21
Location: New York
|
| We have and are considering that approach of using message priority which would put a retraction higher up on the queue for the subscriber to read. But in some cases, the application (which has not been written yet because it is still in the design phase) wants the subscriber to NEVER see the message if they have not yet consumed it from the MQ queue. That is really the heart of the question. How does the app or the event broker totally get rid of any given message after the event broker has delivered it to the MQ queue? |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Aug 03, 2005 9:43 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| mktgurutsm wrote: |
| How does the app or the event broker totally get rid of any given message after the event broker has delivered it to the MQ queue? |
Expiration is the only way. Well, the other way is to write a very complicated process that will read the broker database and find all subscribers who might have received a particular publication and then make client connections to all queue managers that those subscribers are hooked up to and then try and identify each subscriber queue and then try and remove the correct message. But don't do that. You'll hate your life three months after it goes live.
And remind your architects/designers that the whole point of pub/sub is that publishers are disconnected completely from subscribers. The job of the publisher is to publish information on topics. The job of the subscribers is to handle information on the topics they are interested in.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| wschutz |
| Posted: Wed Aug 03, 2005 9:50 am Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
Have you looked at using JMS (or XMS) real-time transport into the event broker instead of MQ? Since no "queueing" is involved, there's nothing to be deleted .... in this case, you MIGHT want to use retained publications (which your publisher can delete "at will").
Of course, we could always argue that there is a timing issue that might rear its ugly head .... 
_________________
-wayne |
|
| Back to top |
|
|
| mktgurutsm |
| Posted: Thu Aug 04, 2005 7:51 am Post subject: |
|
|
Novice
Joined: 08 Jan 2004
Posts: 21
Location: New York
|
| Wayne - think we might have a potential solution using Correlid. Question now is do we have access programatically to the event broker subscription list? |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Aug 04, 2005 8:03 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| mktgurutsm wrote: |
| Wayne - think we might have a potential solution using Correlid. Question now is do we have access programatically to the event broker subscription list? |
Yes, it's a database.
BUT NO.
DO NOT IMPLEMENT THIS.
Use Expiry.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| mktgurutsm |
| Posted: Thu Aug 04, 2005 9:35 am Post subject: |
|
|
Novice
Joined: 08 Jan 2004
Posts: 21
Location: New York
|
Jeff -
Expiry is also an option, but we are considering the other option of using Correlid and access to the event broker subscription list. Then if the app decides to "pull back" a message, they can do so.
-Given the scenario:
-App connects to z/OS event broker.
-Event broker publishes to 50 traditional MQ queues with Correlid.
-48 of the traditional MQ queues consume the message. 2 do not.
-App decides to "pull back" the message from any of the MQ queues (2 in this case) that have not yet consumed the message.
-App accesses the event broker subscription list, determines which 50 queues were previously published to.
-Event broker using Correlid of previously published message does MQ GET to the 50 subscriber queues with Correlid.
I agree that using message expiry is a much simpler solution. Perhaps using a very short expiry is a better solution. When we meet with IBM and the app next week, we will discuss both of these options.
In your opinion, are both viable options given the description of the app? |
|
| Back to top |
|
|
| wschutz |
| Posted: Thu Aug 04, 2005 9:41 am Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
Your design is making an assumption, which might be true today, and might not be true in the future...
| Quote: |
| Event broker using Correlid of previously published message does MQ GET to the 50 subscriber queues with Correlid |
The assumption is that the subscription queues will always be local queues (same qgmr as the broker). If they are remote queues, then the design fails.
_________________
-wayne |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Aug 04, 2005 9:53 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| mktgurutsm wrote: |
| In your opinion, are both viable options given the description of the app? |
In my opinion, building a pub/sub system that breaks the separation between publishers and subscribers is not a viable option.
In my opinion, building a system to rely on an undocumented and proprietary database layout that is subject to change at the whim of a vendor is not a viable option.
Do I need to tell you my opinion on doing both of those two things?
There, may, in fact, be a much better option than using Expiry. But that depends on your business requirements and how they are interpreted... The need to rescind a message is a good place to start if you decide to go down that road.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
|
|
