| Author |
Message
|
| belchman |
 Posted: Thu Jan 27, 2022 11:42 am Post subject: MQ PUB/SUB and Possible Dup Messages |
 |
|
Partisan
Joined: 31 Mar 2006
Posts: 386
Location: Ohio, USA
|
Hello all,
Been a while! And boy do I have a doozie of a problem I am hoping you can help me with.
We have this mechanism in prod that uses MQ pub/sub. The publisher gets info from customers and publishes a message about that interaction. The publisher (or publishers) is a system that runs in an LPAR on z/OS.
There are 5 nodes in scope of this discussion and 4 are MQ for z/OS and 1 on MQ for AIX. They are all in the same MQ cluster as well. The z/OS queue managers are running in the same LPAR as is the application that is the publisher.
This is the part that has me baffled. There are 2 subscribers and both of them are MQ for z/OS queue managers. Lets call them MQP1 and MQP2 out of the entire set of MQP1/2/3/4. I am thinking that any of the 4 z/OS queue managers can publish if they desire to.
This is the fun part. The destination on the subscription is a cluster queue to the MQ for AIX queue manager that all 4 z/OS queue managers can see. To me, this means that a publication (message) will be sent to MQ for AIX from any z/OS queue manager that subscribes to that publication.
Not that it has a bearing on this, but it may, the actual topics that relate to the topic strings are also shared in the same cluster as the cluster queue that is the subscription destination.
Help me out. I am not the industry's leading MQ pub/sub expert! With this setup, it seems to me that at each publish event, the MQ for AIX queue manager will get a message from each subscriber. In this case 2 messages for each publication.
You are probably asking me why is it that the MQ for AIX queue manager is not the only subscriber and the destination not a cluster queue? I wonder the same thing and have no idea.
What do you think?
_________________
Make three correct guesses consecutively and you will establish a reputation as an expert. ~ Laurence J. Peter |
|
| Back to top |
|
 |
| hughson |
| Posted: Thu Jan 27, 2022 7:13 pm Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
I'm going to restate your problem description in my own words to see if I have understood it.
You have 2 subscribers, one on MQP1 and one on MQP2. Both subscribers have as their destination queue, a cluster queue that resides on an AIX Queue Manager.
When one message is published, it will match both subscribers and each subscriber will be delivered a copy of that message. Since the subscriber queue is actually the same downstream queue on the AIX QMgr, there will be two copies on that queue for each message published.
Unless the subscribers are different in some way? Different topic string or perhaps subscribing using selectors on message properties as well? Otherwise, always gonna get two copies of the message.
I assume that the setup you describe has been done for some sort of redundancy?
If all 5 queue managers are in the same cluster, then clustered pub/sub could be used instead. Make a subscription on AIX Qmgr using a non-clustered queue as the destination. Advertise the topic in the cluster instead of the destination queue.
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| belchman |
| Posted: Fri Jan 28, 2022 4:20 am Post subject: |
|
|
Partisan
Joined: 31 Mar 2006
Posts: 386
Location: Ohio, USA
|
Thank you for you input Morag. You have describe my problem accurately and I was not around when this was done so have no idea why if was done this way.
Yesterday after work, when thinking about this problem, I wondered if maybe on the separate and distinct LPARs there was a different pub/sub broker and a pub from LPAR1 would only go to the sub on LPAR1 and the same for LPAR2.
Then no dups would occur. I will look into that.
I have a meeting with the publishing app today to get more details about what they are doing. I suspect it is possible that they occasional get duplicates that they publish. We will see.
The thing that diminishes the former theory (2 subscribers sending AIX the same message) is that the report from the consumption side (the AIX app) is that not every message is duplicated. This makes the issue harder to nail down.
We will definitely have to redo this scheme somehow once we figure out the cause. The clustered pub/sub seems like a perfect fit and also seems like what the initial system was supposed to be since they have the topics in the cluster.
If anyone else has any thoughts on this, I would appreciate your input.
Thanks
_________________
Make three correct guesses consecutively and you will establish a reputation as an expert. ~ Laurence J. Peter |
|
| Back to top |
|
|
| belchman |
| Posted: Fri Jan 28, 2022 10:47 am Post subject: |
|
|
Partisan
Joined: 31 Mar 2006
Posts: 386
Location: Ohio, USA
|
Upon finding this gem by Mr Ware, I may see what we have improper settings.
https://www.mqtechconference.com/sessions_v2014/MQTC_pubsub_networks.pdf
On pages 44 and 45 of that document it warns of NOT clustering the topic and that is actually what we have implemented on each queue manager MQP1 and MQP2. The topic is in the cluster.
The only issue I have is that the endpoint getting the actual message put on its local queue has anecdotal evidence that only about 10% of the messages are duplicated over a period of time. Out of 800k messages, only 80k are duplicated.
I would think they would all be duplicated. Either the evidence is wrong or I am wrong.
_________________
Make three correct guesses consecutively and you will establish a reputation as an expert. ~ Laurence J. Peter |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Jan 29, 2022 7:46 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| belchman wrote: |
Upon finding this gem by Mr Ware, I may see what we have improper settings.
https://www.mqtechconference.com/sessions_v2014/MQTC_pubsub_networks.pdf
On pages 44 and 45 of that document it warns of NOT clustering the topic and that is actually what we have implemented on each queue manager MQP1 and MQP2. The topic is in the cluster.
The only issue I have is that the endpoint getting the actual message put on its local queue has anecdotal evidence that only about 10% of the messages are duplicated over a period of time. Out of 800k messages, only 80k are duplicated.
I would think they would all be duplicated. Either the evidence is wrong or I am wrong. |
Have you thought setting up a clustered topic with topic hosts being MQP1 and MQP2. Your topic hosts would highly available and you would not propagate the messages throughout the whole cluster... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| hughson |
| Posted: Sun Jan 30, 2022 9:59 pm Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
| belchman wrote: |
| The only issue I have is that the endpoint getting the actual message put on its local queue has anecdotal evidence that only about 10% of the messages are duplicated over a period of time. Out of 800k messages, only 80k are duplicated. |
Are the two subscriptions identical? Post the output of a DISPLAY SUB here so we can view.
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| belchman |
| Posted: Mon Jan 31, 2022 5:52 am Post subject: |
|
|
Partisan
Joined: 31 Mar 2006
Posts: 386
Location: Ohio, USA
|
The subscriptions are on z/OS so I cannot perform a DIS SUB. I can only tell you by visual inspection via MQ Explorer.
Here are key properties of the subscription. I will see if the z/OS guy can get a display when he gets in. He is west coast so that will be later today.
Topic name: <null>
Topic String: tdd/events
Scope: All
Destination: <cluster queue on AIX>
We think the issue is due to the Publication Scope and/or Subscription Scope properties of the topic which is ANY. If MQP1 (or a process connected to) publishes, then we want MQP1's subscription to be the only one that is delivered. So we are going to change the scope to QMGR. The same for MQP8.
Here are key properties of the topic:
Topic string: tdd/events
Publish scope: as parent
Subscription scope: as parent
Cluster: <clustername>
It is still not 100% decided to remove the topic from the cluster. I will let all know if we decide to do that as well.
Sorry I cannot offer the rationale as to why we did this this way. It was before my time. But it appears it was for resiliency. If MQP1 or MQP8 is down, they wanted it to still work.
David's words on page 44/45 of that link were a catalyst to our findings.
_________________
Make three correct guesses consecutively and you will establish a reputation as an expert. ~ Laurence J. Peter |
|
| Back to top |
|
|
| belchman |
| Posted: Mon Jan 31, 2022 5:56 am Post subject: |
|
|
Partisan
Joined: 31 Mar 2006
Posts: 386
Location: Ohio, USA
|
So all know. MQP1 and MQP2 are the only publishers and the only subscribers.
If it was me setting this up from scratch. MQP1 and MQP2 would be the publishers and the AIX queue manager would be the only subscriber. That's effectively what is happening anyway but the subscriber is getting 2 copies of the same publication.
Since this is production, its not that easy to change it  Inertia and all.
_________________
Make three correct guesses consecutively and you will establish a reputation as an expert. ~ Laurence J. Peter |
|
| Back to top |
|
|
| belchman |
| Posted: Wed Feb 02, 2022 9:24 am Post subject: |
|
|
Partisan
Joined: 31 Mar 2006
Posts: 386
Location: Ohio, USA
|
We replicated the issue in DEV and figured out how to resolve it.
In a nutshell, all we did was 1) add the DEV topics to the MQ cluster (to match production) and 2) reduced the scope of the publication and subscriptions to QMGR from ALL (the opposite of production).
_________________
Make three correct guesses consecutively and you will establish a reputation as an expert. ~ Laurence J. Peter |
|
| Back to top |
|
|
|
|
