ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexIBM MQ API SupportPub/Sub with cluster of machines

Post new topicReply to topic
Pub/Sub with cluster of machines View previous topic :: View next topic
Author Message
koushikt
PostPosted: Tue Jan 27, 2015 5:52 pm Post subject: Pub/Sub with cluster of machines Reply with quote

Novice

Joined: 26 Jan 2015
Posts: 14

I am building a publisher/subscriber infrastructure with IBM MQ which is going to run on a cluster of machines. I just have one publisher and multiple apps are going to hook to it as a subscriber to the topic which it wants to listen to.

The problem is the apps run on a cluster of machines. The way I want it to work is every app running on its own cluster should receive its message once and only once.

Since every app must get it once and I don't control how many apps could hook on it, I can't set up a read once queue or a browsing queue. But if I use a pub/sub every app will get it on every machine/thread it is running on.

How do I solve this problem? Is there a way to set up read once at subscription name level and without having to face the duplicate subscription name error which MQ doesn't like.

I am doing this with C/C++.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Tue Jan 27, 2015 7:18 pm Post subject: Re: Pub/Sub with cluster of machines Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 7862
Location: US: west coast, almost. Otherwise, enroute.

koushikt wrote:
The problem is the apps run on a cluster of machines. The way I want it to work is every app running on its own cluster should receive its message once and only once.

I'm a bit confused. Why do you believe that topic subscribers will receive duplicate copies of publications to that topic?
_________________
I would tell you a UDP joke, but you might not get it.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Wed Jan 28, 2015 5:50 am Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17448

There's a big difference between a subscription and a subscription name.

The basic fact of pub/sub is that every subscriber gets a copy of every message published.

If you want to publish a message that will only go to a single subscriber, you must publish that message to a topic that is unique to that subscriber.

Whether or not those subscribers are all connected to the same queue manager or are connected to different queue managers in a cluster doesn't change the above.
Back to top
View user's profile Send private message
koushikt
PostPosted: Wed Jan 28, 2015 6:08 am Post subject: Reply with quote

Novice

Joined: 26 Jan 2015
Posts: 14

The same app is running multiple instances across the cluster. I just want one instance to get the message and not all.
For example, if I have 10 apps running on a cluster of 4 machines. I want each app to get the message exactly once and not once per cluster.

Hope this explains.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Wed Jan 28, 2015 6:27 am Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17448

You have ten apps running.

You have one app sending messages.

Do you want each message to be received by only one app?

Or each message received by every app?

Nothing about pub/sub or MQ clustering changes the once-and-only-once nature of MQ between sender and receiver.
Back to top
View user's profile Send private message
koushikt
PostPosted: Wed Jan 28, 2015 6:48 am Post subject: Reply with quote

Novice

Joined: 26 Jan 2015
Posts: 14

I want every app to receive it. But every app is running multiple instances. I want every app to receive once across all its instances.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Wed Jan 28, 2015 7:01 am Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17448

What does "multiple instances" mean?

Separate threads within a single process? Separate processes?

Separate subscriptions?

You will always get every message published at every subscription. If you want several instances of an application to only receive each published message once, then you must have only one subscription for each group of application instances.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Wed Jan 28, 2015 8:39 am Post subject: Reply with quote

Grand Poobah

Joined: 18 Nov 2003
Posts: 19430
Location: LI,NY

Been there, done that.
Create a manual subscription with target (cluster alias, delivery queue).
The total number of your app instances will need to listen to all of the delivery queue instances in the cluster. Delivering to the cluster alias delivery queue, allows you to load balance the delivery of the subscription across the active instances... Up to your app to make sure all messages are getting picked up... Please realize that there is no way to keep a delivery sequence intact this way...
Have fun
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
koushikt
PostPosted: Tue Feb 03, 2015 1:06 pm Post subject: Reply with quote

Novice

Joined: 26 Jan 2015
Posts: 14

I ended up creating a different queue for each app pool and subscribed the queues to the main topic.
Back to top
View user's profile Send private message
aspre1b
PostPosted: Fri Nov 18, 2016 1:02 am Post subject: Reply with quote

Acolyte

Joined: 05 Jul 2007
Posts: 73
Location: Coventry, UK

@fjb_saper

Quote:

Been there, done that.
Create a manual subscription with target (cluster alias, delivery queue).
The total number of your app instances will need to listen to all of the delivery queue instances in the cluster. Delivering to the cluster alias delivery queue, allows you to load balance the delivery of the subscription across the active instances... Up to your app to make sure all messages are getting picked up... Please realize that there is no way to keep a delivery sequence intact this way...




I've got a similar problem and I came up with the same solution you have suggested.

4 x queue managers in a cluster. 4 brokers on each node. A duplicate flow running on each broker and only one should consume the subscription message via a clustered alias/local queue.

The only problem I have is that I'm publishing the topic to the cluster, so I need to set up the manual subscription on one of the 4 MQ queue managers within the cluster. This would leave me with a single point of failure, as HA/multi-instance queue managers are not deployed. If I lost that qmgr, as I understand it the subscription would not be load balanced to another node.

Is there a way to make a single subscription resilient in the cluster?[/quote]
Back to top
View user's profile Send private message
aspre1b
PostPosted: Fri Nov 18, 2016 1:20 am Post subject: Reply with quote

Acolyte

Joined: 05 Jul 2007
Posts: 73
Location: Coventry, UK

Writing it down, I think I have now solved my problem.

Create local topics on each qmgr.
Create local subscriptions on each qmgr.
Subscriptions could point to a clustered alias/local queue if I want to load balance.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Fri Nov 18, 2016 4:39 am Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17448

Admin subscriptions are great, but it's hard to balance against an app that issues an MQOPEN on a topic.

Pros and Cons of each approach.
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
aspre1b
PostPosted: Mon Nov 21, 2016 12:27 am Post subject: Reply with quote

Acolyte

Joined: 05 Jul 2007
Posts: 73
Location: Coventry, UK

It's a shame you can't have a clustered subscription that can deliver to a clustered queue. You could have a single clustered subscription pointing to a clustered local queue used by a message flow. If that node instance failed, the subscription would persist in the cluster and still deliver to the clustered local queue.

I can understand why it would not be desirable as you would expect a client to only connect to one system to get a subscription.
There would be a question as to which node it falls over to, and would require the destination to be present.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Mon Nov 21, 2016 4:31 am Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17448

Nothing prevents you from defining the same Sub object on more than one queue manager.
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Mon Nov 21, 2016 5:58 am Post subject: Reply with quote

Grand Poobah

Joined: 18 Nov 2003
Posts: 19430
Location: LI,NY

The trick was to define the subscription with the cluster alias on each publishing node. As the cluster alias was not defined on any publishing node, the subscription was load balanced and only one message per publication was ever sent...

However if you define a clustered topic, you should be good to define a single clustered subscription using the cluster alias.

This should load balance the subscription without single point of failure ...

Have fun
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexIBM MQ API SupportPub/Sub with cluster of machines
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.