| Author |
Message
|
| Adrien |
 Posted: Sat Jul 24, 2010 6:06 am Post subject: Clustered queue as a subscription destination |
 |
|
Novice
Joined: 23 Nov 2009
Posts: 14
|
Hello !
In my mq topolgy I need to create an administrative (durable) subscription with a destination that points to a clustered queue.
I have two message broker instances which are clones of each other. The input queues of the flows are clustered for the purpose of load balancing and to prevent any service downtime.
It works perfectly when client applications put messages on the clustered queues directly. But when the source of message comes from a subscription, it seems that the messages are first forwarded to the QM where the subscription has been made and destination resolution take place there. I expected destination resolution to be done on the publisher side (because it presumably knows subscription details using the proxy sub).
Now when the QM where the subscription has been done is down, the messages are blocked on the publisher side altough there is at least one instance of the clustered queue available. I have a single point of failure in this case : the QM where the subscription has been done.
The only workaround I have found is to create subscriptions on the publisher so if the QM where the subscription is made is down, it doesn't matter (the publisher is also down). But it only works when there is one publisher by topic ! If there are multiple publishers they depend on the one which has the subscription.
My questions are : Is it the desired behaviour ? Am I missing something there ? Have you some insight to work around this limitation ?
Thanks a lot for your answers. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Sat Jul 24, 2010 9:42 am Post subject: Re: Clustered queue as a subscription destination |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Adrien wrote: |
| My questions are : Is it the desired behaviour ? Am I missing something there ? Have you some insight to work around this limitation ? |
It's certainly not why you desire but it's working as designed. Even though the queues are clustered, the subscriber application (in this case a broker flow) is connected to and subscribing from a given queue manager. So if that subscriber (which I accept is a copy of the other) is down then no subscriptions will flow. If you're looking for load balancing then both flows should be running & subscribed at which point subscriptions will flow.
Now it may be possible to create a subscription with an alias queue as it's target and break the link to a given queue manager. Can't say I've tried it myself & would welcome insights from other with more pub/sub than me.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Jul 24, 2010 3:35 pm Post subject: Re: Clustered queue as a subscription destination |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Adrien wrote: |
Hello !
In my mq topolgy I need to create an administrative (durable) subscription with a destination that points to a clustered queue.
I have two message broker instances which are clones of each other. The input queues of the flows are clustered for the purpose of load balancing and to prevent any service downtime.
It works perfectly when client applications put messages on the clustered queues directly. But when the source of message comes from a subscription, it seems that the messages are first forwarded to the QM where the subscription has been made and destination resolution take place there. I expected destination resolution to be done on the publisher side (because it presumably knows subscription details using the proxy sub).
Now when the QM where the subscription has been done is down, the messages are blocked on the publisher side altough there is at least one instance of the clustered queue available. I have a single point of failure in this case : the QM where the subscription has been done.
The only workaround I have found is to create subscriptions on the publisher so if the QM where the subscription is made is down, it doesn't matter (the publisher is also down). But it only works when there is one publisher by topic ! If there are multiple publishers they depend on the one which has the subscription.
My questions are : Is it the desired behaviour ? Am I missing something there ? Have you some insight to work around this limitation ?
Thanks a lot for your answers. |
Yes you are missing something important here.
In the topic, at time of subscription you can specify the subscription's destination. In order for what you want to work, you would have to specify a cluster alias as the qmgr for the destination and you need to have your subscriber to listen on ALL qmgrs in the cluster hosting an instance of the destination. This way you would have no single point of failure...
The other way to consume the subscription is to use a queue consumer. The message's destination will still show the topic, but it is less elegant than the topic consumer... but works like a charm for us and allows for a truly anonymous broker... (load balanced)
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Adrien |
| Posted: Sat Jul 24, 2010 4:35 pm Post subject: |
|
|
Novice
Joined: 23 Nov 2009
Posts: 14
|
So if I understood you two correctly, I will have to make a subscription on each QM which has an instance of the clustered queue. So when one QM is down the other one still have a subscription active.
But if I do that, I assume I will receive n (2 in my case) copies of each message (one per subscription). So I will loose load balancing and simply will have the same messages duplicated to my 2 brokers...
Edit : after a bit of searching, I saw that using subscription grouping and right subcorrelid setup I can make two subscriptions to the same queue and have mq ignore all but one subscription. I wonder if it would work with a clustered queue though...
| Quote: |
| In the topic, at time of subscription you can specify the subscription's destination. In order for what you want to work, you would have to specify a cluster alias as the qmgr for the destination and you need to have your subscriber to listen on ALL qmgrs in the cluster hosting an instance of the destination. This way you would have no single point of failure... |
What do you mean by specifying a "cluster alias as the qmgr for the destination" ? Is it simply specifying a clustered queue as the destination or the words "cluster alias" has another meaning ? For now, in my administrative subscription, I just leave the destination qmgr as blank and use the clustered queue name as destination queue (So I have load balancing but not single point of failure which is the point where the only subscription is made).
And I found that in another topic (http://www.mqseries.net/phpBB2/viewtopic.php?t=53702):
| Quote: |
I guess what you are looking for is a design with no single point of failure.
Use a cluster alias as subscription qmgr and your publication becomes load balanced. That means that your application needs as many instances as qmgrs hosting the subscription queue in the cluster. It also most probably means that you will be reading the messages off the queue from a queue and not a topic.
Using a topic probably can be done but is much more trickier...
The nice part is that if one of the instances goes down you would still receive the messages on the others...
Now add to that a clustered subscription for a local subscription on multiple brokers and you get a system with no single point of failure... |
You seemed to describe the same thing as in this topic. It appears to be exactly what I need. I just don't get the "cluster alias" part again  |
|
| Back to top |
|
|
| Vitor |
| Posted: Sun Jul 25, 2010 1:11 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Adrien wrote: |
| What do you mean by specifying a "cluster alias as the qmgr for the destination" ? Is it simply specifying a clustered queue as the destination or the words "cluster alias" has another meaning ? |
It's a specific meaning. Discussed here & in the docs.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Adrien |
| Posted: Sun Jul 25, 2010 1:56 am Post subject: |
|
|
Novice
Joined: 23 Nov 2009
Posts: 14
|
| Vitor wrote: |
| Adrien wrote: |
| What do you mean by specifying a "cluster alias as the qmgr for the destination" ? Is it simply specifying a clustered queue as the destination or the words "cluster alias" has another meaning ? |
It's a specific meaning. Discussed here & in the docs. |
I searched a lot in the docs by I didn't find anything about it. Using google I found a lot of posts where people use it but no one describe what it is...
Sorry Vitor for being a bit lost (and I know you don't like giving someone the answer directly, which I understand) but can you please point me to an explanation or explain it briefly yourself ? Thank you. In the meantime I will continue to dig myself. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Jul 25, 2010 5:43 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
One subscription with the cluster alias should be good.
A cluster alias is a Remote queue definition, usually as follows:
| Code: |
| def qr(clustername) rqname(' ') rqmname(' ') cluster(clustername) xmitq(' ') |
For a better understanding  Intercommunications manual, and cluster manual  See Qmgr alias and queue resolution / path resolution.
You can have a single instance in the cluster or multiple instances of the cluster alias in the cluster. You need to understand the implications of having a single instance of the cluster alias in the cluster (single point of failure).
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Adrien |
| Posted: Sun Jul 25, 2010 6:01 am Post subject: |
|
|
Novice
Joined: 23 Nov 2009
Posts: 14
|
| Thank you. I never took the time to read the famous intercommunication manual but it is about time now. |
|
| Back to top |
|
|
| Adrien |
| Posted: Sun Jul 25, 2010 12:16 pm Post subject: |
|
|
Novice
Joined: 23 Nov 2009
Posts: 14
|
If I use a cluster alias in my subscription destination qmgr I will say to the queue manager which emit the publication that it can be routed to any queue manager in the cluster. There must be at least two cluster alias instances to have no single point of failure : that way the alias can "always" be resolved.
But today I use a blank qmgr destination. My publications are being load balanced the same way. What is the difference with the above ? I suspect that the publisher QM understand the blank QM as the QM where the subscription is made. And it's that which make my subscription QM a single point of failure. With the cluster alias it will know that the publication can be forwarded to any QM instead of just the QM where the sub has been defined. So finally the destination resolution really takes place at the publisher QM but that resolution was bogus in my setting.
Since I am heavily conjecturing here, I will try this tomorrow at work and give feedback then. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Jul 25, 2010 12:40 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Adrien wrote: |
If I use a cluster alias in my subscription destination qmgr I will say to the queue manager which emit the publication that it can be routed to any queue manager in the cluster. There must be at least two cluster alias instances to have no single point of failure : that way the alias can "always" be resolved.
But today I use a blank qmgr destination. My publications are being load balanced the same way. What is the difference with the above ? I suspect that the publisher QM understand the blank QM as the QM where the subscription is made. And it's that which make my subscription QM a single point of failure. With the cluster alias it will know that the publication can be forwarded to any QM instead of just the QM where the sub has been defined. So finally the destination resolution really takes place at the publisher QM but that resolution was bogus in my setting.
Since I am heavily conjecturing here, I will try this tomorrow at work and give feedback then. |
You're right. Even with the blank qmgr it acts like a reply to destination where it gets filled in with the subscribing qmgr. You can see that with requesting full feedback and doing the subscription with RFHUtil(c).
Like I said the downside of it is that the subscribing application needs to run a consumer on every instance of the subscribing destination.
This will be a problem if you have message affinity in your topic.
Also we found it easier to just have a queue consumer consume the publications. The Destination of the consumed message is still a topic, but you loose the capability of using a topic consumer.
To use a topic consumer you might have to subscribe on each instance, joining the initial subscription (avoid duplicate pub messages).
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Adrien |
| Posted: Sun Jul 25, 2010 1:04 pm Post subject: |
|
|
Novice
Joined: 23 Nov 2009
Posts: 14
|
The architecture and the business flows are designed to have no messages affinity so I will be fine. And I will effectively use an instance of message broker on each clustered queue instance as the subscribing application.
And I will use a queue consumer like you since it will effectively be easier. Message broker will consume messages with message broker on the destintation queue without knowing that they come from an administrative subscription. In fact they will even be mixed with other business message which come from classic queue messaging.
I will report back tomorrow (success or failure)... Thanks for the info.  |
|
| Back to top |
|
|
| Adrien |
| Posted: Sun Aug 15, 2010 6:05 am Post subject: |
|
|
Novice
Joined: 23 Nov 2009
Posts: 14
|
Sorry for being so late in my answer, I have been very busy at work recently.
I tried the cluster alias solution in the destination qmgr field (instead of blank). My cluster alias has been defined on two queue managers for redundancy. It doesn't work as expected : the message is always routed to the queue manager where the subscription has been defined (via a DEFINE SUB command). When I examine the properties of the proxy subscription on the publisher, I effectively see the subscribing qm as the destination qm and the queue SYSTEM.INTER.QMGR.PUBS as the destination queue. The message reach this intermediary destination first and then is sent to the real destination, thus making it a single point of failure. By the way, the consumer app. is already a queue consumer which have no idea that the message originates from a publication.
I will retry in the next few days. Perhaps I've gone wrong on some detail. 
If that solution cannot work in my context, I will look to the grouping subscription mechanism which can eliminate duplicates (http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/topic/com.ibm.mq.amqnar.doc/ps22830_.htm). But I don't know if it can do it when the destination is a multi-instance clustered queue... |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Aug 15, 2010 6:21 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Have you looked at an administrative subscription like in
| Code: |
| def sub dest(queue) destqmgr(clusteralias) topicstr(mytopic) |
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Adrien |
| Posted: Sun Aug 15, 2010 6:36 am Post subject: |
|
|
Novice
Joined: 23 Nov 2009
Posts: 14
|
| Yes that's exactly the subscription I made. It should work ? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Aug 15, 2010 10:16 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Adrien wrote: |
| Yes that's exactly the subscription I made. It should work ? |
So what have you tried and what was the result? And remember to set your alias up with bind not fixed as well as the destination queue (bind not fixed).
If you don't all messages will go to the same destination, once one has been chosen, instead of being load balanced... Also check if you need clusterQ sequence ANY or if the default (QMGR) still works for you.
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
