ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General IBM MQ Support » Durable subscriptions to a queue

Post new topic  Reply to topic Goto page 1, 2  Next
 Durable subscriptions to a queue « View previous topic :: View next topic » 
Author Message
simonalexander2005
PostPosted: Wed May 31, 2017 1:50 am    Post subject: Durable subscriptions to a queue Reply with quote

Acolyte

Joined: 13 Jun 2016
Posts: 55

Hi,

Say I have the following configuration (MQ8/IIB9):

Code:
QMGR0/Broker0:
Local Queue -> IIB Message Flow -> Alias Queue -> Topic

DurableSubscription1 -> (QMGR1) Cluster Alias Queue1 -> Local Queue
DurableSubscription2 -> (QMGR2) Cluster Alias Queue2 -> Local Queue


Subscriptions are configured on QMGR0, and all three QMGRs are in the same cluster.

Now, I understand that the delivery options (NPMSGDLV, PMSGDLV) can be set so that either

a) every subscription succeeds or fails together (ALL)
b) every subscription is individual (ALLAVAIL)
or c) every durable subscription must succeed, or none do (ALLDUR).

And I have managed to set it up successfully using any of those options.

The issue I have is, that if QMGR2 (say) goes down; it doesn't receive the messages it misses when it comes back up.

Is there any way to associate the qmgr (with a subscription ID of some kind, perhaps?) so that when it starts up again it receives the messages it's missed?


Last edited by simonalexander2005 on Wed May 31, 2017 6:21 am; edited 4 times in total
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Wed May 31, 2017 2:05 am    Post subject: Re: Durable subscriptions to a queue Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

simonalexander2005 wrote:

Is there any way to associate the qmgr (with a subscription ID of some kind, perhaps?) so that when it starts up again it receives the messages it's missed?

What is the definition of a DURABLE subscription...
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
simonalexander2005
PostPosted: Wed May 31, 2017 2:11 am    Post subject: Re: Durable subscriptions to a queue Reply with quote

Acolyte

Joined: 13 Jun 2016
Posts: 55

fjb_saper wrote:
simonalexander2005 wrote:

Is there any way to associate the qmgr (with a subscription ID of some kind, perhaps?) so that when it starts up again it receives the messages it's missed?

What is the definition of a DURABLE subscription...


Well, that's what I thought it was - one that holds the message until the subscriber comes back online... but I can't seem to get it to do it - it won't keep the message for when the queue manager comes back online.

All the documentation I've read talks about the subscriber being an application, rather than a queue on a clustered queue manager - could that have something to do with it?
Back to top
View user's profile Send private message
zpat
PostPosted: Wed May 31, 2017 4:13 am    Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5849
Location: UK

A retained publication holds the latest value published on that topic.

An administrative subscription will persist across QM restarts etc.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
simonalexander2005
PostPosted: Wed May 31, 2017 4:49 am    Post subject: Reply with quote

Acolyte

Joined: 13 Jun 2016
Posts: 55

zpat wrote:
A retained publication holds the latest value published on that topic.

An administrative subscription will persist across QM restarts etc.


Retained publication is something I haven't come across before, thank you, that was helpful.

The IBM documentation states that "To ensure that a publication is retained, use the MQPMO_RETAIN put-message option". Is this something I would need to set in IIB or on the Alias queue; or on the Topic? I'm assuming the IIB MQOutput Node, but I can't see any related option anywhere.

Also, it seems that it only keeps one message per topic string. Is there a way to keep all unpublished messages for a subscriber?
Back to top
View user's profile Send private message
zpat
PostPosted: Wed May 31, 2017 6:14 am    Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5849
Location: UK

There is no such thing as unpublished messages.

If you mean - can you keep all published messages until they are retrieved? Yes - use an administrative subscription with the destination of a permanent queue. They will stay in the queue until consumed (assuming they are persistent messages).

If you are publishing in IIB, you would normally use the publication node. You can set the various PMO options in ESQL.

However it's also possible and easier to use pub/sub without any application awareness, in the following way

Publish by putting to a topic alias queue (i.e. MQOUTPUT node).

Subscribe by having an admin sub feeding a queue and your consumer getting from that queue.

This way you will never miss any published messages.

You can have more than one subscription on the same topic (to different queues) and/or have more than one sub feeding the same queue.

I don't see a need for retained publications in this set up and it's probably not what you want.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
simonalexander2005
PostPosted: Wed May 31, 2017 6:17 am    Post subject: Reply with quote

Acolyte

Joined: 13 Jun 2016
Posts: 55

zpat wrote:
use an administrative subscription with the destination of a permanent queue.


I believe we are doing that, but the "permanent queue" is on a different queue manager in the cluster, which could mean the network connection fails, or the other queue manager goes down. What happens to the message in that case (assuming ALLAVAIL is set)?

edit - just checked, our layout is:
[Alias to Topic -> Subscription -> Alias queue] -> [Alias Queue (just one instance, but in cluster) -> Local Queue]

where [] denotes different queue managers in a cluster.

If I put a message on the "Alias queue" and the 2nd qmgr is down, it sits on the cluster transmit queue until it comes back up.

If I put a message on the "Alias to Topic" queue, it either just gets lost - or I can't see where it goes anyway; or it's not being created at all for the downed qmgr.


Last edited by simonalexander2005 on Wed May 31, 2017 6:35 am; edited 2 times in total
Back to top
View user's profile Send private message
mqjeff
PostPosted: Wed May 31, 2017 6:29 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

... if you are sending messages (whether published or not) to queues in a cluster, and one member of the cluster stops, then messages will not go to that qmgr... but instead be load balanced against the remaining queues.

If you're sending subscription messages to clustered queues, you may (likely will) get duplicate messages, as each active subscription will get a copy of every publication and load balance that across the cluster...
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
simonalexander2005
PostPosted: Wed May 31, 2017 6:30 am    Post subject: Reply with quote

Acolyte

Joined: 13 Jun 2016
Posts: 55

mqjeff wrote:
... if you are sending messages (whether published or not) to queues in a cluster, and one member of the cluster stops, then messages will not go to that qmgr... but instead be load balanced against the remaining queues.

If you're sending subscription messages to clustered queues, you may (likely will) get duplicate messages, as each active subscription will get a copy of every publication and load balance that across the cluster...


We've only got one instance of each queue in the cluster, so there's no "remaining queues" to be load-balanced against. Like I said in my edit to my previous post, it doesn't even seem to sit on the cluster xmit queue in this case
Back to top
View user's profile Send private message
mqjeff
PostPosted: Wed May 31, 2017 7:02 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

Does the administrative subscription exist on more than one qmgr? You say "Subscriptions are configured on QMGR0". If qmgr0 is the one that's down, then the subscription doesn't exist.
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
zpat
PostPosted: Wed May 31, 2017 7:06 am    Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5849
Location: UK

If your admin sub is defined on the same qmgr as the publication event, it will go to a local queue or a transmit queue.

I suggest you set the destination queue as a local alias queue which can refer to a clustered queue or remote queue definition as you wish.

There is also a concept of pub/sub clusters but you probably don't want to bother with these yet.

Use topic objects to control the publication behaviour.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
simonalexander2005
PostPosted: Thu Jun 01, 2017 1:08 am    Post subject: Reply with quote

Acolyte

Joined: 13 Jun 2016
Posts: 55

mqjeff wrote:
Does the administrative subscription exist on more than one qmgr? You say "Subscriptions are configured on QMGR0". If qmgr0 is the one that's down, then the subscription doesn't exist.


zpat wrote:
If your admin sub is defined on the same qmgr as the publication event, it will go to a local queue or a transmit queue.

I suggest you set the destination queue as a local alias queue which can refer to a clustered queue or remote queue definition as you wish.


I think I perhaps oversimplified our layout when I was trying to explain the issue. I'll try and give more detail:

Every server is in the same cluster.

Messages can come in to any one of two servers (call them GW1 and GW2). These servers use cluster queue priorities to route the incoming messages to any one of four IIB servers (IIB1-4). So each incoming message is just processed once, by one of the four IIB servers.

These four IIB servers are configured identically (apart from the q priorities), and each processes the incoming messages using IIB. IIB writes the messages to an alias queue on the same IIB server, which points to a Topic defined on the IIB server. There are then admin subscriptions (the same configuration on each IIB server) that publish the messages to an alias queue, again on the IIB server.

The alias queues that are local to the IIB servers point to Alias queues which are in the cluster on further MQ servers (OUT1 and OUT2, say). There's only one instance of each of those cluster queues (so, say, msgType1.TO.OUT1, msgType1.TO.OUT2, msgType2.TO.OUT1, ...)

With the subscriptions set to ALLAVAIL, if OUT1 is down then I want it to pick up all the missed messages when it comes back up. I assumed that because we have the local Alias Queues as a bridge it would do that, but it appears not. If I put messages directly onto those alias queues using MQ explorer then they make it through when it comes back up; but if I try and do it using the Alias Topics; it just seems to ignore that subscription completely.

Does that clarify the situation?

Thank you for your time and patience in walking through this with me.
Back to top
View user's profile Send private message
zpat
PostPosted: Thu Jun 01, 2017 1:37 am    Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5849
Location: UK

Alias queues cannot hold messages. If the base queue of the alias is unresolvable then the message cannot be put to it.

Subscriptions don't publish messages. Publishers publish and subscribers receive - the decoupled nature is important.

It might be safer to use a remote queue definition so that message delivery is always possible.

To be honest I have never tried what you are doing. You can't pick up "missed messages" from the pub/sub engine.

Either they are delivered to a queue (or to a programmatic subscriber) at the time of publication or they are not, (aside from the last one on a retained pub).

The pub/sub engine does not hold messages (other than the last one on a retained pub).

Pub/sub is not always suitable, using a routing flow in IIB to distribute messages can be a better option for more precise control.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
simonalexander2005
PostPosted: Thu Jun 01, 2017 1:51 am    Post subject: Reply with quote

Acolyte

Joined: 13 Jun 2016
Posts: 55

zpat wrote:
Alias queues cannot hold messages. If the base queue of the alias is unresolvable then the message cannot be put to it.


OK Yes I understand that; but if I put directly to the local alias queue, the message just sits on the xmit queue until the OUT1 server comes back up. Do you know why, if I use pub/sub to do exactly the same thing (subscription points to that same alias queue), it decides not to receive the message at all?

zpat wrote:
You can't pick up "missed messages" from the pub/sub engine.

That's a shame, I'll have to figure out another way around it then. Thank you again for your time
Back to top
View user's profile Send private message
zpat
PostPosted: Thu Jun 01, 2017 3:08 am    Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5849
Location: UK

simonalexander2005 wrote:

OK Yes I understand that; but if I put directly to the local alias queue, the message just sits on the xmit queue until the OUT1 server comes back up. Do you know why, if I use pub/sub to do exactly the same thing (subscription points to that same alias queue), it decides not to receive the message at all?


No, some IBM'er might.

Perhaps explorer has resolved the alias earlier and remembers it's a cluster queue, and pub/sub resolves in real-time and can't find it.

Try taking it out of the cluster, then starting explorer to put to the queue (or use another tool like RFHUTIL) - does it work then?
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » General IBM MQ Support » Durable subscriptions to a queue
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.