ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » IBM MQ Java / JMS » MQ Pub/Sub - Subscription Expiry and Queue Depth

Post new topic  Reply to topic
 MQ Pub/Sub - Subscription Expiry and Queue Depth « View previous topic :: View next topic » 
Author Message
kandarpar
PostPosted: Wed Dec 05, 2018 7:20 am    Post subject: MQ Pub/Sub - Subscription Expiry and Queue Depth Reply with quote

Newbie

Joined: 29 Jul 2010
Posts: 6

Hi, I have a scenario where -

I have a Subscription to a Topic with an expiry of 24 hrs.

I have a process that consumes events off of this topic as they are published.

all these are Durable subscriptions and Persistent.


let's say if the Consuming process can process a Million messages a day but the max expected events is 2 Million per day with subscription expiry set to 24 hrs, based on a simple math, do the rest of 2 Million expire and discarded(DLQed)? ( Assumption that all these 2 million messages are published around the same time)

also, if the Max Q Depth (the physical object for the Topic) is set to 1 Million, what would happen if we increase the Subscription expiry to say 72-96 hrs?

is there any impact on the Subscription messages if the current Q Depth exceeds the Maximum even if the Subscription expiry is set to say Unlimited?

Please let me know.

Thank You
-Ravi
Back to top
View user's profile Send private message
hughson
PostPosted: Wed Dec 05, 2018 8:50 pm    Post subject: Re: MQ Pub/Sub - Subscription Expiry and Queue Depth Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

kandarpar wrote:
Hi, I have a scenario where -

I have a Subscription to a Topic with an expiry of 24 hrs.

I have a process that consumes events off of this topic as they are published.

all these are Durable subscriptions and Persistent.


let's say if the Consuming process can process a Million messages a day but the max expected events is 2 Million per day with subscription expiry set to 24 hrs, based on a simple math, do the rest of 2 Million expire and discarded(DLQed)? ( Assumption that all these 2 million messages are published around the same time)


If all these 2 million messages are published around the same time and have successfully been put (by the pub/sub engine in the queue manager) onto the subscriber queue, but the subscription application cannot get through more than 1 million of them before the subscription expires then when the subscription is removed by expiry processing, so is it's queue and messages (assuming a managed queue which is the default for JMS and this question is in the JMS forum). If you use your own queue, then the messages are not touched when the subscription expires.

kandarpar wrote:
also, if the Max Q Depth (the physical object for the Topic) is set to 1 Million, what would happen if we increase the Subscription expiry to say 72-96 hrs?

If the Max Queue Depth is not big enough to hold the 2 million messages that were published, the messages will be DLQed. This is controlled by the topic object attribute PMSGDLV, which by default is set to ALLDUR. So persistent messages sent to a Durable subscription (your scenario) must be DLQed if they cannot be given to the subscriber queue, and if that also fails, the publisher will receive MQRC_PUBLICATION_FAILURE (2502).

If you have a DLQ-handler, then those DLQed messages could eventually make it onto the subscriber queue, and with the extended expiry it might eventually get to process them.

kandarpar wrote:
is there any impact on the Subscription messages if the current Q Depth exceeds the Maximum even if the Subscription expiry is set to say Unlimited?


The current Queue Depth cannot exceed the Maximum Queue Depth - that's what a maximum means! Any messages beyond the maximum will be DLQed or if they cannot be put there either, will cause a publisher failure.

Your application design needs to decide whether it wants to process all the messages, or only be around for 24 hours. It would seem that you cannot do both.

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
kandarpar
PostPosted: Thu Dec 06, 2018 7:26 am    Post subject: MQ Pub/Sub - Subscription Expiry and Queue Depth Reply with quote

Newbie

Joined: 29 Jul 2010
Posts: 6

Hi Morag,

Thank You for the prompt reply. really helpful.

A follow up question is -


if there are no constraints on the time by which these events are required to be processed,

do you think increasing the subscription expiry to more than the current set limit, and also increasing the Max Q Depth to make sure the Queues hold all the messages before these can be processed by the subscribing application would be a possible solution?


if it is a possible to increase the subscription expiry and increase the MAX QDepth, do you see any implications of it on the storage or performance on the MQ PUB/SUB Broker?

Thank You and much appreciated.

-Ravi
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Thu Dec 06, 2018 9:02 am    Post subject: Re: MQ Pub/Sub - Subscription Expiry and Queue Depth Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20695
Location: LI,NY

kandarpar wrote:
Hi Morag,

Thank You for the prompt reply. really helpful.

A follow up question is -


if there are no constraints on the time by which these events are required to be processed,

do you think increasing the subscription expiry to more than the current set limit, and also increasing the Max Q Depth to make sure the Queues hold all the messages before these can be processed by the subscribing application would be a possible solution?


if it is a possible to increase the subscription expiry and increase the MAX QDepth, do you see any implications of it on the storage or performance on the MQ PUB/SUB Broker?

Thank You and much appreciated.

-Ravi

I feel you're asking the wrong questions here.
The questions to ask are:
  1. do the published messages have message affinitiy i.e. do they need to be consumed in order?
  2. Can you run multiple consumer instances
  3. Can you consume all the messages from 24 hours in 24 hours or less.


Take a very good look at question 3. Because if the answer is no, you can forget about increasing space and queue depth. It just won't help.

Have fun
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
kandarpar
PostPosted: Thu Dec 06, 2018 9:58 am    Post subject: MQ Pub/Sub - Subscription Expiry and Queue Depth Reply with quote

Newbie

Joined: 29 Jul 2010
Posts: 6

answers to questions -

1.There is no message affinity and don't have to be processed in order

2. Yes. There are multiple consumer instances but those multiple instances together consume all these 1 Million messages in a 24 hr. timeframe

3. currently these messages arriving in 24 hrs cannot be processed with multiple consumer instances in a 24 hr time frame - with Subscription expiry set to 24 hrs, these subscriptions are getting expired.

there is no continuous stream of millions of messages getting published on successive days in a week, only on one day in a week.

so checking to see if increasing the subscription expiry helps solve this?

Thanks
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Thu Dec 06, 2018 1:07 pm    Post subject: Re: MQ Pub/Sub - Subscription Expiry and Queue Depth Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20695
Location: LI,NY

kandarpar wrote:
answers to questions -

1.There is no message affinity and don't have to be processed in order

2. Yes. There are multiple consumer instances but those multiple instances together consume all these 1 Million messages in a 24 hr. timeframe

3. currently these messages arriving in 24 hrs cannot be processed with multiple consumer instances in a 24 hr time frame - with Subscription expiry set to 24 hrs, these subscriptions are getting expired.

there is no continuous stream of millions of messages getting published on successive days in a week, only on one day in a week.

so checking to see if increasing the subscription expiry helps solve this?

Thanks

So what you're telling us is that once a week you publish 1 Million msgs.
You have 7 days to consume them.
Make sure your queue depth and log space are adequate
Ensure that the messages are consumed as soon as possible

I trust it doesn't take 3 days to consume them all... So yes an increased subscription expiry would help...

Have fun
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
hughson
PostPosted: Thu Dec 06, 2018 1:34 pm    Post subject: Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

I would also question what your intent in using Subscription Expiry is?
  • Do you not trust that the application will remove the subscription when it is done?
  • Does the same subscription get made the following week for the next 1 million messages? Why not just leave the previous one?
  • Why are you not simply letting the subscription stay around - the expiry is clearly complicating the issue you have
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
kandarpar
PostPosted: Thu Dec 06, 2018 1:53 pm    Post subject: MQ Pub/Sub - Subscription Expiry and Queue Depth Reply with quote

Newbie

Joined: 29 Jul 2010
Posts: 6

The application would remove the subscription when it is done.


The question is when any of the consuming application's instances cannot consume a message before the configured subscription expiry time expires ?

when you say "letting the subscription stay around" until it is picked up by one of the consumer instances - what to do you recommend to leave the subscription stay around until it is picked by one of the consuming application's instances?


Thanks
Back to top
View user's profile Send private message
hughson
PostPosted: Thu Dec 06, 2018 7:44 pm    Post subject: Re: MQ Pub/Sub - Subscription Expiry and Queue Depth Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

kandarpar wrote:
The application would remove the subscription when it is done.
Suggest you consider removing expiry from the equation then. It appears to be buying you nothing but complication.

kandarpar wrote:
The question is when any of the consuming application's instances cannot consume a message before the configured subscription expiry time expires ?
Like I said, complication.

kandarpar wrote:
when you say "letting the subscription stay around" until it is picked up by one of the consumer instances - what to do you recommend to leave the subscription stay around until it is picked by one of the consuming application's instances?
I'm not sure what you mean by "picked up by one of the consumer instances? I just suggest you leave the subscription around until all the messages have been consumed, and then the application can remove it. Perhaps there are more complications in your environment that you haven't described to us, hence my original question asking why you were using expiry at all.

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » IBM MQ Java / JMS » MQ Pub/Sub - Subscription Expiry and Queue Depth
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.