| Author |
Message
|
| kandarpar |
 Posted: Wed Dec 05, 2018 7:20 am Post subject: MQ Pub/Sub - Subscription Expiry and Queue Depth |
 |
|
Newbie
Joined: 29 Jul 2010
Posts: 6
|
Hi, I have a scenario where -
I have a Subscription to a Topic with an expiry of 24 hrs.
I have a process that consumes events off of this topic as they are published.
all these are Durable subscriptions and Persistent.
let's say if the Consuming process can process a Million messages a day but the max expected events is 2 Million per day with subscription expiry set to 24 hrs, based on a simple math, do the rest of 2 Million expire and discarded(DLQed)? ( Assumption that all these 2 million messages are published around the same time)
also, if the Max Q Depth (the physical object for the Topic) is set to 1 Million, what would happen if we increase the Subscription expiry to say 72-96 hrs?
is there any impact on the Subscription messages if the current Q Depth exceeds the Maximum even if the Subscription expiry is set to say Unlimited?
Please let me know.
Thank You
-Ravi |
|
| Back to top |
|
 |
| hughson |
| Posted: Wed Dec 05, 2018 8:50 pm Post subject: Re: MQ Pub/Sub - Subscription Expiry and Queue Depth |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
| kandarpar wrote: |
Hi, I have a scenario where -
I have a Subscription to a Topic with an expiry of 24 hrs.
I have a process that consumes events off of this topic as they are published.
all these are Durable subscriptions and Persistent.
let's say if the Consuming process can process a Million messages a day but the max expected events is 2 Million per day with subscription expiry set to 24 hrs, based on a simple math, do the rest of 2 Million expire and discarded(DLQed)? ( Assumption that all these 2 million messages are published around the same time) |
If all these 2 million messages are published around the same time and have successfully been put (by the pub/sub engine in the queue manager) onto the subscriber queue, but the subscription application cannot get through more than 1 million of them before the subscription expires then when the subscription is removed by expiry processing, so is it's queue and messages (assuming a managed queue which is the default for JMS and this question is in the JMS forum). If you use your own queue, then the messages are not touched when the subscription expires.
| kandarpar wrote: |
| also, if the Max Q Depth (the physical object for the Topic) is set to 1 Million, what would happen if we increase the Subscription expiry to say 72-96 hrs? |
If the Max Queue Depth is not big enough to hold the 2 million messages that were published, the messages will be DLQed. This is controlled by the topic object attribute PMSGDLV, which by default is set to ALLDUR. So persistent messages sent to a Durable subscription (your scenario) must be DLQed if they cannot be given to the subscriber queue, and if that also fails, the publisher will receive MQRC_PUBLICATION_FAILURE (2502).
If you have a DLQ-handler, then those DLQed messages could eventually make it onto the subscriber queue, and with the extended expiry it might eventually get to process them.
| kandarpar wrote: |
| is there any impact on the Subscription messages if the current Q Depth exceeds the Maximum even if the Subscription expiry is set to say Unlimited? |
The current Queue Depth cannot exceed the Maximum Queue Depth - that's what a maximum means! Any messages beyond the maximum will be DLQed or if they cannot be put there either, will cause a publisher failure.
Your application design needs to decide whether it wants to process all the messages, or only be around for 24 hours. It would seem that you cannot do both.
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| kandarpar |
| Posted: Thu Dec 06, 2018 7:26 am Post subject: MQ Pub/Sub - Subscription Expiry and Queue Depth |
|
|
Newbie
Joined: 29 Jul 2010
Posts: 6
|
Hi Morag,
Thank You for the prompt reply. really helpful.
A follow up question is -
if there are no constraints on the time by which these events are required to be processed,
do you think increasing the subscription expiry to more than the current set limit, and also increasing the Max Q Depth to make sure the Queues hold all the messages before these can be processed by the subscribing application would be a possible solution?
if it is a possible to increase the subscription expiry and increase the MAX QDepth, do you see any implications of it on the storage or performance on the MQ PUB/SUB Broker?
Thank You and much appreciated.
-Ravi |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Dec 06, 2018 9:02 am Post subject: Re: MQ Pub/Sub - Subscription Expiry and Queue Depth |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| kandarpar wrote: |
Hi Morag,
Thank You for the prompt reply. really helpful.
A follow up question is -
if there are no constraints on the time by which these events are required to be processed,
do you think increasing the subscription expiry to more than the current set limit, and also increasing the Max Q Depth to make sure the Queues hold all the messages before these can be processed by the subscribing application would be a possible solution?
if it is a possible to increase the subscription expiry and increase the MAX QDepth, do you see any implications of it on the storage or performance on the MQ PUB/SUB Broker?
Thank You and much appreciated.
-Ravi |
I feel you're asking the wrong questions here.
The questions to ask are:
- do the published messages have message affinitiy i.e. do they need to be consumed in order?
- Can you run multiple consumer instances
- Can you consume all the messages from 24 hours in 24 hours or less.
Take a very good look at question 3. Because if the answer is no, you can forget about increasing space and queue depth. It just won't help.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| kandarpar |
| Posted: Thu Dec 06, 2018 9:58 am Post subject: MQ Pub/Sub - Subscription Expiry and Queue Depth |
|
|
Newbie
Joined: 29 Jul 2010
Posts: 6
|
answers to questions -
1.There is no message affinity and don't have to be processed in order
2. Yes. There are multiple consumer instances but those multiple instances together consume all these 1 Million messages in a 24 hr. timeframe
3. currently these messages arriving in 24 hrs cannot be processed with multiple consumer instances in a 24 hr time frame - with Subscription expiry set to 24 hrs, these subscriptions are getting expired.
there is no continuous stream of millions of messages getting published on successive days in a week, only on one day in a week.
so checking to see if increasing the subscription expiry helps solve this?
Thanks |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Dec 06, 2018 1:07 pm Post subject: Re: MQ Pub/Sub - Subscription Expiry and Queue Depth |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| kandarpar wrote: |
answers to questions -
1.There is no message affinity and don't have to be processed in order
2. Yes. There are multiple consumer instances but those multiple instances together consume all these 1 Million messages in a 24 hr. timeframe
3. currently these messages arriving in 24 hrs cannot be processed with multiple consumer instances in a 24 hr time frame - with Subscription expiry set to 24 hrs, these subscriptions are getting expired.
there is no continuous stream of millions of messages getting published on successive days in a week, only on one day in a week.
so checking to see if increasing the subscription expiry helps solve this?
Thanks |
So what you're telling us is that once a week you publish 1 Million msgs.
You have 7 days to consume them.
Make sure your queue depth and log space are adequate
Ensure that the messages are consumed as soon as possible
I trust it doesn't take 3 days to consume them all... So yes an increased subscription expiry would help...
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| hughson |
| Posted: Thu Dec 06, 2018 1:34 pm Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
I would also question what your intent in using Subscription Expiry is?
- Do you not trust that the application will remove the subscription when it is done?
- Does the same subscription get made the following week for the next 1 million messages? Why not just leave the previous one?
- Why are you not simply letting the subscription stay around - the expiry is clearly complicating the issue you have
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| kandarpar |
| Posted: Thu Dec 06, 2018 1:53 pm Post subject: MQ Pub/Sub - Subscription Expiry and Queue Depth |
|
|
Newbie
Joined: 29 Jul 2010
Posts: 6
|
The application would remove the subscription when it is done.
The question is when any of the consuming application's instances cannot consume a message before the configured subscription expiry time expires ?
when you say "letting the subscription stay around" until it is picked up by one of the consumer instances - what to do you recommend to leave the subscription stay around until it is picked by one of the consuming application's instances?
Thanks |
|
| Back to top |
|
|
| hughson |
| Posted: Thu Dec 06, 2018 7:44 pm Post subject: Re: MQ Pub/Sub - Subscription Expiry and Queue Depth |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
| kandarpar wrote: |
| The application would remove the subscription when it is done. |
Suggest you consider removing expiry from the equation then. It appears to be buying you nothing but complication.
| kandarpar wrote: |
| The question is when any of the consuming application's instances cannot consume a message before the configured subscription expiry time expires ? |
Like I said, complication.
| kandarpar wrote: |
| when you say "letting the subscription stay around" until it is picked up by one of the consumer instances - what to do you recommend to leave the subscription stay around until it is picked by one of the consuming application's instances? |
I'm not sure what you mean by "picked up by one of the consumer instances? I just suggest you leave the subscription around until all the messages have been consumed, and then the application can remove it. Perhaps there are more complications in your environment that you haven't described to us, hence my original question asking why you were using expiry at all.
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
|
|
