| Author |
Message
|
| PeterPotkay |
 Posted: Tue Jul 15, 2008 3:00 pm Post subject: Runaway Publisher |
 |
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
MQ 6.0
Publisher and Subscribers both connected to the same Queue Manager.
Publisher puts to StreamQueue
Subscriber gets from SubQueue
And the Pub Sub Broker sits in between them.
If SubQueue fills up, then messages immediately started to be put to the QM's DLQ by the Pub/Sub Broker. Obviously this is a problem if the Publisher is going nuts. The DLQ will fill up if you are lucky and I assume the Broker will then start getting errors. Or if unlucky the DLQ may grow so big your hard drive fills. Then things really go bad. Even the DLQ filling up without the disk can be problematic. Luckily I saw this happening and I Put Inhibited the stream queue.
Is there anyway to make the Pub Sub Broker realize that the Subscriber Q is full, and not accept anymore Publications to that queue? Let the Publisher get an error instead of my DLQ filling?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Tue Jul 15, 2008 4:39 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Of course, you can have a DLQ Handler generate a dereg subscription message.
But pretend that the subscriber queue was at a remote qmgr at a different institution... what would you want the publisher to do in that case? |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue Jul 15, 2008 4:58 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
If I couldn't get a hold of support at the other institution, I would stop the RCVR channel.
Actually, I have my RCVR's Message Retry Interval set to 1000 and Message Retry Time set to 1, so the messages would go to my DLQ 1 a second, giving me plenty of time to react before the DLQ got too deep. But I don't have that luxury with the local Pub Sub Broker. It will dump the messages to the DLQ as fast as the publisher feeds them in.
There's gotta be a way to tell the Pub Sub Broker to not blindly accept an unlimited amount of publications. Right? 
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Jul 15, 2008 8:33 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Peter,
I would have thought that you set it up so that each subscriber supplies it's own subscription queue.
Advantages are that in case you have a "dead" subscriber it is way easier to identify, and it should not disrupt any other subscribers....
Why are you trying to link the publisher to the subscriber (dead subscriber) as the model cleary separates them?
If there is something wrong with the publisher (too many msgs in too short a time) clearly you would have to come up for a solution at the publisher end.
By the way if your messages have no affinity you could start multiple instances of the subscriber to consume the messages faster. Only one of those instances is subscribed...
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| AkankshA |
| Posted: Tue Jul 15, 2008 9:13 pm Post subject: Re: Runaway Publisher |
|
|
Grand Master
Joined: 12 Jan 2006
Posts: 1494
Location: Singapore
|
| PeterPotkay wrote: |
| Is there anyway to make the Pub Sub Broker realize that the Subscriber Q is full, and not accept anymore Publications to that queue? |
Won't this defeat the very pupose of pub sub - to keep publishers unaware of subscribers and vice versa...
| PeterPotkay wrote: |
Let the Publisher get an error instead of my DLQ filling? |
well... as mqjeff said.. do something customised... or may be u can have a look at your topic tree
even MB does not have a solution for this kinda requirement..
_________________
Cheers |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jul 16, 2008 5:29 am Post subject: Re: Runaway Publisher |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| AkankshA wrote: |
| PeterPotkay wrote: |
| Is there anyway to make the Pub Sub Broker realize that the Subscriber Q is full, and not accept anymore Publications to that queue? |
Won't this defeat the very pupose of pub sub - to keep publishers unaware of subscribers and vice versa...
|
That's a good point, but I don't want the Publisher to be aware of the Subscriber. Obviously the Publisher and we realize that you can't publish 1,000,000,000,000,000,000 messages. We all know that it is possible that underlying resources fill up. If the Subscriber Q fills up (I do have one for each subscriber), then the DQL fills up. If the DLQ fill sup, I assume the Input stream queue fills up as the Pub Sub Broker can no longer take those messages in. Once the stream q is full, I assume finally then the Publisher will get an error on its MQPUT.
My point is why should the point of failure be so far "up front". By that time the DLQ has filled and a common resource, the DLQ, has been compromised. Now the whole QM has "lost" its DLQ and we will have negative impacts should a channel need to put to it.
OK, I guess in a way a Pub Sub Broker is just like a channel from a remote Queue Manager. In both cases the intermediate putter (Broker or channel) are coded to put to the DLQ if the destination q is N/A. We all know that channels are just coded this way and there is no way around it. It has to be this way due to the very asynchronous nature between 2 QMs.
Put if the publisher and the broker and the subscriber are all on the the same QM, wouldn't it be beneficial to have the point of failure be limited to the Publisher and Sunscriber, instead of taking out the entire DLQ and then the entire stream queue?
At least with QM to QM channels I have the option of throttling how fast the messages go to the DLQ by using the RCVR channel's Message Retry parms. But in the case of the Pub Sub Broker I'm at the mercy of the Publisher.
Remember, I'm lookling for a solution that will protect the shared QM at 3 AM when the publisher really goes nuts. Any solutions requiring human intervention may be to slow.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| AkankshA |
| Posted: Wed Jul 16, 2008 6:02 am Post subject: Re: Runaway Publisher |
|
|
Grand Master
Joined: 12 Jan 2006
Posts: 1494
Location: Singapore
|
| PeterPotkay wrote: |
That's a good point, but I don't want the Publisher to be aware of the Subscriber. Obviously the Publisher and we realize that you can't publish 1,000,000,000,000,000,000 messages. We all know that it is possible that underlying resources fill up. If the Subscriber Q fills up (I do have one for each subscriber), then the DQL fills up. If the DLQ fill sup, I assume the Input stream queue fills up as the Pub Sub Broker can no longer take those messages in. Once the stream q is full, I assume finally then the Publisher will get an error on its MQPUT.
|
I doubt that.... if the broker is unable to find subscriber queue and DLQ is not available then it shall discard it.. Got to test this as this is how i expect it to behave..
Well i completely agree to your point from the perspective of an MQ user but not probably pub sub architects had something else in mind.. they must have thought of pub sub so that irrespective of weather subscriber is sleeping or dead, publisher should not get impacted.. somewhat on the grounds of asynchronous messageing...
_________________
Cheers |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jul 16, 2008 9:44 am Post subject: Re: Runaway Publisher |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| AkankshA wrote: |
| PeterPotkay wrote: |
That's a good point, but I don't want the Publisher to be aware of the Subscriber. Obviously the Publisher and we realize that you can't publish 1,000,000,000,000,000,000 messages. We all know that it is possible that underlying resources fill up. If the Subscriber Q fills up (I do have one for each subscriber), then the DQL fills up. If the DLQ fill sup, I assume the Input stream queue fills up as the Pub Sub Broker can no longer take those messages in. Once the stream q is full, I assume finally then the Publisher will get an error on its MQPUT.
|
I doubt that.... if the broker is unable to find subscriber queue and DLQ is not available then it shall discard it.. Got to test this as this is how i expect it to behave..
|
You probably had your Broker properties set to discard non persistent publications.
http://publib.boulder.ibm.com/infocenter/wmqv6/v6r0/topic/com.ibm.mq.amqnar.doc/amq622k.htm
I'm looking for ideas when it comes to persistent publications. The broker won't discard those.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jul 16, 2008 2:30 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Peter,
Stop looking at pub / sub as a delivery between 2 endpoints. Imagine multiple endpoints: If the publisher goes crazy but 3 out of 4 subscriber can maintain pace... should the one that cannot penalize everybody else?
What you can do is determine what symptoms are manifested by the provider goes crazy. Then put some alert on this, inhibit the corresponding stream queue and page the corresponding application teams to resolve the issue. 04:00 AM disaster resolved.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Jul 17, 2008 6:18 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| fjb_saper wrote: |
Peter,
Stop looking at pub / sub as a delivery between 2 endpoints. Imagine multiple endpoints: If the publisher goes crazy but 3 out of 4 subscriber can maintain pace... should the one that cannot penalize everybody else?
|
Excellent point, I didn't think of the situation with more than 1 subscriber.
| fjb_saper wrote: |
| What you can do is determine what symptoms are manifested by the provider goes crazy. Then put some alert on this, inhibit the corresponding stream queue and page the corresponding application teams to resolve the issue. 04:00 AM disaster resolved. |
But by inhibiting the stream queue I guess I just penalized Subscribers #2-4 just because #1 allowed its queue to fill.
No good answer methinks. Make sure the DLQ is big, make sure there is plenty of disk space, alert when the subscriber q is starting to fill up not when its full, and hope the runaway publisher doesn't put too many messages that are too big too fast in the middle of the night . Its a Denial of Service attack in a way, and its just as likely in point to point as it is with Pub Sub.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mlr |
| Posted: Thu Jul 17, 2008 10:52 pm Post subject: |
|
|
Novice
Joined: 29 Jun 2008
Posts: 13
|
Broker qm.ini parameters:
| Quote: |
DLQNonPersistentPublication=yes|no
If the broker fails to send a nonpersistent publication to a subscriber, this attribute indicates whether the broker should put the publication to the dead-letter queue.
The default is DLQNonPersistentPublication=yes.
DiscardNonPersistentPublication=yes|no
If the broker fails to send a nonpersistent publication to a subscriber and cannot write the publication to the dead-letter queue, this attribute indicates whether the broker can discard the publication.
The default is:
DiscardNonPersistentPublication=no if SyncPointIfPersistent=no.
DiscardNonPersistentPublication=yes if SyncPointIfPersistent=yes.
Note:
If SyncPointIfPersistent=yes is set, DiscardNonPersistentPublication=no must not be set.
|
|
|
| Back to top |
|
|
|
|
