ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General Discussion » Selectively Prevent Flooding

Post new topic  Reply to topic
 Selectively Prevent Flooding « View previous topic :: View next topic » 
Author Message
Jeff.VT
PostPosted: Tue Jul 09, 2019 8:11 am    Post subject: Selectively Prevent Flooding Reply with quote

Acolyte

Joined: 02 Mar 2017
Posts: 68

I have a 3rd party that keeps complaining that every time our connection goes down, that we flood them with messages when the connectivity is restored, which causes a feedback loop that perpetuates a constant downtime until somebody intervenes.

I don't know the specifics of their configuration, (But IBM is contracted directly and built it for them, so I probably can't blame dumb architecture too much).

Now I send this 3rd party two types of messages.

TYPE 1, has a super high volume, and requires a response from them to complete the transaction. If that response takes >20 seconds to receive, my application stops looking for the response. So messages >20 seconds are garbage for me. (92% of them take less than 200 ms for me to get a response, and >99.96% of them have a total turnaround time of <5 seconds)

TYPE 2, goes over the same channel & xmitq, has a much smaller volume, but does not require a response, and is presumed that if I send one, that they got it.

-----------

MY solution to this was to add a 20.1 second CAPEXPIRY to the alias queue sending the TYPE 1 messages. I tell them about it, and they tell me to back it out. Apparently they have a system where the message on their side sort of hops from station to station, doing things along the way. And they say that it would be a problem for them if it does its thing at station 1, only to self-destruct at station 2.

I can't just add something to clear out messages from the XMITQ if they back up because that would wipe out TYPE 2 messages as well.

Is there anything else you guys could think of? Is there any (easy) way to... I dunno... remove an expiry as it passes through the channel? Or I dunno... be more selective in purging the XMITQ... I suppose I'm asking because I can't think of too many options, and thought you guys might have an idea.
Back to top
View user's profile Send private message
markt
PostPosted: Tue Jul 09, 2019 8:24 am    Post subject: Reply with quote

Knight

Joined: 14 May 2002
Posts: 502

Why are you using the same channel/xmitq? Separating classes of traffic is very commonly done using multiple routes between the same qmgrs
Back to top
View user's profile Send private message
Jeff.VT
PostPosted: Tue Jul 09, 2019 8:26 am    Post subject: Reply with quote

Acolyte

Joined: 02 Mar 2017
Posts: 68

Just thought of a weird work-around...

I have a script that watches for when 3rd party channels are down, and will adjust clustered alias queues to have a lower priority when they're down (so not-down alternative routes are prioritized during these downtimes).

I can make a 'decoy' cluster alias queue on another queue manager. The priority would be lower when the real connection is up, but higher when its down... That way when the connection has downtime, all messages are sent to be destroyed until connectivity is restored. Rather than waiting on the XMITQ to flood the 3rd party.

It probably won't solve the whole problem, but it might prevent some long downtimes, and get them off my customer's back for a bit.
Back to top
View user's profile Send private message
Jeff.VT
PostPosted: Tue Jul 09, 2019 8:28 am    Post subject: Reply with quote

Acolyte

Joined: 02 Mar 2017
Posts: 68

markt wrote:
Why are you using the same channel/xmitq? Separating classes of traffic is very commonly done using multiple routes between the same qmgrs


It's configured like that for a lot of the connections & message types. But not for these. And to be honest, that configuration is rarely my decision, it's usually the decision of the 3rd party how we connect to them (when I say 3rd party, I'm generally talking about Governments around the world - and I don't get to tell governments how to architect their system)
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Tue Jul 09, 2019 4:17 pm    Post subject: Re: Selectively Prevent Flooding Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2492
Location: Melbourne, Australia

Jeff.VT wrote:
I have a 3rd party that keeps complaining that every time our connection goes down, that we flood them with messages when the connectivity is restored, which causes a feedback loop that perpetuates a constant downtime until somebody intervenes.....

It seems that the real problems are the reliability of the n/w connection and the 3rd party app(s) design. They should be the focus of attention, particularly for high volume messaging. Don't apply band-aids, fix the root causes.

MQ is behaving normally. When the channel starts (on a retry), all the messages on the xmitq are sent through to the 3rd pary as soon as possible.
Do you also receive a "flood" of response messages, with nearly all of them beyond your 20 second QoS time?
_________________
Glenn
Back to top
View user's profile Send private message
Jeff.VT
PostPosted: Wed Jul 10, 2019 7:10 am    Post subject: Re: Selectively Prevent Flooding Reply with quote

Acolyte

Joined: 02 Mar 2017
Posts: 68

gbaddeley wrote:
Do you also receive a "flood" of response messages, with nearly all of them beyond your 20 second QoS time?


Ya. They send back a response for every message they receive, regardless of how over that QoS time it is... TO BE FAIR, they didn't come up with that 20 seconds limit, I did by looking at how long they generally take and adding in a reasonable buffer.

But it seems like they designed the system in a way where they assume our clients' customers would wait an infinite amount of time for a response, which is ludicrous.

Anyway - thanks for the responses. I agree, that it seems like it needs to be incumbent on them for most any solution. And if I'm being honest, this government shouldn't have been shocked by this message volume. It's not like this number is some unknown quantity, it's a well-researched number that appears in any number of government reports. They should have prepared for it properly.
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Jul 10, 2019 7:43 am    Post subject: Re: Selectively Prevent Flooding Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

Jeff.VT wrote:
this government shouldn't have been shocked


Jeff.VT wrote:
They should have prepared for it properly.


Welcome to every government project I've ever been associated with, irrespective of the nationality / politics of the government involved, and a majority of the large organizational projects I've worked on. Enjoy your stay.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
RogerLacroix
PostPosted: Wed Jul 10, 2019 9:26 am    Post subject: Reply with quote

Jedi Knight

Joined: 15 May 2001
Posts: 3252
Location: London, ON Canada

<Vendor_Plug>
Well, if you want to control the "flood" of messages going to the government then have a look at MQ Channel Throttler. I created it, based on another customer's very similar issue.
</Vendor_Plug>

Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter
Back to top
View user's profile Send private message Visit poster's website
PeterPotkay
PostPosted: Wed Jul 10, 2019 4:47 pm    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

Jeff.VT, what does the other side want to happen when the connection is restored and there is a large backlog of messages waiting on your side?

Do they want them all, just at some slower pace than MQ and the restored network link flood them over?
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Jul 10, 2019 5:03 pm    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

I support MQ on z/OS and midrange. This "flooding" is quite common where the hugely provisioned mainframe MQ sender channel(s) overwhelms the puny midrange MQ. While I'm sympathetic ...
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Jul 11, 2019 4:09 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

bruce2359 wrote:
I support MQ on z/OS and midrange. This "flooding" is quite common where the hugely provisioned mainframe MQ sender channel(s) overwhelms the puny midrange MQ. While I'm sympathetic ...





_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » General Discussion » Selectively Prevent Flooding
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.