Author |
Message
|
Jeff.VT |
Posted: Tue Jul 09, 2019 8:11 am Post subject: Selectively Prevent Flooding |
|
|
Acolyte
Joined: 02 Mar 2017 Posts: 71
|
I have a 3rd party that keeps complaining that every time our connection goes down, that we flood them with messages when the connectivity is restored, which causes a feedback loop that perpetuates a constant downtime until somebody intervenes.
I don't know the specifics of their configuration, (But IBM is contracted directly and built it for them, so I probably can't blame dumb architecture too much).
Now I send this 3rd party two types of messages.
TYPE 1, has a super high volume, and requires a response from them to complete the transaction. If that response takes >20 seconds to receive, my application stops looking for the response. So messages >20 seconds are garbage for me. (92% of them take less than 200 ms for me to get a response, and >99.96% of them have a total turnaround time of <5 seconds)
TYPE 2, goes over the same channel & xmitq, has a much smaller volume, but does not require a response, and is presumed that if I send one, that they got it.
-----------
MY solution to this was to add a 20.1 second CAPEXPIRY to the alias queue sending the TYPE 1 messages. I tell them about it, and they tell me to back it out. Apparently they have a system where the message on their side sort of hops from station to station, doing things along the way. And they say that it would be a problem for them if it does its thing at station 1, only to self-destruct at station 2.
I can't just add something to clear out messages from the XMITQ if they back up because that would wipe out TYPE 2 messages as well.
Is there anything else you guys could think of? Is there any (easy) way to... I dunno... remove an expiry as it passes through the channel? Or I dunno... be more selective in purging the XMITQ... I suppose I'm asking because I can't think of too many options, and thought you guys might have an idea. |
|
Back to top |
|
|
markt |
Posted: Tue Jul 09, 2019 8:24 am Post subject: |
|
|
Knight
Joined: 14 May 2002 Posts: 508
|
Why are you using the same channel/xmitq? Separating classes of traffic is very commonly done using multiple routes between the same qmgrs |
|
Back to top |
|
|
Jeff.VT |
Posted: Tue Jul 09, 2019 8:26 am Post subject: |
|
|
Acolyte
Joined: 02 Mar 2017 Posts: 71
|
Just thought of a weird work-around...
I have a script that watches for when 3rd party channels are down, and will adjust clustered alias queues to have a lower priority when they're down (so not-down alternative routes are prioritized during these downtimes).
I can make a 'decoy' cluster alias queue on another queue manager. The priority would be lower when the real connection is up, but higher when its down... That way when the connection has downtime, all messages are sent to be destroyed until connectivity is restored. Rather than waiting on the XMITQ to flood the 3rd party.
It probably won't solve the whole problem, but it might prevent some long downtimes, and get them off my customer's back for a bit. |
|
Back to top |
|
|
Jeff.VT |
Posted: Tue Jul 09, 2019 8:28 am Post subject: |
|
|
Acolyte
Joined: 02 Mar 2017 Posts: 71
|
markt wrote: |
Why are you using the same channel/xmitq? Separating classes of traffic is very commonly done using multiple routes between the same qmgrs |
It's configured like that for a lot of the connections & message types. But not for these. And to be honest, that configuration is rarely my decision, it's usually the decision of the 3rd party how we connect to them (when I say 3rd party, I'm generally talking about Governments around the world - and I don't get to tell governments how to architect their system) |
|
Back to top |
|
|
gbaddeley |
Posted: Tue Jul 09, 2019 4:17 pm Post subject: Re: Selectively Prevent Flooding |
|
|
Jedi Knight
Joined: 25 Mar 2003 Posts: 2527 Location: Melbourne, Australia
|
Jeff.VT wrote: |
I have a 3rd party that keeps complaining that every time our connection goes down, that we flood them with messages when the connectivity is restored, which causes a feedback loop that perpetuates a constant downtime until somebody intervenes..... |
It seems that the real problems are the reliability of the n/w connection and the 3rd party app(s) design. They should be the focus of attention, particularly for high volume messaging. Don't apply band-aids, fix the root causes.
MQ is behaving normally. When the channel starts (on a retry), all the messages on the xmitq are sent through to the 3rd pary as soon as possible.
Do you also receive a "flood" of response messages, with nearly all of them beyond your 20 second QoS time? _________________ Glenn |
|
Back to top |
|
|
Jeff.VT |
Posted: Wed Jul 10, 2019 7:10 am Post subject: Re: Selectively Prevent Flooding |
|
|
Acolyte
Joined: 02 Mar 2017 Posts: 71
|
gbaddeley wrote: |
Do you also receive a "flood" of response messages, with nearly all of them beyond your 20 second QoS time? |
Ya. They send back a response for every message they receive, regardless of how over that QoS time it is... TO BE FAIR, they didn't come up with that 20 seconds limit, I did by looking at how long they generally take and adding in a reasonable buffer.
But it seems like they designed the system in a way where they assume our clients' customers would wait an infinite amount of time for a response, which is ludicrous.
Anyway - thanks for the responses. I agree, that it seems like it needs to be incumbent on them for most any solution. And if I'm being honest, this government shouldn't have been shocked by this message volume. It's not like this number is some unknown quantity, it's a well-researched number that appears in any number of government reports. They should have prepared for it properly. |
|
Back to top |
|
|
Vitor |
Posted: Wed Jul 10, 2019 7:43 am Post subject: Re: Selectively Prevent Flooding |
|
|
Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
Jeff.VT wrote: |
this government shouldn't have been shocked |
Jeff.VT wrote: |
They should have prepared for it properly. |
Welcome to every government project I've ever been associated with, irrespective of the nationality / politics of the government involved, and a majority of the large organizational projects I've worked on. Enjoy your stay. _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
|
RogerLacroix |
Posted: Wed Jul 10, 2019 9:26 am Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001 Posts: 3258 Location: London, ON Canada
|
<Vendor_Plug>
Well, if you want to control the "flood" of messages going to the government then have a look at MQ Channel Throttler. I created it, based on another customer's very similar issue.
</Vendor_Plug>
Regards,
Roger Lacroix
Capitalware Inc. _________________ Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
Back to top |
|
|
PeterPotkay |
Posted: Wed Jul 10, 2019 4:47 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001 Posts: 7717
|
Jeff.VT, what does the other side want to happen when the connection is restored and there is a large backlog of messages waiting on your side?
Do they want them all, just at some slower pace than MQ and the restored network link flood them over? _________________ Peter Potkay
Keep Calm and MQ On |
|
Back to top |
|
|
bruce2359 |
Posted: Wed Jul 10, 2019 5:03 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008 Posts: 9442 Location: US: west coast, almost. Otherwise, enroute.
|
I support MQ on z/OS and midrange. This "flooding" is quite common where the hugely provisioned mainframe MQ sender channel(s) overwhelms the puny midrange MQ. While I'm sympathetic ... _________________ I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
Back to top |
|
|
Vitor |
Posted: Thu Jul 11, 2019 4:09 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
bruce2359 wrote: |
I support MQ on z/OS and midrange. This "flooding" is quite common where the hugely provisioned mainframe MQ sender channel(s) overwhelms the puny midrange MQ. While I'm sympathetic ... |
_________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
|
|