ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexGeneral IBM MQ SupportIs dynamic client conn rebalancing in qmgr group possible?

Post new topicReply to topic Goto page 1, 2  Next
Is dynamic client conn rebalancing in qmgr group possible? View previous topic :: View next topic
Author Message
fredmoore
PostPosted: Wed Nov 07, 2018 4:31 am Post subject: Is dynamic client conn rebalancing in qmgr group possible? Reply with quote

Novice

Joined: 23 Mar 2009
Posts: 23

Hi folks,

any help with this is appreciated.


Scenario

1\ Many MQ clients connect to a 4 different queue managers, forming a queue manager group, using a client channel definition table (CCDT)

2\ One of the queue manager becomes unavailable, and the clients connected to that queue manager will scatter across the remaining queue managers

3\ The unavailable queue manager comes up again after a very short downtime, and some of the new connections will be directed to it

4\ EVENTUALLY the connections will be balanced across the 4 queue manager

Side information

The client applications are actually instances of an IBM Integration Bus MessageFlow putting messages via an MQOutput node that has a policy pointing to a CCDT.

Problem

Clients will not disconnect as long as they don't hit an error, so it takes a very long time for the connections to be balanced over 4 queue managers after the failure of one of them, during this time the performance on the overloaded queue manager is unacceptable.

Question

Is there a way to configure the system to perform some sort of "dynamic rebalancing" of active connections that is transparent to the application?


Thanks in advance for your help!
Cheers,
F.
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Nov 07, 2018 5:12 am Post subject: Re: Is dynamic client conn rebalancing in qmgr group possibl Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25310
Location: Ohio, USA

fredmoore wrote:
Clients will not disconnect as long as they don't hit an error, so it takes a very long time for the connections to be balanced over 4 queue managers after the failure of one of them, during this time the performance on the overloaded queue manager is unacceptable


I'm surprised that any connections rebalance without disconnecting and reconnecting.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
fredmoore
PostPosted: Wed Nov 07, 2018 5:56 am Post subject: Reply with quote

Novice

Joined: 23 Mar 2009
Posts: 23

Quote:
I'm surprised that any connections rebalance without disconnecting and reconnecting.


Let me try to explain it better:

1\ connections are balanced only upon reconnect

2\ due to the client logic not disconnecting often enough this means that it will take a long time for client connections to be evenly distributed again across the 4 queue managers (roughly the time it takes for most of them to disconnect and reconnect once)

Hence my question about dynamic and application transparent rebalancing of "live" connections: is this achievable?

TIA,
F.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Nov 07, 2018 6:19 am Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8187
Location: US: west coast, almost. Otherwise, enroute.

fredmoore wrote:

1\ connections are balanced only upon reconnect


How are you currently implementing connection balancing? What tool?

What asks for balancing? The client app? Something else?
_________________
I would tell you a UDP joke, but you might not get it.
Back to top
View user's profile Send private message
fredmoore
PostPosted: Wed Nov 07, 2018 6:26 am Post subject: Reply with quote

Novice

Joined: 23 Mar 2009
Posts: 23

Quote:
How are you currently implementing connection balancing? What tool?


No tool other that MQ itself: we have a simple CCDT with four queue managers in it, forming a queue manager group

Quote:
What asks for balancing? The client app? Something else?


Balancing (and rebalancing) is required to distribute the load across all available queue managers and as evenly as possible, for performance purposes: it's a busy system.
Back to top
View user's profile Send private message
exerk
PostPosted: Wed Nov 07, 2018 6:31 am Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 5955

Set affinities so that when a queue manager does become 'available' again, any clients that disconnect from a non-affinity queue manager will connect to their preference.

For example, if you have 16 clients, set 4 each to have an affinity to one queue manager, and to connect to any available thereafter, or in order of preference to the other three.

It does mean having multiple CCDT files spread across 'groups' of clients, but it's about the only way I can think of to do it.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.

Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Nov 07, 2018 6:49 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25310
Location: Ohio, USA

fredmoore wrote:
Quote:
How are you currently implementing connection balancing? What tool?


No tool other that MQ itself: we have a simple CCDT with four queue managers in it, forming a queue manager group

Balancing (and rebalancing) is required to distribute the load across all available queue managers and as evenly as possible, for performance purposes: it's a busy system.


Nothing (not even an external tool like an F5) will break a connection that an application is maintaining. If the application is using a connection, nothing within the network or MQ will willfully break that, because that's going to cause data loss within the application.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Nov 07, 2018 7:19 am Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8187
Location: US: west coast, almost. Otherwise, enroute.

I'm puzzled by your initial premise - that a qmgr becomes unavailable.

Does this occur frequently? If so, what are the root causes? Network? Server hardware failure? O/S software failure? MQ internal failure? Is your MQ server software current? Is your MQ client software current?

I've seen far more client-side failures than server-side. Clients reconnect can be painfully slow - affecting SLAs. Unless MQ internals have changed, each channel entry in the CCDT entry is tried alphabetically by channel name, nine times before the next entry is tried. Do you have only one CCDT for all clients? As suggested earlier in this thread, smaller individual CCDTs might improve reconnect.
_________________
I would tell you a UDP joke, but you might not get it.
Back to top
View user's profile Send private message
fredmoore
PostPosted: Wed Nov 07, 2018 7:44 am Post subject: Reply with quote

Novice

Joined: 23 Mar 2009
Posts: 23

Quote:
Does this occur frequently? If so, what are the root causes? Network? Server hardware failure? O/S software failure? MQ internal failure? Is your MQ server software current? Is your MQ client software current?


It does not happen frequently, when it does it is sometimes simply due to planned hw/sw maintenance, plus a "pinch" of all sorts of unplanned stuff.

The point here was how to exploit available resources evenly, especially when the queue manager comes back again.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Nov 07, 2018 8:19 am Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8187
Location: US: west coast, almost. Otherwise, enroute.

fredmoore wrote:
Quote:
Does this occur frequently? If so, what are the root causes? Network? Server hardware failure? O/S software failure? MQ internal failure? Is your MQ server software current? Is your MQ client software current?


It does not happen frequently, when it does it is sometimes simply due to planned hw/sw maintenance, plus a "pinch" of all sorts of unplanned stuff.

The point here was how to exploit available resources evenly, especially when the queue manager comes back again.

There is no single "thing" that is going to fix this.

Both client-bindings and server-bindings apps are responsible for behaving, well, responsibility - like connecting or reconnecting, as necessary. Client-bindings apps suffer the slings and arrows of network and small-platform fortunes. Less so with server-bindings apps.

This thread has given you some things to ponder.

Does the SLA tolerate some transaction throughput misses?
_________________
I would tell you a UDP joke, but you might not get it.
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Nov 07, 2018 8:50 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25310
Location: Ohio, USA

fredmoore wrote:
The point here was how to exploit available resources evenly, especially when the queue manager comes back again.


So let me see if I have your requirement straight.

You have 4 queue managers, with a group of applications connected across them. One queue manager becomes unavailable, and the applications in response reconnect to one of the others & restart their work.

When the errant queue manager becomes available again, you're looking for a way to forcibly disconnect the applications (causing whatever they're doing to fail) so they reconnect to their "preferred" queue manager.

Are you looking to do this only to the applications which are on the "wrong" queue manager, or are you happy to abend all the applications and stop all processing until they reconnect? On a "busy system" I'd expect the former.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Nov 07, 2018 9:25 am Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8187
Location: US: west coast, almost. Otherwise, enroute.

The OP mentioned that apps don't disconnect frequently. So, the app MQCONNects, then waits around for some transaction activity, repeat, repeat - never or seldom disconnecting - like a long-running (redundant) batch job.

Interesting, but bad app design, trying to circumvent (improve on) on MQ's plodding CCDT reconnect process should a failure occur.
_________________
I would tell you a UDP joke, but you might not get it.
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Nov 07, 2018 10:12 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25310
Location: Ohio, USA

bruce2359 wrote:
The OP mentioned that apps don't disconnect frequently. So, the app MQCONNects, then waits around for some transaction activity, repeat, repeat - never or seldom disconnecting - like a long-running (redundant) batch job.

Interesting, but bad app design, trying to circumvent (improve on) on MQ's plodding CCDT reconnect process should a failure occur.


I know, I know - I'm trying to get the OP thinking about what he's asking for and how it can't be fixed at the MQ level


_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Nov 07, 2018 1:20 pm Post subject: Re: Is dynamic client conn rebalancing in qmgr group possibl Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8187
Location: US: west coast, almost. Otherwise, enroute.

fredmoore wrote:

Problem

Clients will not disconnect as long as they don't hit an error,

Please be more precise. By "error" you mean an application logic error? Something else?

fredmoore wrote:
so it takes a very long time for the connections to be balanced over 4 queue managers after the failure of one of them, during this time the performance on the overloaded queue manager is unacceptable.

This is different from the app deciding to MQDISC. This is qmgr shutting down.

How long is a "very long time?" And how do you determine when (how long) before the workload appears balanced? How do you know when it is balanced? What tooling do you use?
_________________
I would tell you a UDP joke, but you might not get it.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Wed Nov 07, 2018 5:15 pm Post subject: Reply with quote

Jedi Council

Joined: 15 May 2001
Posts: 7466

Vitor wrote:

When the errant queue manager becomes available again, you're looking for a way to forcibly disconnect the applications (causing whatever they're doing to fail) so they reconnect to their "preferred" queue manager.


This is how DataPower does it with its MQ QM Groups. It is constantly checking for the return to availability of its preferred queue manager and once it sees that, it gracefully ends the connections to the secondary QM and reconnects to the preferred. Works reliably and out of the box. But its the application (DataPower in this case) that does the rebalancing of connections to the queue managers.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum IndexGeneral IBM MQ SupportIs dynamic client conn rebalancing in qmgr group possible?
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.