| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| MQ6 Workload balancing on short time scale |
« View previous topic :: View next topic » |
| Author |
Message
|
| pvnes |
 Posted: Mon Jul 21, 2008 10:12 pm Post subject: MQ6 Workload balancing on short time scale |
 |
|
Newbie
Joined: 21 Jul 2008
Posts: 3
|
Hi,
As a user of MQ series within our organization i noticed something curious which not has been explained yet. We have a MQ6 cluster queue which distributes its load over 5 local queues on local queuemanagers. Configuration parameters for the local queues are:
QUEUE(A.....) TYPE(QLOCAL)
ACCTQ(QMGR) ALTDATE(2006-11-21)
ALTTIME(11.48.0 BOQNAME( )
BOTHRESH(0) CLUSNL( )
CLUSTER( ) CLWLPRTY(0)
CLWLRANK(0) CLWLUSEQ(QMGR)
CRDATE(2006-11-21) CRTIME(11.48.0
CURDEPTH(0) DEFBIND(OPEN)
DEFPRTY(0) DEFPSIST(NO)
DEFSOPT(SHARED) DEFTYPE(PREDEFINED)
DESCR( ) DISTL(NO)
GET(ENABLED) HARDENBO
INITQ( ) IPPROCS(5)
MAXDEPTH(10000) MAXMSGL(102400)
MONQ(QMGR) MSGDLVSQ(FIFO)
NOTRIGGER NPMCLASS(NORMAL)
OPPROCS(10) PROCESS( )
PUT(ENABLED) QDEPTHHI(3)
QDEPTHLO(1) QDPHIEV(ENABLED)
QDPLOEV(DISABLED) QDPMAXEV(ENABLED)
QSVCIEV(NONE) QSVCINT(999999999)
RETINTVL(999999999) SCOPE(QMGR)
SHARE STATQ(QMGR)
TRIGDATA( ) TRIGDPTH(1)
TRIGMPRI(0) TRIGTYPE(FIRST)
USAGE(NORMAL)
When measured over an interval of several seconds the load over the queues seems to be balanced equally. When looking in more detail (partial seconds) the load distribution seems no to be round robin, i.e. the load is sent consequtive to local queue A,B,C,D,E,A,B,... When the volume of messages per unit of time is low it seems more round robin, but when the load increases the deviation from the round robin model seems to grow. Why bother you think ? In the situation i am talking about MQ is the interface for a Enterprise Service Bus where we messages have a short transition time (seconds) and overhead on on the transistion time by the ESB should be minimal. Each system processing messages from the local queue has a limited number of threads for processing messages. When the load is not balanced round robin, some systems will have considerable more messages to process at specific moments in time. When the number of messages put on the local queue gets higher than the number of threads, increase of transition time due to queueing is inevitable. This is acceptable if all nodes are fully utilized, only measuring shows that some nodes are fully utilized when others are not or considerably less utilized. Below a short example of the load distribution over time. First column is the local queue, second column is the timestamp the messages is processed by the local queue.
"C" "17-07-08 07:58:33,635000000"
"D" "17-07-08 07:58:33,698000000"
"D" "17-07-08 07:58:34,589000000"
"B" "17-07-08 07:58:35,030000000"
"B" "17-07-08 07:58:35,437000000"
"A" "17-07-08 07:58:36,877000000"
"A" "17-07-08 07:58:36,976000000"
"d" "17-07-08 07:58:38,041000000"
"C" "17-07-08 07:58:38,384000000"
"C" "17-07-08 07:58:39,619000000"
"d" "17-07-08 07:58:39,730000000"
"D" "17-07-08 07:58:40,668000000"
"D" "17-07-08 07:58:41,003000000"
"B" "17-07-08 07:58:41,888000000"
"B" "17-07-08 07:58:42,442000000"
"A" "17-07-08 07:58:43,525000000"
"A" "17-07-08 07:58:43,922000000"
"d" "17-07-08 07:58:45,433000000"
"C" "17-07-08 07:58:45,522000000"
"C" "17-07-08 07:58:46,540000000"
"d" "17-07-08 07:58:47,250000000"
"D" "17-07-08 07:58:48,374000000"
"D" "17-07-08 07:58:48,540000000"
"B" "17-07-08 07:58:49,453000000"
I have three questions;
- How does MQ loadbalance, round robin ?
- Does it loadbalance on queue or queuemanager ? (based on config parameters above is think it is on queue)
- Can this observation caused by MQ ?
Thanks for all your thoughts,
Peter |
|
| Back to top |
|
 |
| PeterPotkay |
| Posted: Tue Jul 22, 2008 5:13 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Jul 22, 2008 10:27 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| pvnes wrote: |
| CLUSNL( ) CLUSTER( ) DEFBIND(OPEN) |
According to your extract, your queue is not clustered and is not setup for round robin...
You should have a value in either clusnl or cluster and defbind should say notfixed...
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| pvnes |
| Posted: Tue Jul 22, 2008 9:19 pm Post subject: |
|
|
Newbie
Joined: 21 Jul 2008
Posts: 3
|
@PeterPotkay: I am familiar with IBM infocenter, only can not find details about how the loadbalancing is executed in detail. As i described loadbalancing does work, only at a small a timescale the load is inbalance. I think this excerpt explains it.
| Quote: |
Note that the distribution of user messages is not always exact, because administration and maintenance of the cluster causes messages to flow Start of changeacrossEnd of change channels. |
If we need a more evenly distributed load is a hardware loadbalancer with MQ6 still an option?[/quote] |
|
| Back to top |
|
|
| mlr |
| Posted: Tue Jul 22, 2008 10:03 pm Post subject: |
|
|
Novice
Joined: 29 Jun 2008
Posts: 13
|
do you have admin or maintenance msgs at the same time?
wher do your timings come from. if , as i suspect, they come from the app consuming the msgs rather than from the qmgr putting the msgs, the timings only show that your app reads msgs unevely.
the cluster workload algorithm is exact, and fully published
you could open a pmr with IBM and have your assertion disproved by submitting a trace |
|
| Back to top |
|
|
| pvnes |
| Posted: Tue Jul 22, 2008 10:10 pm Post subject: |
|
|
Newbie
Joined: 21 Jul 2008
Posts: 3
|
@mlr: Timings do indeed come from the application which receives signals from the mq adapter (MQ Adapter initiates the application to retrieve msgs from the queue).
| Quote: |
| the cluster workload algorithm is exact, and fully published |
Not true if i read correctly: Excerpt from Websphere MQ V6 Fundamentals chapter 8.4.2 'The workload balancing algorithm'. "...However, this is not exact and is affected by many considerations. ..."
Same information in infocenter from IBM "Note that the distribution of user messages is not always exact, because administration and maintenance of the cluster causes messages to flow " |
|
| Back to top |
|
|
| exerk |
| Posted: Wed Jul 23, 2008 1:42 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
Bear in mind also that the load balancing will be affected by load on the queue managers within the cluster, i.e. if one is under more load than the others, it may be 'missed out' of a round-robin session.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| sami.stormrage |
| Posted: Fri Jul 25, 2008 6:27 am Post subject: |
|
|
Disciple
Joined: 25 Jun 2008
Posts: 186
Location: Bangalore/Singapore
|
And similar thing would happen, if the Q is part of multiple clusters.. messages simply jump from the overloaded channel to the lesser used.. even when the bind is open for a particular cluster Q...
_________________
*forgetting everything  * |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Jul 25, 2008 9:41 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| sami.stormrage wrote: |
| And similar thing would happen, if the Q is part of multiple clusters.. messages simply jump from the overloaded channel to the lesser used.. even when the bind is open for a particular cluster Q... |
There is nothing in the base product that considers how "overloaded" a channel is and causes a message to "simply jump" to a lesser used channel. Unless the channel can't start in the first place.
| exerk wrote: |
Bear in mind also that the load balancing will be affected by load on the queue managers within the cluster, i.e. if one is under more load than the others, it may be 'missed out' of a round-robin session.
|
There is nothing in the base product that considers what load a destination QM is under. I suppose if the destination server is so overloaded that the channel to it takes forever to start, that could be considered avoiding a loaded server, but once the channel starts the cluster will not check anything on that destination QM to see how busy it is compared to others.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| exerk |
| Posted: Fri Jul 25, 2008 10:24 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| Quote: |
| exerk wrote: |
Bear in mind also that the load balancing will be affected by load on the queue managers within the cluster, i.e. if one is under more load than the others, it may be 'missed out' of a round-robin session.
|
There is nothing in the base product that considers what load a destination QM is under. I suppose if the destination server is so overloaded that the channel to it takes forever to start, that could be considered avoiding a loaded server, but once the channel starts the cluster will not check anything on that destination QM to see how busy it is compared to others. |
Always happy to stand corrected 
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Jul 26, 2008 10:51 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| sami.stormrage wrote: |
| And similar thing would happen, if the Q is part of multiple clusters.. messages simply jump from the overloaded channel to the lesser used.. even when the bind is open for a particular cluster Q... |
Not true. If the queue is truly opened with bind options open (This is an app responsibility), there will be no load balancing of the messages. The only way this would happen is if the app opened the queue in bind not fixed even though the defbind is open... Or if the queue is really moving through name resolution forcing a defbind notfixed somewhere along the way.
@sami --posting for the joy of seeing your count go up is nice. Try sticking to the published facts better and maybe pepper your posts with more references to the manuals...
So let's see you put a little bit more thought into your 100th post?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
