| Author |
Message
|
| zicolino |
 Posted: Wed Feb 14, 2007 11:27 pm Post subject: ClusterWorkload Distribution different for new and old queue |
 |
|
Novice
Joined: 14 Feb 2007
Posts: 10
|
Situation:
We have 4 z/OS LPARs with one overlapping MQ Cluster(FUCHC00). Every LPAR has one QMGR (A to D).
On z/OS we have on every LPAR two clusterqueues defined, a CLUSTER.QUEUE.OLD and a CLUSTER.QUEUE.NEW.
The difference between the two cluster queues is the following:
- CLUSTER.QUEUE.NEW was defined on all 4 LPARs at the same time.
- CLUSTER.QUEUE.OLD was defined on 2 LPARs (A and B) 6 months ago than on LPAR C and D.
Actions #1:
A) If we put 100 messages from a remote QMGR to the CLUSTER.QUEUE.NEW than we have a workload distribution of
- 25 messages on QMGR A,B,C, and D (25/25/25/25)
B) If we put 100 messages from a remote QMGR to the CLUSTER.QUEUE.OLD than we have a diverse workload distribution of 38/38/12/12.
- 38 messages on QMGR A and B
- 12 messages on QMGR C and D
Findings:
We can see also in CASE B that 25 messages came in over every cluster receiver channel to the 4 QMGRs. But once on the z/OS arrived,
the messages are moved internal again between the 4 CLUSTER.QUEUE.OLDs until to the 38/38/12/12 result.
This internal movements must be based on the MQ workload algorithm.
Actions #2:
We asked us now, what are the differences between the old and new queues? Perhaps some internal counters which are used from
the MQ algorithm?
We did a "RESET QSTATS" on the CLUSTER.QUEUE.OLD. But the result was the same 32/32/12/12.
Actions #3:
We did as next a "DELETE CLUSTER.QUEUE.OLD" and afterwards a "DEFINE CLUSTER.QUEUE.OLD" (on all 4 QMGRs). We thought that we can clean
with this action all stored counters on the old queue and achieve the same result as with the CLUSTER.QUEUE.NEW.
Unfortunately the result was still 38/38/12/12.
Question:
What can we do to achieve a 25/25/25/25 workload distribution with the CLUSTER.QUEUE.OLD?
Optional Question:
Can anybody explain as the internal workload algorithm in other words than in the manual described?
We studied the 12 steps "...when choosing a destination for a messages" but can't find some hints to the above characteristics.
THANKS
Zicolino |
|
| Back to top |
|
 |
| Mr Butcher |
| Posted: Thu Feb 15, 2007 1:42 am Post subject: |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
which MQ Version?
same program putting to both queues? with the same open and put options?
same cluster attributes for channels, queues, ....?
same channels used for "OLD" and "NEW" queue?
as you put from remote - are your gatewaydefinitions to the cluster the same for "NEW" and "OLD"?
do all repositories (full and partial) have the same knowledge about the same cluster objects? (display on every queuemanager and compare)
what happenes if you define a "CLUSTER.QUEUE.NEW2" at all queuemanagers and try the same? (maybe it is not the "OLD" queue that does not work as expected, but the "NEW" one?)
no more guesses right now..... sounds tricky 
_________________
Regards, Butcher |
|
| Back to top |
|
|
| HubertKleinmanns |
| Posted: Thu Feb 15, 2007 4:38 am Post subject: |
|
|
Shaman
Joined: 24 Feb 2004
Posts: 732
Location: Germany
|
| Mr Butcher wrote: |
which MQ Version?
same program putting to both queues? with the same open and put options?
same cluster attributes for channels, queues, ....?
same channels used for "OLD" and "NEW" queue?
as you put from remote - are your gatewaydefinitions to the cluster the same for "NEW" and "OLD"?
do all repositories (full and partial) have the same knowledge about the same cluster objects? (display on every queuemanager and compare)
what happenes if you define a "CLUSTER.QUEUE.NEW2" at all queuemanagers and try the same? (maybe it is not the "OLD" queue that does not work as expected, but the "NEW" one?)
no more guesses right now..... sounds tricky |
And additionally:
- Are the queues defined with the same attributes?
- Are the queues in the same clusters?
- Do you use the CLUSNL attribute -> does the namelist always contain the same cluster names?
- Do you have more than one (or a different number of) cluster channels defined?
- What does the commands DISPLAY QCLUSTER(CLUSTER.QUEUE.OLD) and DISPLAY QCLUSTER(CLUSTER.QUEUE.NEW) show?
_________________
Regards
Hubert |
|
| Back to top |
|
|
| zicolino |
| Posted: Thu Feb 15, 2007 6:18 am Post subject: |
|
|
Novice
Joined: 14 Feb 2007
Posts: 10
|
which MQ Version?
- V6.0 on z/OS, V5.3 CSD 11 on Solaris
same program putting to both queues? with the same open and put options?
- yes
same cluster attributes for channels, queues, ....?
- yes
same channels used for "OLD" and "NEW" queue?
- yes
as you put from remote - are your gatewaydefinitions to the cluster the same for "NEW" and "OLD"?
- yes (with QMGR ALIAS)
do all repositories (full and partial) have the same knowledge about the same cluster objects? (display on every queuemanager and compare)
- yes
what happenes if you define a "CLUSTER.QUEUE.NEW2" at all queuemanagers and try the same? (maybe it is not the "OLD" queue that does not work as expected, but the "NEW" one?)
- Strange ... the NEW2 queue show once more a different manner |
|
| Back to top |
|
|
| HubertKleinmanns |
| Posted: Thu Feb 15, 2007 6:26 am Post subject: |
|
|
Shaman
Joined: 24 Feb 2004
Posts: 732
Location: Germany
|
Do you have installed the actual PTFs on z/OS?
It is because I found some strange behaviour using cluster queues with the combination MQv6 on z/OS and MQv5.3 on Solaris.
And additionally you should install CSD13 on Solaris!
_________________
Regards
Hubert |
|
| Back to top |
|
|
| HubertKleinmanns |
| Posted: Thu Feb 15, 2007 6:27 am Post subject: |
|
|
Shaman
Joined: 24 Feb 2004
Posts: 732
Location: Germany
|
| HubertKleinmanns wrote: |
| ... And additionally you should install CSD13 on Solaris! |
Or migrate to version 6 on Solaris too ...
_________________
Regards
Hubert |
|
| Back to top |
|
|
| zicolino |
| Posted: Thu Feb 15, 2007 6:54 am Post subject: |
|
|
Novice
Joined: 14 Feb 2007
Posts: 10
|
- Are the queues defined with the same attributes?
-> yes
- Are the queues in the same clusters?
-> no
QMGR A and B:
Cluster namelist name . : FUCHCREPOS
FUCHCREPOS = FUCHC00,FUCHC01,FUCHSDSF
Queue name . . . . . . . . : TEST.CLUSTER.QUEUE
Disposition . . . . . . . . : GROUP
Cluster name . . . . . . . : FUCHC00
QMGR C and D:
Cluster namelist name . : FUCHCREPOS
FUCHCREPOS = FUCHC00,FUCHC01,FUCHCSDSF
Queue name . . . . . . . . : TEST.CLUSTER.QUEUE
Disposition . . . . . . . . : GROUP
Cluster name . . . . . . . : FUCHC01
- Do you use the CLUSNL attribute -> does the namelist always contain the same cluster names?
-> see above
- Do you have more than one (or a different number of) cluster channels defined?
-> yes, on the Gateway we have:
DEFINE CHANNEL('CLUS.TO.QMGRA') CHLTYPE(CLUSSDR) +
CLUSTER(FUCHC00)
DEFINE CHANNEL('CLUS.TO.GW1') CHLTYPE(CLUSRCVR) +
CLUSTER(FUCHC00) +
and
DEFINE CHANNEL('CLUS.TO.QMGRC') CHLTYPE(CLUSSDR) +
CLUSTER(FUCHC01) +
DEFINE CHANNEL('CLUS.TO.GW2') CHLTYPE(CLUSRCVR) +
CLUSTER(FUCHC01) +
- What does the commands DISPLAY QCLUSTER(CLUSTER.QUEUE.OLD) and DISPLAY QCLUSTER(CLUSTER.QUEUE.NEW) show?
+QMGRC DISPLAY QCLUSTER(CLUSTER.QUEUE.NEW)
CSQM293I +QMGRC CSQMDRTC 4 QCLUSTER FOUND MATCHING REQUEST CRITERIA
CSQM201I +QMGRC CSQMDRTC DISPLAY QCLUSTER DETAILS
QUEUE(CLUSTER.QUEUE.NEW)
TYPE(QCLUSTER)
QSGDISP(COPY)
CLUSTER(FUCHC00)
DESCR(TEST QUEUE)
PUT(ENABLED)
DEFPRTY(0)
DEFPSIST(YES)
DEFBIND(NOTFIXED)
CLWLRANK(0)
CLWLPRTY(0)
CLUSQMGR(QMGRA)
QMID(QMGRA.B4808DA7FEE77345)
CLUSQT(QLOCAL)
CLUSDATE(2007-02-13)
CLUSTIME(17.39.2
ALTDATE(2007-02-13)
ALTTIME(18.48.55)
END QCLUSTER DETAILS
CSQM201I +QMGRC CSQMDRTC DISPLAY QCLUSTER DETAILS
QUEUE(CLUSTER.QUEUE.NEW)
TYPE(QCLUSTER)
QSGDISP(COPY)
CLUSTER(FUCHC00)
DESCR(TEST QUEUE)
PUT(ENABLED)
DEFPRTY(0)
DEFPSIST(YES)
DEFBIND(NOTFIXED)
CLWLRANK(0)
CLWLPRTY(0)
CLUSQMGR(QMGRB)
QMID(QMGRB.B69BC1A40F029E83)
CLUSQT(QLOCAL)
CLUSDATE(2007-02-13)
CLUSTIME(17.40.2
ALTDATE(2007-02-13)
ALTTIME(18.48.55)
END QCLUSTER DETAILS
CSQM201I +QMGRC CSQMDRTC DISPLAY QCLUSTER DETAILS
QUEUE(CLUSTER.QUEUE.NEW)
TYPE(QCLUSTER)
QSGDISP(COPY)
CLUSTER(FUCHC01)
DESCR(TEST QUEUE)
PUT(ENABLED)
DEFPRTY(0)
DEFPSIST(YES)
DEFBIND(NOTFIXED)
CLWLRANK(0)
CLWLPRTY(0)
CLUSQMGR(QMGRC)
QMID(QMGRC.BB21843F14B0AEA8)
CLUSQT(QLOCAL)
CLUSDATE(2007-02-13)
CLUSTIME(17.40.43)
ALTDATE(2007-02-13)
ALTTIME(18.48.31)
END QCLUSTER DETAILS
CSQM201I +QMGRC CSQMDRTC DISPLAY QCLUSTER DETAILS
QUEUE(CLUSTER.QUEUE.NEW)
TYPE(QCLUSTER)
QSGDISP(COPY)
CLUSTER(FUCHC01)
DESCR(TEST QUEUE)
PUT(ENABLED)
DEFPRTY(0)
DEFPSIST(YES)
DEFBIND(NOTFIXED)
CLWLRANK(0)
CLWLPRTY(0)
CLUSQMGR(QMGRD)
QMID(QMGRD.BB218701EFF3570A)
CLUSQT(QLOCAL)
CLUSDATE(2007-02-13)
CLUSTIME(17.40.43)
ALTDATE(2007-02-13)
ALTTIME(18.48.31)
END QCLUSTER DETAILS
***********************************************
+QMGRC DISPLAY QCLUSTER(CLUSTER.QUEUE.OLD)
CSQM293I +QMGRC CSQMDRTC 4 QCLUSTER FOUND MATCHING REQUEST CRITERIA
CSQM201I +QMGRC CSQMDRTC DISPLAY QCLUSTER DETAILS
QUEUE(CLUSTER.QUEUE.OLD)
TYPE(QCLUSTER)
QSGDISP(COPY)
CLUSTER(FUCHC00)
DESCR(........)
PUT(ENABLED)
DEFPRTY(0)
DEFPSIST(YES)
DEFBIND(NOTFIXED)
CLWLRANK(0)
CLWLPRTY(0)
CLUSQMGR(QMGRA)
QMID(QMGRA.B4808DA7FEE77345)
CLUSQT(QLOCAL)
CLUSDATE(2006-11-21)
CLUSTIME(11.08.17)
ALTDATE(2007-02-14)
ALTTIME(14.17.31)
END QCLUSTER DETAILS
CSQM201I +QMGRC CSQMDRTC DISPLAY QCLUSTER DETAILS
QUEUE(CLUSTER.QUEUE.OLD)
TYPE(QCLUSTER)
QSGDISP(COPY)
CLUSTER(FUCHC00)
DESCR(........)
PUT(ENABLED)
DEFPRTY(0)
DEFPSIST(YES)
DEFBIND(NOTFIXED)
CLWLRANK(0)
CLWLPRTY(0)
CLUSQMGR(QMGRB)
QMID(QMGRB.B69BC1A40F029E83)
CLUSQT(QLOCAL)
CLUSDATE(2006-11-21)
CLUSTIME(11.08.16)
ALTDATE(2007-02-14)
ALTTIME(14.17.31)
END QCLUSTER DETAILS
CSQM201I +QMGRC CSQMDRTC DISPLAY QCLUSTER DETAILS
QUEUE(CLUSTER.QUEUE.OLD)
TYPE(QCLUSTER)
QSGDISP(COPY)
CLUSTER(FUCHC01)
DESCR(......)
PUT(ENABLED)
DEFPRTY(0)
DEFPSIST(YES)
DEFBIND(NOTFIXED)
CLWLRANK(0)
CLWLPRTY(0)
CLUSQMGR(QMGRC)
QMID(QMGRC.BB21843F14B0AEA8)
CLUSQT(QLOCAL)
CLUSDATE(2007-01-0
CLUSTIME(15.41.16)
ALTDATE(2007-02-14)
ALTTIME(14.17.53)
END QCLUSTER DETAILS
CSQM201I +QMGRC CSQMDRTC DISPLAY QCLUSTER DETAILS
QUEUE(CLUSTER.QUEUE.OLD)
TYPE(QCLUSTER)
QSGDISP(COPY)
CLUSTER(FUCHC01)
DESCR(....)
PUT(ENABLED)
DEFPRTY(0)
DEFPSIST(YES)
DEFBIND(NOTFIXED)
CLWLRANK(0)
CLWLPRTY(0)
CLUSQMGR(QMGRD)
QMID(QMGRD.BB218701EFF3570A)
CLUSQT(QLOCAL)
CLUSDATE(2007-01-0
CLUSTIME(15.41.16)
ALTDATE(2007-02-14)
ALTTIME(14.17.53)
END QCLUSTER DETAILS [/img] |
|
| Back to top |
|
|
| zicolino |
| Posted: Thu Feb 15, 2007 7:01 am Post subject: |
|
|
Novice
Joined: 14 Feb 2007
Posts: 10
|
We don't think that the UNIX System has an influence on the matter. It has something to do with the z/OS MQ algorithm.
If we put 100 messages from QMGRA (direct on z/OS into the ClusterQueue not over the gateway) it distributes different than when you put from QMGRB, or C or D. So it's always a different workload distribution. |
|
| Back to top |
|
|
| HubertKleinmanns |
| Posted: Thu Feb 15, 2007 7:54 am Post subject: |
|
|
Shaman
Joined: 24 Feb 2004
Posts: 732
Location: Germany
|
zicolino,
When I am reading your last posts correct, then QMgr A and B are in one cluster and QMgr C and D in another one. Regarding your initial post, the QMgr A and B got the same number of messages as well as the QMgr C and D.
So - within the same cluster - the message distribution works as expected. But the clusters FUCHC00 and FUCHC01 got different numbers of messages.
I never would guarentee, that messages are evenly distributed within different clusters!
MQ uses internal counters, to decide, which queue will get the next message. I am quite sure, that these counters within one cluster would lead to evenly distributed messages BUT I HAVE NO IDEA WHAT HAPPENS IN DIFFERENT CLUSTERS! (just a guess: Maybe the internal counters for the old queues had some initial values in one cluster, and therefore the message distribution is not equal whereas the new queues got the same initial value - because they were created at the same time - and so the message are uniformly distributed)
By the way, the distributing (Solaris) QMgr must be a member of both clusters - right?
And when you put from the z/OS QMgrs, should find your messages only on the QMgr, which is local to the putting application (unless you modify the new attribute CLWLUSEQ attribute).
_________________
Regards
Hubert |
|
| Back to top |
|
|
| zicolino |
| Posted: Thu Feb 15, 2007 8:33 am Post subject: |
|
|
Novice
Joined: 14 Feb 2007
Posts: 10
|
1.)
I putted 1000 messages to the NEW2 queue and the result was: 125/125/250/500
=> I can't agree. Within the same cluster the message distribution doesn't work as expected.
2.)
"...MQ uses internal counters, to decide,..." this was our assumption as well and therefore we wanted to reset the attributes but weren't successful (see inital post).
3.) The Solaris GW is in both Clusters with two different CLUSTER RCVR Channels.
4) I putted 1000 messages to NEW2 queue with CLWLUSEQ=ANY on Q-level:
put 1000 on QMGR A => 250(A)/125(B)/250(C)/375(D)
put 1000 on QMGR B => 125(A)/250(B)/250(C)/375(D)
put 1000 on QMGR C => 125(A)/125(B)/250(C)/500(D)
put 1000 on QMGR D => 141(A)/140(B)/283(C)/436(D)
on every cluster queue you see a different pattern.
How can I reset the whole distribution pattern for all queues to an evenly mattern? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Feb 15, 2007 1:20 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
I have seen that stopping the cluster sender (in fact the cluster receiver on the receiver machine) and starting it again may have some influence on load balancing. (more even balance after restart)...
We run with 4 qmgrs in the cluster and the workload is pretty well balanced if the counters are o.k.
You should see an even balancing after the next IPL. Don't know how often you would do one though.
Check out if a cycling of the channel (stop start) for both targets would give you some result.
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Feb 15, 2007 6:17 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Workload balancing, if I remember Nigelg correctly (or was it ivans? - more likley ivans) is done exclusively based on channel counts.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| HubertKleinmanns |
| Posted: Fri Feb 16, 2007 12:43 am Post subject: |
|
|
Shaman
Joined: 24 Feb 2004
Posts: 732
Location: Germany
|
| jefflowrey wrote: |
| Workload balancing, if I remember Nigelg correctly (or was it ivans? - more likley ivans) is done exclusively based on channel counts. |
You are right, but I saw the list of ivans, HOW the channel count is calculated - it was a long and very complex mechanism!
_________________
Regards
Hubert |
|
| Back to top |
|
|
| zicolino |
| Posted: Fri Feb 16, 2007 12:53 am Post subject: |
|
|
Novice
Joined: 14 Feb 2007
Posts: 10
|
I've defined 3 new cluster queues on z/OS. I've putted 1000 messages over the same Unix-gateway to the cluster. The workload distribution was 3 times different.
If workload distribution is done exclusively based on channel status, why we have different results?
They were transfered over the same channel. |
|
| Back to top |
|
|
| zicolino |
| Posted: Fri Feb 16, 2007 1:17 am Post subject: |
|
|
Novice
Joined: 14 Feb 2007
Posts: 10
|
In addition to above:
The gateway channel distributes 3 times correct (from remote to z/OS => ok) but afterwards the messages are routed internally on z/OS 3 times different. |
|
| Back to top |
|
|
|
|
