| Author |
Message
|
| awatson72 |
 Posted: Thu Apr 05, 2007 7:20 am Post subject: Question regarding interconnected cluster |
 |
|
Acolyte
Joined: 14 Apr 2004
Posts: 69
Location: Freeport, Maine
|
I currently have a cluster with two full repository QMs, and three partials, and I'm considering adding a second cluster to the same QMs, but I'm not sure I'd need to. The existing cluster handles OLTP type work. The new cluster, (if neccessary to create it) would handle more batch-like work. What are the things to consider when making this sort of decision? (Weighing the increased complexity of a second cluster versus any benefit achieved by having different clusters to handle the different types of traffic).
Thanks for any insight.
_________________
Andrew Watson
L.L. Bean, Inc. |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Thu Apr 05, 2007 7:23 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
You should not design MQ clustering based on any application criteria at all.
Other than the basic need to load-balance work between instances of the same application.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| awatson72 |
| Posted: Thu Apr 05, 2007 7:57 am Post subject: |
|
|
Acolyte
Joined: 14 Apr 2004
Posts: 69
Location: Freeport, Maine
|
That's a strong recommendation. So it sounds like in your opinion, one cluster would suffice...
I believe we originally were considering two clusters, perhaps mistakenly thinking that you'd need a separate cluster to prevent messages from the OLTP application from getting mixed in the cluster transmit queue with large batches of messages from the batch application, and not getting processed in a timely fashion. Based on testing it would seem that the transmit queue is sophisticated enough to guard against that.
_________________
Andrew Watson
L.L. Bean, Inc. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Apr 05, 2007 8:40 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
It is a strong recommendation, but it is only my opinion.
And I'm very biased against overlapped clusters, and multiple clusters where one will do just fine.
Most of this comes out of a) an understanding that clustering was designed to do two things, provide simplified administration and provide workload balancing and b) some disasters I've heard of when people went way overboard and designed for application specific clusters, and overlapped everything.
Also, it seems to me that the MQ network should be designed and managed by the MQ administrators, based on physical network and administrative requirements, rather than by MQ developers based on application requirements.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Apr 05, 2007 10:51 am Post subject: |
|
|
Guest
|
Designing means understanding your applications, data, current- and future-workloads, hardware capacity, and more importantly, service-level agreements.
None of these really haa anything directly to do with MQ clusters. MQ clusters allow multiple instances of an application to process your work (OLTP and batch).
If all server hardware is at maximum capacity (RAM, disk, cpu, telecom), then any new OLTP and/or batch will/might be a problem for you. Add more (hardware) - another processor with MQ, for example, might save OLTP and batch.
Your fundamental question seems to be 'should we mix OLTP and batch?' It depends. Sorry, it's the traditional IT answer to generic questions, as this one is.
In the right (wrong?) circumstances, it is possible that batch could overwhelm OLTP service-levels. An example: midrange platform, OLTP is fine, but near hardware capacity and within service-levels. Along comes a batch application that now creates (endlessly) millions and millions of 100meg messages - all inbound from a much faster MQ (like a mainframe), and all launching a consuming program.
Understanding the apps, workloads, etc., you should be able to plan for growth of existing applications, as well as new ones. |
|
| Back to top |
|
|
| awatson72 |
| Posted: Thu Apr 05, 2007 11:58 am Post subject: |
|
|
Acolyte
Joined: 14 Apr 2004
Posts: 69
Location: Freeport, Maine
|
Bruce,
I would expect your scenario would cause problems regardless of whether multiple clusters were used - simply of matter of too much volume, too little capacity.
So knowing what I know about the applictions/data workloads, etc., I'll plan to go forward using the existing cluster for the batch work. Thanks for the input.
_________________
Andrew Watson
L.L. Bean, Inc. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Apr 05, 2007 1:01 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
And just to be on the safe side (we did hit some minor problems until we fixed this) I would make sure that OLTP and batch traffic have different priorities. OLTP traffic should have a higher priority than Batch traffic in any case....
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| shassey |
| Posted: Thu Apr 19, 2007 4:27 am Post subject: |
|
|
Apprentice
Joined: 25 Dec 2001
Posts: 25
Location: shassey
|
I would like to clarify my understanding of the previous reply as to why setting priorities might be important...
We did some testing of both batch traffic and request/reply traffic on the same cluster to see if they could share a cluster without impacting each other and it appeared to work just fine. Is it true that the cluster creates separate 'channels' under the covers for messages travelling through? What I am unclear about is that there is only one SYSTEM.CLUSTER.TRANSMIT.QUEUE queue on a queue manager. Even if you had separate clusters - there is still only one SYSTEM.CLUSTER.TRANSMIT.QUEUE . Can anyone explain?
Thanks
>And just to be on the safe side (we did hit some minor problems until we >fixed this) I would make sure that OLTP and batch traffic have different >priorities. OLTP traffic should have a higher priority than Batch traffic in >any case.... |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Apr 19, 2007 5:12 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Yes, an MQ Cluster buids a fully connected network using auto-defined channels.
Yes, the System.Cluster.Transmit.Queue is not a bottleneck for traffic. Under the covers, the cluster MCA uses the CorrelationID of messages on the S.C.T.Q to get messages for each individual destination qmgr.
Yes, using multiple clusters is not going to improve throughput.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Apr 19, 2007 12:27 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
But using different priorities will improve your response time during the batch transmission.
Imagine a batch process where you dump thousands of messages into a queue that is load balanced in the cluster.
Any request comming in and going to the same qmgr as the batch will have to wait behind those in the xmitq. If the priorities are higher for online than batch (remember the default for cluster xmitq is priority) the online messages will move to the head of the queue and not have to wait until the batch is transferred.
Of course this comes with a slight performance impact but still way less than if the message had to wait in the xmitq until the batch has been transferred...
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
