| Author |
Message
|
| PeterPotkay |
 Posted: Wed Oct 20, 2004 4:10 pm Post subject: When is it time for an additional broker? |
 |
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
I have 2 Brokers running in an MQSeries cluster as follows on 2 servers:
MQSeries 5.3.0.8
WB-IMB 5.0.4
Windows 2000 Advanced Server
MSCS
4 GB RAM (Server has capacity for 8GB)
Dual processors per server Pent III 1.26 GHz
What criteria do I use to determine that it is time to add a third server to run a third broker in this MQ cluster? Is it as simple as watching memory and cpu use, and if it crosses X consistently, then its time for another broker? If yes, what is X?
Or is there some other criteria to use? Other than some hi volume batch jobs which cause those particular queues to back up, none of the Input queues back up ever.
Or is it better to just get to 8 GB of mem and/or upgrade the CPUs and keep it at 2 brokers?
I guess what I am asking is when do you know you need capacity?
And do you scale vertically or horizontaly?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Wed Oct 20, 2004 4:59 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Well... I know you know this, and I'm not helping by repeating it...
But it's always better to understand your performance requirements before deciding that you need to do stuff to improve performance.
| PeterPotkay wrote: |
| Other than some hi volume batch jobs which cause those particular queues to back up, none of the Input queues back up ever. |
I'd say then, that you should only increase the number of instances of the message flows servicing those queues, before you even begin to think about scaling the hardware.
Particularly, if you're not seeing that you're binding on CPU resources or I/O resources, then scale in the following manner until you ARE binding: additional instances, additional execution groups.
Then, once you do hit CPU/IO limits, you can add another broker, and scale horizontally the particular flows that you're having issues with.
Scale vertically, by adding additional brokers on the same hardware, when you reach the limit of addressable memory by a single process.
At least, that's my opinion.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Oct 20, 2004 5:23 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
CPU gets high but does not peg at 100% during those batch runs. We have 2 execution groups per server, to hopefully take advantage of the dual processors. The batch runs happen early Saturday AM, so there is not a lot of other business going thru.
I am not consired so much about the batch runs, as I am about day to day operations. When do you say "Uh Oh. Its time to throw more hardware at this problem."?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| kirani |
| Posted: Wed Oct 20, 2004 10:24 pm Post subject: |
|
|
Jedi Knight
Joined: 05 Sep 2001
Posts: 3779
Location: Torrance, CA, USA
|
Try to creae more EG's and distribute your message flow by load and deploy them to these EG's evenly. Adding more memory to the box will not help you much. Adding more CPU will definitely help.
Also, it makes sense to add another broker on same machine only if the queue manager logs/data reside on different drive than the first broker.
Can you run more than one instance of Batch message flow?
_________________
Kiran
IBM Cert. Solution Designer & System Administrator - WBIMB V5
IBM Cert. Solutions Expert - WMQI
IBM Cert. Specialist - WMQI, MQSeries
IBM Cert. Developer - MQSeries
|
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Oct 21, 2004 2:35 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| PeterPotkay wrote: |
| I am not consired so much about the batch runs, as I am about day to day operations. When do you say "Uh Oh. Its time to throw more hardware at this problem."? |
When your CPU is consistently high most of the time, or when your flows are slowing down because of disk I/O operations, or when you are stressing the total amount of memory on the box.
Until one of those points, I'd scale vertically either by adding more instances and then adding more EGs.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Oct 21, 2004 7:49 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| kirani wrote: |
| Adding more memory to the box will not help you much. Adding more CPU will definitely help. |
But isn't a flow passing data between nodes in memory? I am thinking about flows in the future that will be waiting for an HTTP response. While they wait, I assume they have their little chunck of memory tied up, correct? So to little memory might be a bottleneck.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Oct 21, 2004 8:19 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
We have driven our 12-CPU P680 to 100% using one broker. You don't need more than one broker. You need enough execution groups.
Use additional brokers to isolate work for service reasons, for example we have an online broker that is 24x7 and one that is not. |
|
| Back to top |
|
|
| KeeferG |
| Posted: Thu Oct 21, 2004 8:34 am Post subject: |
|
|
Master
Joined: 15 Oct 2004
Posts: 215
Location: Basingstoke, UK
|
Adding more instances will improve performance without using up as much memory as increasing EG's as each EG runs inside its own memory space. The best way for scaling is to first increase instances then EG's until your processors run at your max. It used to be that you would scale an EG per processor as well as this gave predicitable results. Then once you have reached you maximum start to add new machines.
Its is also worth bearing in mind that if you want good system availability then you should plan your CPU max to be able to handle the load of the other machine in the cluster so if 2 machines then max cpu is 50%, if three machines then 66% etc. This allows you to take a machine out for maintenance without affecting service.
It gets fun when you are having 30 brokers in your cluster balancing the load of 150m messages a day.
_________________
Keith Guttridge
-----------------
Using MQ since 1995 |
|
| Back to top |
|
|
| kirani |
| Posted: Thu Oct 21, 2004 10:22 pm Post subject: |
|
|
Jedi Knight
Joined: 05 Sep 2001
Posts: 3779
Location: Torrance, CA, USA
|
Peter,
How much memory are you using currently?
In our environment we have Windows 2000 server with 8 CPU and 8 GB RAM. We do very heavy transformation for Online and Batch transactions. We feel that we need more CPU power to allow parallel processing. We have very tight SLA, so it's kind of difficult to run online and batch message flows together. When they run together we start to see online queues build-up, which should not be the case for our app.
_________________
Kiran
IBM Cert. Solution Designer & System Administrator - WBIMB V5
IBM Cert. Solutions Expert - WMQI
IBM Cert. Specialist - WMQI, MQSeries
IBM Cert. Developer - MQSeries
|
|
| Back to top |
|
|
|
|
