| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| Lots of Execution Groups? |
« View previous topic :: View next topic » |
| Author |
Message
|
| PeterPotkay |
 Posted: Fri Sep 16, 2005 7:39 am Post subject: Re: # of Execution Group |
 |
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| QMan wrote: |
we have 10 EG - may increase to 12
Total # of FLows ~ 140 - evenly spread - grouped according to application.
|
Evenly spread by # of flows, (each of your 10 EGs has ~14 flows) or do you look at the size of the messages a flow deals with, and if it is big, then it counts more (EG1 has 15 flows that deal with small messages, and EG2 has only 5 flows, because they are BIG message)?
Thanks for sharing your experiences...
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
 |
| QMan |
| Posted: Fri Sep 16, 2005 8:13 am Post subject: # of EG Groups |
|
|
Novice
Joined: 25 Jun 2003
Posts: 11
|
Hi Peter,
I was not being clear on "evenly spreading of the msg. flows"
My Flows are distributed in the EG - as per the Interface and type -
meaning - Synchronous - small msgs (sap bapi call) - e.g. 6 Flows are in 1 EG
Critical orders interface - 3 flows in 1 EG
not so critical - meaning if fails, can be processed again at later time - monster msg.s > 10 - 20 MB - total 15 flows are in 1 EG
sap interface - big files , create thousands of idocs and send to sap - 10 msg flows are in 1 EG
so, basically no clear rule - I have created more EG - for easier control and some balance for performance..
clearly if I apply cpu rule, I have to live with max 4 EG, each with say 40 flows.. The problem is config manager --> broker control.. if any flows are running, any change to the EG will take a long time..
also - We still run on WMQI 2.1
Hope to migrate to 5 sometimes..
thanks.
_________________
Cheers!
Q Man
Websphere MQ Specialist, MQ Workflow Solun. Expert |
|
| Back to top |
|
|
| KeeferG |
| Posted: Wed Sep 21, 2005 7:16 am Post subject: |
|
|
Master
Joined: 15 Oct 2004
Posts: 215
Location: Basingstoke, UK
|
In any of my high performance engagements I used to always run a performance test of a single instance of a message flow in a single execution group. Initialy I tested this using IBM internal tools then IH03 tools to standardise it and allow the client to use the tools after the engagement had finished. The purpose of this was to find out peak performance of a flow in ideal conditions and to look at CPU utilisation.
This helps in determining what flows can coexist inside execution groups.
In some of the higher throughput systems, say 15,000+ mps we would often run 15/20 instances of a single flow per group over several groups distributed across several brokers.
The initial performance figures were recorded for each flow to allow better future planning. Grouping by application etc is still done.
_________________
Keith Guttridge
-----------------
Using MQ since 1995 |
|
| Back to top |
|
|
| hopsala |
| Posted: Mon Oct 17, 2005 3:28 pm Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
Just dug this up, I must say it's been a brilliant read.
Peter, what came about all this? How did you eventually set up your new environemnt? Any conclusions? |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue Oct 18, 2005 9:44 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Still struggling with this. Next week my new DEV Broker servers arrive, and I am going to have make a decision. I just don't know what the correct thing to do is.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Oct 26, 2005 11:25 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
For what its worth, here is what we are going to try:
We will have 10 generic Execution Groups, named EG1, EG2, EG3, etc.
EG10 will be reserved for all Batch style flows.
EG9 will be reserved for all flows that use HTTP Input nodes. If we have a problem with HTTP hitting our Brokers, we can stop that EG without effecting other flows.
EGs 1-8 will be populated with all other flows.
We will try and balance out these 8 EGs so their memory and CPU usage are relatively equal (see more below on this).
Because the EGs are named generically, it is easy to shuffle flows if we want to, and we are not tied to division names that change every 4.3 years.
A Request Flow and its corresponding Reply Flow will always be deployed to the same EG.
Whenever possible, all closely related flows will be kept together in the same EG, i.e., all the EOR flows will be in the same EG.
Bar files will only contain the flows and sets that are changing.
Bar files will be called EG_ReleaseDate_Comments.bar, for example, EG3_11012005_BRSauto.bar
The developers will keep a spreadsheet hosted on their website that will show which flows and sets are present in each EG. (see below for info on how we should split the flows up)
If we ever need to rebuild the entire Broker, we will use the above spreadsheet to let us know which flows and sets need to be deployed to which EGs (new bar files would be made for this hope-I-never-have-to event)
We will try this out on the new DEV Broker next week.
If this concept is a giant mess in DEV, we have 2 weeks before the new QA environment is here to tweak the concept.
What we do in PVCS as a result of all this was not yet discussed.
I don't know how to post the pictures, but.....
Attached is a picture of the Execution Group's memory and CPU usage on our Prod Broker, sorted by memory usage. The text file from MO71's Queue Usage * command (make sure you view it with word wrap turned off) is a list of all the open queues on the Broker, sorted by the PID that has it open. Together, these 2 docs allow you to see which EGs are using the most memory or CPU time. Interesting point - ABC UW and ABC PROP are in the top 5. Combining all of the ABC flows into one is not a good idea if we want to even it all out!
As the developers separate all the flows to the 10 generic EGs, the key is not to stick the biggest memory and CPU users in the same EG. Spread the top 8 users between EG1 thru EG8, put batch in EG10, HTTP Input in EG9. Not an exact science, but a good start to make sure we don't have 857 EGs eventually, or 5 EGs, where 2 are gigantic.
As new flows are developed, the below stats and the EG Spreadsheet will be used to determine what is the best EG to deploy to.
Here goes nothing.....  
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| Rahul999 |
| Posted: Mon Oct 26, 2009 7:34 pm Post subject: |
|
|
Centurion
Joined: 14 Mar 2007
Posts: 134
|
| Quote: |
| There are other limits that you'll come up against, like the operating system threads. According to Tim Dunn's article on Determining How Many Copies of a Message Flow Should Run Within WebSphere Business Integration Message Broker V5, it's not possible to run more than 500 threads per process/DataFlowEngine (with one messageflow equated to one or more threads). |
What is the limit in V6, is it the same 500 threads/DataFlowEngine.
I was wondering how many message flows can be run in a execution group in case of V6?
_________________
"For all your days be prepared, and meet them ever alike.
When you are the anvil, bear - when you are the hammer, strike."
- Edwin Markham |
|
| Back to top |
|
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
