| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| Lots of Execution Groups? |
« View previous topic :: View next topic » |
| Author |
Message
|
| JT |
 Posted: Thu Jul 28, 2005 4:58 pm Post subject: |
 |
|
Padawan
Joined: 27 Mar 2003
Posts: 1564
Location: Hartford, CT.
|
|
| Back to top |
|
 |
| tillywern |
| Posted: Tue Sep 13, 2005 7:53 am Post subject: Diversity of OSs |
|
|
Centurion
Joined: 28 Jan 2003
Posts: 109
Location: Colorado
|
I have read through this with interest since I have been tuning NEON, MQSI, WMQI etc. since 2000. In reading through this set of posts I can say that I have experienced a lot of what everyone has stated… Including, but not limited to, the nuances of how the product functions on different platforms.
My current interest is how threading is handled under Solaris. To put my comments in perspective, I worked supporting and tuning an 18 month implementation of WBIMB on AIX platform. Now I am working with WBIMB on Solaris.
The nature of the flow on AIX was to remove data off a queue and insert it, row by row, into an Oracle DB, obviously we had I/O wait here. With this configuration it was easy to see that each EG seemed to only use a single CPU no matter how many threads were used. So we applied a number of threads to the flows in the EG such that we could get % CPU utilization = 100 / # CPUs.
Currently we have a CPU bound flow, primarily data formatting, running on V5 under Solaris. I have seen statistics showing that with a single EG we can drive total CPU utilization to 75% by adding threads. I do not have full analysis of how 75% of 4 CPUs is used. In lieu of a better tool for analysis I use vmstat. Solaris vmstat does not make the break between IO wait and Sys CPU consumption so it is a little more difficult to understand exactly what is being used. So I will try to run again processing data from mpstat instead of vmstat.
System statistics aside, output/throughput on Solaris, seems to follow a more traditional EG bound to a CPU method even though CPU on other processes is expended. I don’t know if would be possible to generate 50% load on a machine just related to thread contention.
In general I apply the following rules of thumb for dedicated WBIMB servers:
1. EGs = # processors is a good place to start
2. # EGS > 2x#CPU seems excessive without data to support it
3. # threads of a flow approximately = ( 100 / #CPU ) / %CPU under load for example on a four CPU machine that is in an enqueue state and using 6% CPU : (100 / 4) /6 = 4.166 threads
4. Use the power of 2 rule for coarse tuning… Multiplying threads OR EGs by 2 or cut them in half. Then rerun test.
5. ALWAYS gather system resource consumption data from an enqueue state with a solid, well defined, baseline of both transaction and reference data.
6. Remember that tuning for both MAX system throughput and transaction priority is rarely attainable. Usually one will suffer when you get to the final tuning phase, >80% CPU utilized.
In summary, a great deal of wisdom and experience has been brought to the table in the posts to which I am adding this. But through it all I think we can say that it is only through rigorous testing and experience that we can maximize utilization and throughput. Every flow is different as is every client environment. Products change and it is rare that one get experience on every supported platform during a major product release.
Baseline environment, test, analyze results, repeat… |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue Sep 13, 2005 4:21 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
tillywern,
I want to shrink the number of EGs on my 2 CPU boxes. It feels "scary" having an EG with 75+ flows in it. How many flows have you jammed into a single EG? What formula do you use in determining that?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Sep 14, 2005 8:28 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Below is a quote from one of our internal emails on this subject. More food for thought.
| Quote: |
BI = Business Ins.
PI = Personel Ins.
# of EGs - I dunno. Seems kinda scary to drop down just to 3, especially if big messages start going thru. On the flip side, we can't keep going they way we are. What if we add EXEC_GROUP_4, where we put all batch stuff, and EXEC_GROUP_5, where all HTTP flows go. Then on each broker, we duplicate the set of 5 EGs, with the same flows in each pair, like this:
EG_1a - BI
EG_1b - BI
EG_2a - PI
EG_2b - PI
EG_3a - Misc
EG_3b - Misc
EG_4a - Batch
EG_4b - Batch
EG_5a - HTTP
EG_5b - HTTP
This does 2 things for us. We don't have all our PI flow (for example) eggs in one EG basket. If an EG on a broker crashes, its partner EG is still proccessing. And when both are working fine, 99.999% of the time, we have twice the processing for every flow. Downside is we have to do 2 deploys per broker - big deal. In the above scenario, do you think any 1 pair of those EGs would have a LOT more flows than the other? Maybe EGs 1 and 2 would be very busy, huh? Could we split these up some more? And now we start EG creep all over again!  And there is my dilemma again.
|
But, if an EG can access more than one CPU, then why have more than 1 Execution Group, even on a 8 CPU box? If that 1 EG can get to all 8 CPUs, then what's the point of 2 EGs, or 8 EGs, if the single EG can farm its threads to all the CPUs itself.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Sep 14, 2005 8:32 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| PeterPotkay wrote: |
| But, if an EG can access more than one CPU, then why have more than 1 Execution Group, even on a 8 CPU box? If that 1 EG can get to all 8 CPUs, then what's the point of 2 EGs, or 8 EGs, if the single EG can farm its threads to all the CPUs itself. |
Controllability... if a flow locks up the only EG you have on a box...
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Sep 14, 2005 10:51 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
OK, so then a broker should have just 2 EGs, twins of each other? When all is good, 99.9999% of the time, each Input queue has IPROCS of at least 2 (more if you choose to have multiple instances of a flow inside an EG). If 1 of the EGs takes a dirtnap for whatever reason, the queues never stop being processed, because the other EG stays up.
On a server with 200 flows, 2 EGs, each with the same 200 flows?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Sep 14, 2005 11:01 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
You might run into threads per process limitations or max-handles limitations and that kind of thing.
I think I remember there also being an impact from EGs on database connection pooling, on the order of "more EGs means more DB connections".
Have the performance support packs not been very helpful?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Sep 14, 2005 11:08 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
No, I have not found anything that gives a hard and fast formula for this. I have gone back and forth with IBM on this, and there is no rules to go by when determining the number of EGs.
I know to many EGs is bad, because each one takes a good amount of memory. 1 seems to be that is all that is needed. Well, two if you want to avoid the problem of 1 EG going down. But are EGs so fragile? Are they any more fragile than the entire broker? Probably not, so why duplicate them? I suppose thats why we have 2+ brokers on separate hardware anyway, so why double EGs on the same machine even.
The max connections to DBs or threads per process is a good point, and you could use that as a ceiling for # of flows in an EG. Other than that, what else would be a reason not to just create 1 EG, and jam all the flows in there?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| JT |
| Posted: Wed Sep 14, 2005 11:10 am Post subject: |
|
|
Padawan
Joined: 27 Mar 2003
Posts: 1564
Location: Hartford, CT.
|
....but then you have to consider the following:
| Quote: |
If you are running WebSphere Business Integration Message Broker with a large number of configured components, the memory footprint of the broker processes (particularly the DataFlowEngine) might exceed their memory limits. In particular, the user process limit might be exceeded, or the address space limit might be reached. You might encounter problems, such as the BIP2106E error message, when running a broker with: - A large number of message flows
- Multiple databases
- Very large input or output messages
Use platform-specific tools to check the maximum size of the failing process. Then check for any user limits (if applicable) or computer limits on process size.There is a hard limit on each operating system for the maximum process size, beyond which failure is inevitable, as shown in the following table:
Platform / Maximum process size
Windows / 2 GB
Solaris / About 3.75 GB
HP-UX / About 3 GB, depending on kernel settings
AIX (32 bit system) / 2 GB
z/OS / 2 GB
Increasing user limits beyond these values will not make any difference. If your broker processes regularly reach these sizes, consider spreading your message flows across more execution groups to reduce the size of each one below these limits. |
http://publib.boulder.ibm.com/infocenter/wbihelp/topic/com.ibm.etools.mft.doc/au16640_.htm |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Sep 14, 2005 11:16 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
EGs aren't fragile.
Message flows are fragile, and a message flow that goes into an infinite loop will lock up the EG, as well as horde the CPU and etc.
And you and I both know that it's very hard to shift blame for a production outage in these kinds of cases to the development team when a few simple defensive tactics on your end could have prevented the problem...
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Sep 14, 2005 5:23 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
JT,
trying to understand...what contributes to a dataflowengine.exe's size? How do you predict / plan for that? Because that is a very good indicator for how many EGs I should plan for. OK, 2 GB is my limit, based on this many flows, and this being the average message size, blah blah blah, boom, I need x EGs. The above quote is from the problem determination section, telling me to find out what the size is of a problem EG. I need this info before there are problems!
Jeff,
| Quote: |
Message flows are fragile, and a message flow that goes into an infinite loop will lock up the EG, as well as horde the CPU and etc.
|
Exactly. So by extension, I should put only 1 flow in an execution group, so it can't hurt anything else. We both know that's silly, so the other way is to just put them all in one. Why bother even doing my original idea of twin EGs. A bad flow will kill both EGs just the same. Maybe in DEV, the first time a flow is deployed, yeah, give it its own EG. But once its stability is proven, what's the logic in not throwing it into the 1 big EG, because EGs are not fragile, can access multiple CPUs, and we have yet to see a formula in predicting the "size" of an EG.
IBM gave us the option of more than 1 EG, but very muddy details, if at all, on determining the # of EGs over 1 we really need.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| JT |
| Posted: Wed Sep 14, 2005 8:15 pm Post subject: |
|
|
Padawan
Joined: 27 Mar 2003
Posts: 1564
Location: Hartford, CT.
|
Peter, when you started talking about stuffing 200 message flows into a single exectuion group, I wanted to make you aware of some boundries (which, you apparently already knew). At some point you're going to come up against the upper limit, maybe not at 200 message flows, but there will be a number that you can't exceed within a single execution group.
There are other limits that you'll come up against, like the operating system threads. According to Tim Dunn's article on Determining How Many Copies of a Message Flow Should Run Within WebSphere Business Integration Message Broker V5, it's not possible to run more than 500 threads per process/DataFlowEngine (with one messageflow equated to one or more threads). That being the case, this limit restricts the number of message flows you can stuff into a single execution group, and subsequently impacts the number of execution groups you'll need.
I understand that you're trying to determine what that mystical number is, I think your answer lies in knowing the characteristics of your message flows, i.e. do they require additional instances, do they process large-size messages. As Tim states in his article, each of those message flow characteristics requires a different solution. One being, spread the message flow over multiple exctuion groups, while the other is to restrict the message flow to a single execution group.
| Quote: |
| How do you predict / plan for that? |
How much knowledge regarding the message flow's performance/resource consumption can you garner from your Q.A. environment, prior to configuring your production brokers ? |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Sep 15, 2005 5:36 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| JT wrote: |
| Quote: |
| How do you predict / plan for that? |
How much knowledge regarding the message flow's performance/resource consumption can you garner from your Q.A. environment, prior to configuring your production brokers ? |
So we are left with throwing it at the wall and seeing if it sticks in QA? Yes, of course we test in QA, but really, shouldn't there be a checklist, a formula, a list of things to consider, ****all in one little section titled "# of EGs you should have"****, instead of what we (don't) have now?
So far I see 1 concrete factor: You can't have more than 500 flows in an EG. That's good, that's something I can calculate.
Next, they say a EG cannot exceed 2 GB on Windows. OK, thank you very much, but how do I calculate the size of my EG ahead of time? Yeah, I can guess the # of flows and the average size of messages, but then what do I do with that #? how does that equate with a "size of an EG"?
Apparently there are OS restrictions of the max # of threads allowed per process. I can figure that out on my own. Would be nice if IBM provided a little chart in this new section of the manual. Obviously they must be aware of these #s already.
Anything else?
Maybe at the conclusion of my p***ing and moaning session, Kirani can put up a sticky post that list all the factors (real ones that you can wrap your mathematical fingers around!) that we come up with.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Sep 15, 2005 8:36 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
There might be more concrete information, believe it or not, in the documentation for 2.1.
I remember it being a lot more detailed on scaling and performance and general administration than the Info Center is.
Of course, how well that information hold through to v5... But there may be some more things we've missed.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| QMan |
| Posted: Fri Sep 16, 2005 6:59 am Post subject: # of Execution Group |
|
|
Novice
Joined: 25 Jun 2003
Posts: 11
|
Hi,
We run on Soralis - 2 CPU - 8 GB memory
MQ, Neon Adapters (sap) and Broker
we have 10 EG - may increase to 12
Total # of FLows ~ 140 - evenly spread - grouped according to application.
I have pondered over this issue and if we go with cpu /eg rule, the EG will become so huge, depoly and contorl is very difficult.
SO far we have no performance issues..
Msg size - very small transactional synchronous for some EG
Very large 10 MB - 40 MB for some EG - these ar batch llike flows - does lots of look ups, DB updates, etc..
_________________
Cheers!
Q Man
Websphere MQ Specialist, MQ Workflow Solun. Expert |
|
| Back to top |
|
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
