| Author |
Message
|
| er_pankajgupta84 |
 Posted: Thu Mar 29, 2012 8:20 am Post subject: Number of Execution Groups or Number of flows in 1 EG |
 |
|
Master
Joined: 14 Nov 2008
Posts: 203
Location: charlotte,NC, USA
|
Hi All,
I know this thing have been discussed here a number of times and I have gone thru all those threads.. but all appear to be quite old threads.
Here is my situation.
Currently we are on 6.1 with fixpack 3 on z/OS. In total 4 brokers different LPARs with 24 Execution Group each. Each LPAR has 10 CPs each. Queues are clusterred & shared.
Target -> moving to wmb 7 on z/OS. Same number of brokers but may be different number of EG.
In total we have around 300 flows few of them we are running with multiple instances as well.
With WMB 6.1 we had a constraint of keeping the Address space memory (EG memory) to max of 1.5 GB. So we have created multiple EG (24) and deployed flows on some logical grouping.
Now with version 7 we can have an address space (EG memory) upto 64 Gb so we can reduce the number of execution groups.
I need to collect facts supporting this decision. I would like your expert advice.
Questions:
1. Is it recommended that # of EG = No. of CPs (or CPU). Does that mean 1 EG cannot use all CPs? According to my knowledge 1 address space (1 EG) can use any available CPs. Is there a contradiction to this?
2. Based on the TCB (Thread) limitation of an Address Space (Process or EG) the recommended working limits are 256 threads per process for Windows and UNIX and 230 TCBs per Address Space on z/OS. Let me keep some for buffer, say 30, so Can I say one EG can have 200 flows (single instance & single input node) can run in 1 EG if there is no memory restriction?
3. Would I see any difference, in performance, if I run 200 flows in one EG or run 20 flows in 10 EGs? If yes then what could be the reason. One reason I see is with 200 flows in one EG there would be lot of pagging to accomodate different size of messages. But will it matter?
4. Based on above mention points, one EG can take 200 flows.. so would I see any difference, in performance, if I run 20 flows with 10 instance each in one EG or run 20 flows with one instance each in 10 EGs? If yes then what could be the reason. This question is similar to question 3 but deals with same flow multiple instances.
Other advantages of keeping minimum EG, I see are:
1. Lesser deployment overhead.
2. optimal use of memory as less JVM caches (for routing etc) are maintained.
3. We use Java driven DB connection pooling so lesse conection pools are required.
Please add to this list.
Disadantages of keeping minimum EG, I see are:
1. Single point of failure - not relevant in our case.
Please add to this list.
I know we need to make trade-off which are dependent on lot's of factor. The best way to do this is to run load test for Questions 3 & 4. That I have kept as an excercise for me.
I need your inputs beforehand.
Any documentation / Article is appreciated.
Thanks, |
|
| Back to top |
|
 |
| lancelotlinc |
| Posted: Thu Mar 29, 2012 9:23 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
POWER7 AIX is a better platform for the magnitude of work your EGs are doing. Talk to your client about migrating to POWER7.
Assuming that POWER7 is not attainable, relatively speaking, z/OS performance is three times slower than AIX. Use the performance report support packs to compare performance characteristics:
http://www-01.ibm.com/support/docview.wss?uid=swg27007150
z/OS has a memory utilization issue, which you have discovered. The solution that you are currently operating could be considered by some to be "brute force" method of solving that memory issue. AIX does not have this limitation, especially on modern versions like POWER7.
Load for AIX Message Broker instances on POWER7 is spread across multiple virtual CPUs automatically through a concept called micropartitioning. You can schedule as many as sixty-four virtual CPUs per core (ten cores max).
With a different platform, like POWER7 AIX, you would not need so many EGs.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Mar 29, 2012 9:31 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| lancelotlinc wrote: |
| z/OS performance is three times slower than AIX. |
You have been told at least once that this is a false statement based on misleading comparisons. |
|
| Back to top |
|
|
| er_pankajgupta84 |
| Posted: Thu Mar 29, 2012 9:35 am Post subject: |
|
|
Master
Joined: 14 Nov 2008
Posts: 203
Location: charlotte,NC, USA
|
@lancelotlinc - Your reply doesn't make any sense to me, in the context I asked my question.
Moving to a new hardware in a big organization cannot come out of the blue. We know how many approval we have to take to go to v7 from v6.1.
Again, I am not sure that whether EG in z/OS can consume multiple CPs. The mainframe person over here told me that it will.
If you are not inclined to z/OS then you can share your inputs for question 3 & 4, from my previous post, with respect to POWER 7 and share other pros & cons with respect to more or less number of EGs.
I would hope it would be more or less same for z/OS.
Thanks. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Mar 29, 2012 10:02 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| mqjeff wrote: |
| lancelotlinc wrote: |
| z/OS performance is three times slower than AIX. |
You have been told at least once that this is a false statement based on misleading comparisons. |
To clarify, Your Mileage May Vary {YMMV}, based on my experience, I have noticed that z/OS performance characteristics with regard to broker message flow latency, transactions per second, CPU utilization, memory utilization is three times slower than same running on POWER7.
The performance reports linked above also support this observation, albeit, in spite of the apples disclaimers.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| er_pankajgupta84 |
| Posted: Thu Mar 29, 2012 10:07 am Post subject: |
|
|
Master
Joined: 14 Nov 2008
Posts: 203
Location: charlotte,NC, USA
|
I would appreciate if people don't try to hijack this thread by comparing z/OS with POWER7 or any other hardware.
I need general feedback on the 4 questions I asked. Question 3 & 4 are applicable to all hardware. The result may differ. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Mar 29, 2012 10:09 am Post subject: Re: Number of Execution Groups or Number of flows in 1 EG |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| er_pankajgupta84 wrote: |
| 1. Is it recommended that # of EG = No. of CPs (or CPU). Does that mean 1 EG cannot use all CPs? According to my knowledge 1 address space (1 EG) can use any available CPs. Is there a contradiction to this? |
No. The number of EGs does not have to equal the number of CPUs.
| er_pankajgupta84 wrote: |
| 2. Based on the TCB (Thread) limitation of an Address Space (Process or EG) the recommended working limits are 256 threads per process for Windows and UNIX and 230 TCBs per Address Space on z/OS. Let me keep some for buffer, say 30, so Can I say one EG can have 200 flows (single instance & single input node) can run in 1 EG if there is no memory restriction? |
Yes.
| er_pankajgupta84 wrote: |
| 3. Would I see any difference, in performance, if I run 200 flows in one EG or run 20 flows in 10 EGs? If yes then what could be the reason. One reason I see is with 200 flows in one EG there would be lot of pagging to accomodate different size of messages. But will it matter? |
Yes.
| er_pankajgupta84 wrote: |
| 4. Based on above mention points, one EG can take 200 flows.. so would I see any difference, in performance, if I run 20 flows with 10 instance each in one EG or run 20 flows with one instance each in 10 EGs? If yes then what could be the reason. This question is similar to question 3 but deals with same flow multiple instances. |
Yes, because your traffic pattern does not match your proposed distribution of threads.
| er_pankajgupta84 wrote: |
| Moving to a new hardware in a big organization cannot come out of the blue. |
Which is why you need to start talking about the benefits sooner rather than later.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Mar 29, 2012 10:12 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| er_pankajgupta84 wrote: |
I would appreciate if people don't try to hijack this thread by comparing z/OS with POWER7 or any other hardware.
I need general feedback on the 4 questions I asked. Question 3 & 4 are applicable to all hardware. The result may differ. |
I'm not hijacking your thread. The questions you posed relate directly to the performance issues with the hardware platform you are on. If the z/OS architecture is the cause of your problem, you won't adequately address this if you don't talk about the hardware.
You are not the first WMB consultant at the client. A dozen others before you have addressed the same questions you are raising. Talk about the potential solutions, not reasons why politics stand in the way.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER
Last edited by lancelotlinc on Thu Mar 29, 2012 10:49 am; edited 1 time in total |
|
| Back to top |
|
|
| er_pankajgupta84 |
| Posted: Thu Mar 29, 2012 10:31 am Post subject: |
|
|
Master
Joined: 14 Nov 2008
Posts: 203
Location: charlotte,NC, USA
|
lancelotlinc wrote:
| Quote: |
er_pankajgupta84 wrote:
1. Is it recommended that # of EG = No. of CPs (or CPU). Does that mean 1 EG cannot use all CPs? According to my knowledge 1 address space (1 EG) can use any available CPs. Is there a contradiction to this?
No. The number of EGs does not have to equal the number of CPUs. |
Can it be less or more? I think with whatever explanation you have given for POWER7 micropartitioning it seems only one EG is sufficient (Keeping single point of failure out of picture)
lancelotlinc wrote:
| Quote: |
er_pankajgupta84 wrote:
3. Would I see any difference, in performance, if I run 200 flows in one EG or run 20 flows in 10 EGs? If yes then what could be the reason. One reason I see is with 200 flows in one EG there would be lot of pagging to accomodate different size of messages. But will it matter?
Yes.
|
Why?
lancelotlinc wrote:
| Quote: |
er_pankajgupta84 wrote:
4. Based on above mention points, one EG can take 200 flows.. so would I see any difference, in performance, if I run 20 flows with 10 instance each in one EG or run 20 flows with one instance each in 10 EGs? If yes then what could be the reason. This question is similar to question 3 but deals with same flow multiple instances.
Yes, because your traffic pattern does not match your proposed distribution of threads. |
Can you elaborate on this? I have same flow deployed, same size of messages coming and when one EG can use all the available CPUs then why is this difference?
lancelotlinc wrote:
| Quote: |
er_pankajgupta84 wrote:
I would appreciate if people don't try to hijack this thread by comparing z/OS with POWER7 or any other hardware.
I need general feedback on the 4 questions I asked. Question 3 & 4 are applicable to all hardware. The result may differ.
I'm not hijacking your thread. The questions you posed relate directly to the performance issues with the hardware platform you are on. If the z/OS architecture is the cause of your problem, you won't adequately address this if you don't talk about the hardware.
You are not the first WMB consultant at the bank. A dozen others before you have addressed the same questions you are raising. Talk about the potential solutions, not reasons why politics stand in the way. |
I am not denying any of your observation about POWER7 or z/OS. I am saying leadership do listen to all these suggestions it just they can give a go ahead in seconds.
I hope you are aware of the obvious concerns.
Thanks.[/code] |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Mar 29, 2012 10:37 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
>>Can you elaborate on this? I have same flow
>> deployed, same size of messages coming
>> and when one EG can use all the available
>>CPUs then why is this difference?
Your client has the concept of tracking state of a transaction inside WMB. An original message can be re-put to the same queue as many as a seven times before it has completed processing. Your client is trying to manage both state and orchestration within a stateless Enterprise Service Bus. Due to this traffic pattern, you have to duplicate effort across EGs and across flows. Not a high-performance implementation to say the least.
Another problem your client has is the fact that they constantly use external parsers to reparse data outside of the MbElement context. Adding multiple parsers into the JRE within WMB is another reason why the memory consumption is so high.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| er_pankajgupta84 |
| Posted: Thu Mar 29, 2012 10:54 am Post subject: |
|
|
Master
Joined: 14 Nov 2008
Posts: 203
Location: charlotte,NC, USA
|
I would like to keep the client message pattern out of the picture here.
Whatever I am asking are general questions.
I asked a questions -
| Quote: |
| Will the same flow deployed in multiple EG perform better than it is deployed with multiple instance in the 1 EG. |
You said "Yes" earlier. When I asked the reason, you mentioned this -
lancelotlinc wrote:
| Quote: |
>>Can you elaborate on this? I have same flow
>> deployed, same size of messages coming
>> and when one EG can use all the available
>>CPUs then why is this difference?
|
I interpret this comment as "No" as an answer to the question that I asked. So I am confused with your answer.
If your answer is "Yes" then I would like to know the reasons. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Mar 29, 2012 10:58 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| er_pankajgupta84 wrote: |
I would like to keep the client message pattern out of the picture here.
Whatever I am asking are general questions.
I asked a questions -
| Quote: |
| Will the same flow deployed in multiple EG perform better than it is deployed with multiple instance in the 1 EG. |
You said "Yes" earlier. When I asked the reason, you mentioned this -
lancelotlinc wrote:
| Quote: |
>>Can you elaborate on this? I have same flow
>> deployed, same size of messages coming
>> and when one EG can use all the available
>>CPUs then why is this difference?
|
I interpret this comment as "No" as an answer to the question that I asked. So I am confused with your answer.
If your answer is "Yes" then I would like to know the reasons. |
>>I would like to keep the client message
>> pattern out of the picture here.
Again, message pattern drives to root cause. Hardware architecture drives to root cause. If you want to eliminate consideration of root causes, and only treat the symptoms, I'm not sure you are addressing your Raison d'etre.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| er_pankajgupta84 |
| Posted: Thu Mar 29, 2012 11:31 am Post subject: |
|
|
Master
Joined: 14 Nov 2008
Posts: 203
Location: charlotte,NC, USA
|
It seems like I myself have hijacked my thread.
The best way to solve a problem is to break it in chunks and then work on it. I don't deny the idea of eliminating the root cause.
I didn't mention that I have a memory problem. I asked simple questions that people might have answers on. I can get answers to these questions myself after conducting the tests. But I am not in position to execute them right away.
I still didn't get your clarification on my question:
| Quote: |
| Will the same flow deployed with single instance in multiple (say 5) EGs performs better than if it is deployed with multiple instances (Say 5) in 1 EG. |
You mentioned "Yes". But why? |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Mar 29, 2012 11:35 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| er_pankajgupta84 wrote: |
| Quote: |
| Will the same flow deployed with single instance in multiple (say 5) EGs performs better than if it is deployed with multiple instances (Say 5) in 1 EG. |
You mentioned "Yes". But why? |
By the way, I enjoy our discussion. Thanks for asking questions.
1 EG = 1 process. Speaking strictly in general terms, platform agnostically, and based on my own observations (thank you Vitor), a multi-process multi-threaded solution will outperform a single-process, multi-threaded solution.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| er_pankajgupta84 |
| Posted: Thu Mar 29, 2012 11:45 am Post subject: |
|
|
Master
Joined: 14 Nov 2008
Posts: 203
Location: charlotte,NC, USA
|
lancelotlinc wrote:
| Quote: |
| 1 EG = 1 process. Speaking strictly in general terms, platform agnostically, and based on my own observations (thank you Vitor), a multi-process multi-threaded solution will outperform a single-process, multi-threaded solution. |
That will hold true if there is a difference between how "Multiple Processes" uses the Available CPs & "Single Process" uses them.
I am looking for a documentation that support the above statement.
Does this statement holds good in POWER7? |
|
| Back to top |
|
|
|
|
