| Author |
Message
|
| ethanlokesh |
 Posted: Tue Sep 18, 2012 3:00 am Post subject: What is the maximum number of clusters on a Queue Manager? |
 |
|
Novice
Joined: 04 Sep 2012
Posts: 12
|
Hi All,
We have two RHCS CLUSTERS QMs having 253 full repos each and total accounting 453 and 455 clusters.
The repos process amqrrmfa is consuming almost 98% CPU utilisation due to which the cluster commands are not processed.  PFB the snapshopt of the utilizations.
3158 mq 14 -1 194m 70m 68m S 93.6 0.2 321:40.00 amqzmuc0
31874 mq 24 -1 371m 223m 222m R 90.3 0.7 390:46.28 amqrrmfa
Simultaneously We are getting FDCs with probeID ZS402020 and ZS402099(occurs frequently) RM193001 XC307070 and XC308010. It seems that the above ProbeIDs have pattern.
Earlier MQ version was 7013 and now we upgraded to 7018.
I am suspecting it is due to the high amount of cluster associated with these QMs.  Please Help. 
Regards,
Ethan
Quote: Life is ofcourse beautiful without Problems but Problem helps you in making your life beautiful. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Tue Sep 18, 2012 4:41 am Post subject: Re: What is the maximum number of clusters on a Queue Manage |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ethanlokesh wrote: |
| We have two RHCS CLUSTERS QMs having 253 full repos each and total accounting 453 and 455 clusters. |
I do hope you mean there are 253 full repos spread across the the clusters not that 253 of the queue managers in a cluster are full repos!
| ethanlokesh wrote: |
| I am suspecting it is due to the high amount of cluster associated with these QMs. |
You think? You have a huge number of overlapping clusters with a huge number of full repositories and you think that's why the repos process is going crazy trying to keep it all in sync???? 
1) Why so many clusters
2) Why so many full repositories (however they're distributed)
3) What is the total number of queue managers involved?
4) What made someone think this topology was a good idea?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Sep 18, 2012 12:52 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
 :scratch: How the .... do you keep all those clusters apart and correctly configured over time??? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| ramires |
| Posted: Tue Sep 18, 2012 12:58 pm Post subject: |
|
|
Knight
Joined: 24 Jun 2001
Posts: 523
Location: Portugal - Lisboa
|
| fjb_saper wrote: |
| :scratch: How the .... do you keep all those clusters apart and correctly configured over time??? |
MQ magic? Or MQ is really good ? 
_________________
Obrigado / Thanks you |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Sep 19, 2012 4:53 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Moved to Clustering forum.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| ethanlokesh |
| Posted: Wed Sep 19, 2012 5:50 am Post subject: Re: What is the maximum number of clusters on a Queue Manage |
|
|
Novice
Joined: 04 Sep 2012
Posts: 12
|
| Vitor wrote: |
I do hope you mean there are 253 full repos spread across the the clusters not that 253 of the queue managers in a cluster are full repos! |
Yes there are 253 full Repos spread across the cluster
| Vitor wrote: |
You think? You have a huge number of overlapping clusters with a huge number of full repositories and you think that's why the repos process is going crazy trying to keep it all in sync???? |
Yes its because there can be 255 Clusters in a namelist as per IBM standards so we had 253 cluster in a namelist till now and those clusters are acting as a full repos.
| Vitor wrote: |
1) Why so many clusters
2) Why so many full repositories (however they're distributed)
|
because there can be 255 Clusters in a namelist as per IBM standards
| Vitor wrote: |
3) What is the total number of queue managers involved? |
Total 2093 clusqmgrs involved
| Vitor wrote: |
4) What made someone think this topology was a good idea? |
This topology was followed since beginning.
But the namelist can have 255 clusters so why not it can have 253 clusters as per IBM standards. So how can you say that it may be the cause of high CPU utilsation? Confused 
Regards,
Ethan |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Sep 19, 2012 5:54 am Post subject: Re: What is the maximum number of clusters on a Queue Manage |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| ethanlokesh wrote: |
| Vitor wrote: |
I do hope you mean there are 253 full repos spread across the the clusters not that 253 of the queue managers in a cluster are full repos! |
Yes there are 253 full Repos spread across the cluster
| Vitor wrote: |
You think? You have a huge number of overlapping clusters with a huge number of full repositories and you think that's why the repos process is going crazy trying to keep it all in sync???? |
Yes its because there can be 255 Clusters in a namelist as per IBM standards so we had 253 cluster in a namelist till now and those clusters are acting as a full repos.
| Vitor wrote: |
1) Why so many clusters
2) Why so many full repositories (however they're distributed)
|
because there can be 255 Clusters in a namelist as per IBM standards
| Vitor wrote: |
3) What is the total number of queue managers involved? |
Total 2093 clusqmgrs involved
| Vitor wrote: |
4) What made someone think this topology was a good idea? |
This topology was followed since beginning.
But the namelist can have 255 clusters so why not it can have 253 clusters as per IBM standards. So how can you say that it may be the cause of high CPU utilsation? Confused
Regards,
Ethan |
The maximum number of cluster names in a namelist is not a rationale for having 250+ clusters. This seems to be your answer to the question 'why so many clusters.'
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Sep 19, 2012 5:59 am Post subject: Re: What is the maximum number of clusters on a Queue Manage |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| ethanlokesh wrote: |
The repos process amqrrmfa is consuming almost 98% CPU utilisation due to which the cluster commands are not processed. |
Did you kill the amqrrmfa process in an attempt to fix the 98% CPU issue? If so, this would result in the FDC.
When did the 98% CPU utilization begin? Did it suddenly increase to 98%? If so, what changed?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| ethanlokesh |
| Posted: Wed Sep 19, 2012 6:05 am Post subject: Re: What is the maximum number of clusters on a Queue Manage |
|
|
Novice
Joined: 04 Sep 2012
Posts: 12
|
| bruce2359 wrote: |
Did you kill the amqrrmfa process in an attempt to fix the 98% CPU issue? If so, this would result in the FDCWhen did the 98% CPU utilization begin? Did it suddenly increase to 98%? If so, what changed? |
FDC's are getting generated before killing 98% utilisation. btw killing the process does not solve the problem. Bouncing the QM solves the problem but in next two hours the utilisation again goes high.
Regards,
Ethan
Quote: Life is ofcourse beautiful without Problems but Problem helps you in making your life beautiful |
|
| Back to top |
|
|
| exerk |
| Posted: Wed Sep 19, 2012 6:06 am Post subject: Re: What is the maximum number of clusters on a Queue Manage |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| ethanlokesh wrote: |
| Yes its because there can be 255 Clusters in a namelist as per IBM standards so we had 253 cluster in a namelist till now and those clusters are acting as a full repos. |
This is a meaningless statement, please clarify what you mean.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Sep 19, 2012 6:08 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Was CPU utilization always high? If not, what changes were made just prior to CPU going to 98%?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Sep 19, 2012 6:43 am Post subject: Re: What is the maximum number of clusters on a Queue Manage |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ethanlokesh wrote: |
| Vitor wrote: |
I do hope you mean there are 253 full repos spread across the the clusters not that 253 of the queue managers in a cluster are full repos! |
Yes there are 253 full Repos spread across the cluster |
You're mad. Utterly, utterly crazy. 3 FRs in a cluster is an exception case. Clusters have 2 FRs. No wonder the repos process is going berzerk.
| ethanlokesh wrote: |
| Vitor wrote: |
You think? You have a huge number of overlapping clusters with a huge number of full repositories and you think that's why the repos process is going crazy trying to keep it all in sync???? |
Yes its because there can be 255 Clusters in a namelist as per IBM standards so we had 253 cluster in a namelist till now and those clusters are acting as a full repos. |
Rubbish. The fact that the parameter happens to allow that number of entries (because 255 is a magic number) doesn't mean you should use it. I bet the dial in your car goes up to over 100mph, but that doesn't mean it's a good idea to drive at that speed.
| ethanlokesh wrote: |
| Vitor wrote: |
1) Why so many clusters
2) Why so many full repositories (however they're distributed)
|
because there can be 255 Clusters in a namelist as per IBM standards |
And because there can be you feel there must be? Or should be?
| ethanlokesh wrote: |
| Vitor wrote: |
3) What is the total number of queue managers involved? |
Total 2093 clusqmgrs involved |
So large but not excessive then.
| ethanlokesh wrote: |
| Vitor wrote: |
4) What made someone think this topology was a good idea? |
This topology was followed since beginning. |
So at the beginning someone thought this was a good idea - why? What requirement were they meeting? Or are you just doing it this way because it's always been done this way and that's much easier than thinking?
| ethanlokesh wrote: |
| But the namelist can have 255 clusters so why not it can have 253 clusters as per IBM standards. So how can you say that it may be the cause of high CPU utilsation? Confused |
See above. The average database can support a very large number of columns in a table, and an even larger numbe of rows. This doesn't stop the CPU required for a query increasing exponentially as you increase both.
I would also like you to post the link where IBM says their "standard" is that the namelist should contain almost the maximum number of entries. Not the link where it says it can contain that number.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| ethanlokesh |
| Posted: Wed Sep 19, 2012 7:31 am Post subject: Re: What is the maximum number of clusters on a Queue Manage |
|
|
Novice
Joined: 04 Sep 2012
Posts: 12
|
| Vitor wrote: |
| ethanlokesh wrote: |
| Vitor wrote: |
I do hope you mean there are 253 full repos spread across the the clusters not that 253 of the queue managers in a cluster are full repos! |
Yes there are 253 full Repos spread across the cluster |
You're mad. Utterly, utterly crazy. 3 FRs in a cluster is an exception case. Clusters have 2 FRs. No wonder the repos process is going berzerk.
|
Sorry I misunderstood the term what I mean to say is one QM is having 253 Clusters in the namelist. So this QM is used as a full repos for those 253 clusters.
| Vitor wrote: |
I would also like you to post the link where IBM says their "standard" is that the namelist should contain almost the maximum number of entries. Not the link where it says it can contain that number. |
See we are using REPOSNL in a Queue Manager so a Namelist in a reposnl can take 255 clusternames the max limit which I have personally tested.
Regards,
Ethan
Quote: Life is ofcourse beautiful without Problems but Problem helps you in making your life beautiful. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Sep 19, 2012 7:48 am Post subject: Re: What is the maximum number of clusters on a Queue Manage |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| ethanlokesh wrote: |
See we are using REPOSNL in a Queue Manager so a Namelist in a reposnl can take 255 clusternames the max limit which I have personally tested.
|
There is no reason to have tested this, as it is documented as the maximum number of entries in a namelist.
Is your business requirement to have the maximum number of cluster names that can fit in a namelist? Is that how your organization decided on the number of clusters? Or, is the business requirement to have clusters that meet a non-WMQ technical specification?
For example: does your organization have 255 grocery stores, where each store is a uniquely named cluster? What happens when your organization opens store number 256, 257, 258?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Sep 19, 2012 8:48 am Post subject: Re: What is the maximum number of clusters on a Queue Manage |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| bruce2359 wrote: |
| ethanlokesh wrote: |
See we are using REPOSNL in a Queue Manager so a Namelist in a reposnl can take 255 clusternames the max limit which I have personally tested.
|
There is no reason to have tested this, as it is documented as the maximum number of entries in a namelist. |
I repeat that the fact the namelist can take that number of entries doesn't mean that it should take that number of entries.
I'm still waiting for that link you found where this is the IBM standard practice.
2 key points I think:
1) Running 253 clusters is an administrative nightmare unless the topology is so simplistic you don't really need that number of clusters.
2) You started out by asking if the large number of clusters is the cause of the high CPU in the repos processes. It clearly is. So your choices as I see them are:
- Accept this is an outcome of your design and suck it up, with a plan to move to bigger hardware
- Raise a PMR and report unacceptably poor performance in this scenario & see if IBM can fix the processes in a future release
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
