| Author |
Message
|
| dtauzell |
 Posted: Thu Oct 22, 2009 6:15 am Post subject: High CPU usage when two servers are processing messages |
 |
|
Apprentice
Joined: 23 May 2002
Posts: 37
|
Scenario:
Several queues, each with hundreds of thousands of messages on them.
Start up JBoss application server with MDBs that pull messages from those queues and then write to a database.
Message are being pulled off and CPU usage on the Queue Manager is low (around 2-3%)
Start up a second JBoss application server with second set of the MDBs pulling messages.
CPU on Queue Manager goes to 50-90% utilization and message are pulled off very slowly.
Reproduced in Development and Production.
MQ Version is:
Name: WebSphere MQ
Version: 6.0.2.2
CMVC level: p600-202-070801
BuildType: IKAP - (Production)
I'm going to open a support ticket, too, but I'm curious if anybody ever ran into this. |
|
| Back to top |
|
 |
| Sam Uppu |
| Posted: Thu Oct 22, 2009 6:21 am Post subject: |
|
|
Yatiri
Joined: 11 Nov 2008
Posts: 610
|
I dont think its an MQ issue. It could be the JBoss app server as there are two instances running. Can you see which PIDs related which app are consuming more CPU resources?. You may need SA to investigate further on this.
Thanks. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Oct 22, 2009 6:23 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Have you confirmed the extra CPU is being used by the queue manager? If so, which specific queue manager process is using it?
Why (from a design perspective) are you using 2 app servers rather than spinning up more MDbs on the first one?
Why (from a wider design perspective) split the lot over 2 physical servers and use a WMQ cluster?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mvic |
| Posted: Fri Oct 23, 2009 8:31 am Post subject: Re: High CPU usage when two servers are processing messages |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| dtauzell wrote: |
| Several queues, each with hundreds of thousands of messages on them. |
MQ does not perform at its best when it has deep queues. Ensure your getter applications have sufficient resources to keep the queues shallow. It'll save you from this sort of pain. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Oct 23, 2009 11:11 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| MQ does not perform at its best when it has deep queues. |
Yes, but I would not expect the mere presence of deep queues to dramatically increase CPU utilization. (Deep Queue... what a great name for a blog about MQ... if only someone...)
Not enough info in the post to make any real assessment.
What is using lots of CPU? Is it MQ? Is the app doing DB work on the same box?
Are the messages big (100meg) or small (1k)?
Is this a new problem? What changed?
Is the box configured with sufficient RAM to reduce paging and its overheads? Sufficient processor resources?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| mvic |
| Posted: Mon Oct 26, 2009 2:48 am Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| bruce2359 wrote: |
| Quote: |
| MQ does not perform at its best when it has deep queues. |
Yes, but I would not expect the mere presence of deep queues to dramatically increase CPU utilization. |
Eliminate the deep queues, you eliminate a large number of reasons for MQ to perform poorly.
OP said they had hundreds of thousands of messages on each of several queues. Not much more needs to be said.
Deep queues can happen on systems where there was a big failure in the draining application(s). It can also happen where the design is for MQ to hold large amounts of data waiting for a big batch job. Don't know if this case falls into either of these descriptions though.
Another thing to remember is, if the application uses selectors with JMS then MQ V6 is not well optimised for this type of application. MQ V7 has better optimisations for this scenario though - worth taking a look. |
|
| Back to top |
|
|
|
|
