| Author |
Message
|
| yupoet |
 Posted: Tue Jul 17, 2012 11:14 pm Post subject: Process amqrmppa takes high CPU rates |
 |
|
Apprentice
Joined: 26 Nov 2008
Posts: 36
|
Platform
AIX
MQ Version
6.0.2.8
Problem
There are 5 amqrmppa processes which occupy 100% CPU time when I checked using TOP command. Then the performance of this MQ is slow.
Observation Points
1. There are lots of SVRCONN running
2. App teams did not perform clean MQCLOSE and MQDISC
3. No high depth queue found
4. No errors found in logs
Question
Can anyone shed some lights on whether something led to any issues and how we can prevent the CPU problem?
Many thanks in advance!
Will attach some screen shot in the following
_________________
With warm regards and thanks!
Last edited by yupoet on Wed Jul 18, 2012 12:08 am; edited 1 time in total |
|
| Back to top |
|
 |
| yupoet |
| Posted: Tue Jul 17, 2012 11:42 pm Post subject: |
|
|
Apprentice
Joined: 26 Nov 2008
Posts: 36
|
Process checking:
ps -efo pcpu,pmem,uid,pid,ppid,comm | sort -r
%CPU %MEM UID PID PPID COMMAND
2.7 0.0 12 11010162 8323180 amqzlaa0_nd
1.0 0.0 12 10485974 11272254 amqrmppa
0.8 0.0 12 13762750 11272254 amqrmppa
0.8 0.0 12 11599958 11272254 amqrmppa
0.8 0.0 12 11337742 11272254 amqrmppa
0.8 0.0 12 8781828 11272254 amqrmppa
0.4 0.0 12 9961632 8323180 amqzlaa0_nd
0.0 1.0 8 6488176 7471336 java
0.0 1.0 0 6750394 4718834 java
0.0 0.0 12 13631656 9502756 amqrmppa
0.0 0.0 12 12451886 8388838 amqzlaa0_nd
0.0 0.0 12 12386336 8388838 amqzlaa0_nd
0.0 0.0 12 12124302 8323180 amqzlaa0_nd
0.0 0.0 12 11796544 8323180 amqzmuc0
0.0 0.0 12 11731156 9502756 amqrmppa
0.0 0.0 12 11468884 9502756 amqrmppa
0.0 0.0 12 11272254 9896058 runmqlsr_nd
0.0 0.0 12 10879004 8323180 amqzfuma
0.0 0.0 12 10616998 8388838 amqzmur0
0.0 0.0 12 10354834 8323180 amqrrmfa
0.0 0.0 12 10158290 8388838 amqzdmaa
0.0 0.0 12 10027028 8388838 amqzmgr0
0.0 0.0 12 9896058 8323180 amqzmgr0
......
ps -efo pmem,uid,pid,ppid,pcpu,comm | sort -r
%MEM UID PID PPID %CPU COMMAND
1.0 8 6488176 7471336 0.0 java
1.0 0 6750394 4718834 0.0 java
0.0 12 13762750 11272254 0.8 amqrmppa
0.0 12 13631656 9502756 0.0 amqrmppa
0.0 12 12451886 8388838 0.0 amqzlaa0_nd
0.0 12 12386336 8388838 0.0 amqzlaa0_nd
0.0 12 12124302 8323180 0.0 amqzlaa0_nd
0.0 12 11796544 8323180 0.0 amqzmuc0
0.0 12 11731156 9502756 0.0 amqrmppa
0.0 12 11599958 11272254 0.8 amqrmppa
0.0 12 11468884 9502756 0.0 amqrmppa
0.0 12 11337742 11272254 0.8 amqrmppa
0.0 12 11272254 9896058 0.0 runmqlsr_nd
0.0 12 11010162 8323180 2.7 amqzlaa0_nd
0.0 12 10879004 8323180 0.0 amqzfuma
0.0 12 10616998 8388838 0.0 amqzmur0
0.0 12 10485974 11272254 1.0 amqrmppa
0.0 12 10354834 8323180 0.0 amqrrmfa
0.0 12 10158290 8388838 0.0 amqzdmaa
0.0 12 10027028 8388838 0.0 amqzmgr0
0.0 12 9961632 8323180 0.4 amqzlaa0_nd
0.0 12 9896058 8323180 0.0 amqzmgr0
0.0 12 9830620 8323180 0.0 amqzlaa0_nd
......
_________________
With warm regards and thanks! |
|
| Back to top |
|
|
| yupoet |
| Posted: Wed Jul 18, 2012 12:04 am Post subject: |
|
|
Apprentice
Joined: 26 Nov 2008
Posts: 36
|
Example of the svrconn channel which are a lot running
1 : dis chs(*) all
AMQ8417: Display Channel Status details.
CHANNEL(aaaaaa) CHLTYPE(SVRCONN)
BUFSRCVD(812954) BUFSSENT(812952)
BYTSRCVD(215644564) BYTSSENT(1379380315)
CHSTADA(2012-07-15) CHSTATI(05.00.42)
COMPHDR(NONE,NONE) COMPMSG(NONE,NONE)
COMPRATE(0,0) COMPTIME(0,0)
CONNAME(aaaaaa) CURRENT
EXITTIME(0,0) HBINT(300)
JOBNAME(00B1005600019C25)
LOCLADDR(::ffff:aaaaaa(1414))
LSTMSGDA(2012-07-1 LSTMSGTI(11.51.22)
MCASTAT(RUNNING) MCAUSER(mapusr)
MONCHL(OFF) MSGS(812951)
RAPPLTAG(WebSphere MQ Client for Java)
RQMNAME( ) SSLCERTI( )
SSLKEYDA( ) SSLKEYTI( )
SSLPEER( ) SSLRKEYS(0)
STATUS(RUNNING) STOPREQ(NO)
SUBSTATE(MQGET) XMITQ( )
AMQ8417: Display Channel Status details.
CHANNEL(aaaaaa) CHLTYPE(SVRCONN)
BUFSRCVD(522325) BUFSSENT(522324)
BYTSRCVD(396758576) BYTSSENT(110210352)
CHSTADA(2012-07-15) CHSTATI(05.00.42)
COMPHDR(NONE,NONE) COMPMSG(NONE,NONE)
COMPRATE(0,0) COMPTIME(0,0)
CONNAME(aaaaaa) CURRENT
EXITTIME(0,0) HBINT(300)
JOBNAME(00A000D60001A287)
LOCLADDR(::ffff:aaaaaa(1414))
LSTMSGDA(2012-07-1 LSTMSGTI(11.51.1
MCASTAT(RUNNING) MCAUSER(mapusr)
MONCHL(OFF) MSGS(522322)
RAPPLTAG(WebSphere MQ Client for Java)
RQMNAME( ) SSLCERTI( )
SSLKEYDA( ) SSLKEYTI( )
SSLPEER( ) SSLRKEYS(0)
STATUS(RUNNING) STOPREQ(NO)
SUBSTATE(RECEIVE) XMITQ( )
_________________
With warm regards and thanks! |
|
| Back to top |
|
|
| exerk |
| Posted: Wed Jul 18, 2012 12:11 am Post subject: Re: Process amqrmppa takes high CPU rates |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| yupoet wrote: |
| 1. There are lots of SVRCONN running |
Please define 'lots', give us a number.
| yupoet wrote: |
| 2. App teams did not perform clean MQCLOSE and MQDISC |
That is certainly not going to help.
How are you running your Listener, i.e. please tell me that it's an object and that you're not running it under inetd?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| yupoet |
| Posted: Wed Jul 18, 2012 12:20 am Post subject: |
|
|
Apprentice
Joined: 26 Nov 2008
Posts: 36
|
Around 200 SVRCONN channels running, and 200 connections when using dis conn(*) to check.
The listener is defined by Define and up along with queue manager.
We did have more than 200 channels running but did not lead to high CPU.
Lately we did have a change to increase system memory from 16GB to 32GB. Then this problem came out.
_________________
With warm regards and thanks! |
|
| Back to top |
|
|
| yupoet |
| Posted: Wed Jul 18, 2012 12:55 am Post subject: |
|
|
Apprentice
Joined: 26 Nov 2008
Posts: 36
|
One more observation: by checking "attribute report for AIX MQ" ,
there were around 300 "Jobs in Minute" in the past, stay at a stable level.
On one day, suddenly it increased to be 700 "Jobs in Minute" and stay at a stable level.
When I check the process using "ps -ef", now there are almost 700.
most of them are amqzlaa0_nd / amqrmppa / amqrrmfa
_________________
With warm regards and thanks! |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Jul 18, 2012 9:03 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
1. Set keepalive on for the queue manager.
2. Get the apps to close and disconnect. This is essential.
3. Apply MAXINST and MAXINSTC values on your svrconn channels.
Maxinstc can protect your QM against too many connections from one single IP address (such as from a rogue app). I set mine to 100. |
|
| Back to top |
|
|
| exerk |
| Posted: Wed Jul 18, 2012 9:11 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| zpat wrote: |
3. Apply MAXINST and MAXINSTC values on your svrconn channels.
Maxinstc can protect your QM against too many connections from one single IP address (such as from a rogue app). I set mine to 100. |
| yupoet wrote: |
MQ Version
6.0.2.8 |
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jul 18, 2012 9:32 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| yupoet wrote: |
| On one day, suddenly it increased to be 700 "Jobs in Minute" and stay at a stable level. |
What does your change management system say was newly implemented on that day?
A new application making new use of the system?
A change to an existing application to add new users or new capacity?
There's no "suddenly" here. There's nothing that will cause this that is not the direct result of a human action. |
|
| Back to top |
|
|
| yupoet |
| Posted: Thu Jul 26, 2012 6:57 pm Post subject: |
|
|
Apprentice
Joined: 26 Nov 2008
Posts: 36
|
MAXINST / MAXINSTC does not apply to MQ v6 I think
_________________
With warm regards and thanks! |
|
| Back to top |
|
|
| yupoet |
| Posted: Thu Jul 26, 2012 9:25 pm Post subject: |
|
|
Apprentice
Joined: 26 Nov 2008
Posts: 36
|
| mqjeff wrote: |
What does your change management system say was newly implemented on that day?
A new application making new use of the system?
A change to an existing application to add new users or new capacity?
There's no "suddenly" here. There's nothing that will cause this that is not the direct result of a human action. |
Sorry that, these info is not transparent to me since the customer keep it highly confidential and I even cannot access the MQ logs / change system / any application facts directly.
I also raised the same queries to clients, but they did not provide the response yet.
What I knew is, many new functions were deployed into the system.
_________________
With warm regards and thanks! |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Jul 26, 2012 10:23 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Check if the apps are repeatedly "polling" the queue, looking for messages at a high rate of MQI calls per second.
Explain to them how to use the MQGMO_WAIT option, or at least put a decent delay in their code between making MQI calls.
Most (new) developers fail to grasp how to use MQ properly. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jul 27, 2012 5:04 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| yupoet wrote: |
I also raised the same queries to clients, but they did not provide the response yet.
What I knew is, many new functions were deployed into the system. |
That right there.
"Many new functions" can very easily be the same thing as "many new connections". |
|
| Back to top |
|
|
|
|
