| Author |
Message
|
| mqlover |
 Posted: Wed Apr 30, 2014 2:03 am Post subject: amqrmppa and amqzlaa0 processes are consuming high CPU |
 |
|
Disciple
Joined: 25 Jul 2010
Posts: 176
|
Hi,
I have seen this topic being discussed quite few sometime earlier but still I havent got any solution to resolve the above problem.
There are 2 qmgrs which are full repos and are in a cluster. These qmgrs process i.e amqrmppa and amqzlaa0 are consuming lot of CPU.
I have restarted the qmgr quite a few times, killed these processes and also listener but still nothing seems to be working.
I checked there arent many SVRCONN connections to the qmgrs.
No messages piling up in any of the queues
Kindly anybody can provide me solution for this?
Thanks in advance |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Wed Apr 30, 2014 6:33 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
How many qmgrs in the cluster? Any big changes happening in the cluster that would have lots of communications to the FRs? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Apr 30, 2014 8:11 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
Define "lots" of CPU.
Whats in the MQ error logs?
What do the FDC files say?
Please don't go around killing processes. Often it can make more problems than it solves.
A QM with active QM to QM channels that is processing lots messages as fast as it can is going to drive the CPU on a server.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| PaulClarke |
| Posted: Wed Apr 30, 2014 8:22 am Post subject: |
|
|
Grand Master
Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand
|
As I'm sure you know the amqrmppa processes are the processes which move messages - ie. the channels. The amqzlaa0 processes are the agent processes. If I saw 'high' CPU on these processes my first thought would be that I was just moving a lot of messages down my channels.
Have you ruled this out ? You seem to be suggesting that the channels are misbehaving in some way, can you say what leads you to this conclusion.
If you issue use Explorer/MO71 to display the sequence number of all running channels and then refresh the display every few seconds, do you see the numbers go up ? If so, by how many. In other words, roughly speaking, how many messages per second are you transmitting/receiving.
If the number is fairly large then you can use RESET QSTATS or equivalent to work out which queues are being used heavily. From there you can use DIS CONN(*) etc to see which applications are involved.
Of course it is possible that you have a looping message. Something that I have done in performance testing is define two remote queues that 'point' to each other. You can then inject a few messages and they will bounce backwards and forwards between the two Queue Managers as fast as possible. Handy for testing but can be confusing if done by accident.
Like Peter says, don't kill MQ processes. It should be a last resort. If you suspect a channel problem then it should be possible to find out which are having problems and, stop them manually, if necessary. Looking at the transmission queue would also be a good way of seeing what these messages were and where they were coming from,
Cheers,
Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com |
|
| Back to top |
|
|
| mqlover |
| Posted: Sun May 04, 2014 9:49 pm Post subject: |
|
|
Disciple
Joined: 25 Jul 2010
Posts: 176
|
Thanks for the reply and sorry for the delay.
Well, lots of CPU i meant is 2 processes of each qmgr i.e 4 process taking 25% of CPU each.
I could see the nujmber of messages on channels increasing upto 30k-40k.
Even the message sequence number increasing.
There are no FDC generated. Error logs are not explaining anything which I can understand.
I tried using RESET QMGR TYPE(STATISTICS) but doesnt seem to working as its the same.
What else approach can I take?
Thanks much for detailed explainations all this while. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun May 04, 2014 10:32 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
In your flow check the delivery count. Route anything above the threshold to the failure flow. You do need to take care of poison messages...
Run a debug level trace. This should show you if your message is in an endless process loop. Sometimes quite easy to program... just missing 1 line of code right?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| PaulClarke |
| Posted: Sun May 04, 2014 11:04 pm Post subject: |
|
|
Grand Master
Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand
|
Well, if you have two channels constantly moving messages and nothing else going on in the system then it does sound to me as though you have two channels looping.
What I would do is.....
- Stop the channels in question.
- Issue a resolve channel action(backout)
- Then modify the transmission queues for GET(ENABLE)
- Then browse the transmission queues and see what the messages are. Look at the XQH header on the front of the message and see where they are destined.
- Then look at the definition of that queue and ensure that it is not a remote queue pointing right back where it came from. Naturally I would use MO71 for this but I suspect you could use Explorer. Other loops can be caused by default transmission queues pointing to the 'other' Queue Manager.
- If you do have looping definitions then fix the queue definitions
- Lastly, you can delete the messages off the transmission queue or move them to a different queue or unload them to a file etc, again I would use MO71 for this.
Hope that helps,
Cheers,
Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com |
|
| Back to top |
|
|
| mqlover |
| Posted: Mon May 05, 2014 12:04 am Post subject: |
|
|
Disciple
Joined: 25 Jul 2010
Posts: 176
|
| Thanks for your suggestions. I shall try the mentioned steps and update the result. |
|
| Back to top |
|
|
| mqlover |
| Posted: Wed May 21, 2014 6:05 pm Post subject: |
|
|
Disciple
Joined: 25 Jul 2010
Posts: 176
|
Thanks for all your suggestions. Finally I could solve it. There was one of the channels which was taking more of resources. Could zero in only after observing the number of msgs coming and passing in through which channel.
Thanks again for your help. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu May 22, 2014 12:56 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9396
Location: US: west coast, almost. Otherwise, enroute.
|
How did you solve this? What exactly did you do?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
