| Author |
Message
|
| jeevan |
 Posted: Thu Feb 05, 2009 2:00 pm Post subject: AMQRMPPA processes |
 |
|
Grand Master
Joined: 12 Nov 2005
Posts: 1432
|
I got incident reported by our recovery team this morning for one of the mq boxes in production. The alert they were getting in patrol was that channel were full. But when I chekced, there were only 1200 channel while the max channels are 5000. when I checked the CPU uses, it was 100%..
I saw one processes running in many instances was AMQRMPPA. I am not sure how many of its instances are normal, but MQ serirs discussions seems little consufed.
Does it only affect while shutting down the queue manager or does it affect while running as running many instances of this processes? I am not sure, one of the posts says that there can be upto 5 instances of it.
Can some one shed light on it.
thanks
Last edited by jeevan on Thu Feb 05, 2009 3:14 pm; edited 1 time in total |
|
| Back to top |
|
 |
| PeterPotkay |
| Posted: Thu Feb 05, 2009 2:06 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
amqrmppa can have up to 64 threads running. So for every 64 active channels you have, you will have at least 1 amqrmppa process.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| jeevan |
| Posted: Thu Feb 05, 2009 3:09 pm Post subject: |
|
|
Grand Master
Joined: 12 Nov 2005
Posts: 1432
|
| PeterPotkay wrote: |
| amqrmppa can have up to 64 threads running. So for every 64 active channels you have, you will have at least 1 amqrmppa process. |
The follwoing is the part of the log.
2/5/2009 10:03:49 - Process(16328.5565) User(XXXXXXXX) Program(amqrmppa.exe)
AMQ9573: Maximum number of active channels reached.
EXPLANATION:
There are too many channels active to start another. The current defined
maximum number of active channels is 5000.
ACTION:
Either wait for some of the operating channels to close or use the stop channel
command to close some channels. Retry the operation when some channels are
available. The maximum number of active channels is a configurable parameter in
the queue manager configuration file.
Actually, the number of channels were just about 1200 but it loggs like this. Also, patrol generated an alert which should be generated if the channel reaches 5000.
I appreciate any explanation of this |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Feb 05, 2009 3:17 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Was the number of channels at 1200 at exactly 10:03:49? Or was it at 1200 at some other time, maybe very close to 10:03:49, and you are assuming it stays at 1200 and never rises to 5000?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| ramires |
| Posted: Thu Feb 05, 2009 3:18 pm Post subject: |
|
|
Knight
Joined: 24 Jun 2001
Posts: 523
Location: Portugal - Lisboa
|
if it says the maximum number was reached, probably it's because it was reached. Do a display for the number of channel status in that moment and count the number, or with a netstat look for the number of established connections.
Regards
joao |
|
| Back to top |
|
|
| jeevan |
| Posted: Thu Feb 05, 2009 3:45 pm Post subject: |
|
|
Grand Master
Joined: 12 Nov 2005
Posts: 1432
|
| PeterPotkay wrote: |
| Was the number of channels at 1200 at exactly 10:03:49? Or was it at 1200 at some other time, maybe very close to 10:03:49, and you are assuming it stays at 1200 and never rises to 5000? |
When I checked probably, it was not exaclty 10:03:49 am. However, I am pretty sure, it nevers reaches to 5000 as the request from sales registers are distributed equally among 6 queue managers. Normally, each qmgr will have 22-2500 connections.
In fact, earlier, it says, the channel has ended abnormally.
----- amqrmrsa.c : 468 --------------------------------------------------------
2/5/2009 09:41:34 - Process(79532.567) User(XXXXXXXX) Program(amqrmppa.exe)
AMQ9573: Maximum number of active channels reached.
EXPLANATION:
There are too many channels active to start another. The current defined
maximum number of active channels is 5000.
ACTION:
Either wait for some of the operating channels to close or use the stop channel
command to close some channels. Retry the operation when some channels are
available. The maximum number of active channels is a configurable parameter in
the queue manager configuration file.
----- amqrcsia.c : 1225 -------------------------------------------------------
2/5/2009 09:41:34 - Process(79532.567) User(XXXXXXXX) Program(amqrmppa.exe)
AMQ9999: Channel program ended abnormally.
EXPLANATION:
Channel program 'SYSTEM.DEF.SVRCONN' ended abnormally.
ACTION:
Look at previous error messages for channel program 'SYSTEM.DEF.SVRCONN' in the
error files to determine the cause of the failure.
----- amqrmrsa.c : 468 --------------------------------------------------------
When it says,
AMQ9999: Channel program ended abnormally.
does it probably referring a new channel connection? I never restarted the channle again and it is running now.
So, I am again confused and thinking whether too many opened AMQRMPPA proceeses make it feel that there are all channel fulled. I am not sure though. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Feb 05, 2009 7:28 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
OK ---
First off, have your applications connect on a different channel and not on SYSTEM.DEF.SVRCONN. This channel should run with an mcauser of 'none' so that it is unusable. It would be the first vector of an attack against the qmgr.
Second if the connection gets flaky between clients and the server the constant retry of the the clients can very quickly exhaust the max active channels as channels are created but discarded because of tcp/ip problems by the client. The qmgr is never notified and keeps the channel alive for the time of the tcp timeout.... Having each app use a different channel allows you to force shutdown that channel allowing others to still play nice.
V7 has now I believe a way of limiting the number of connections to a svrconn channel.
This said experience told us, that when a network glitch happens and all channels are busy, force shutting down the offending channel and restarting it is the fastest way to get back on your feet after you have resolved the original cause.....
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| masteringmq |
| Posted: Fri Feb 06, 2009 12:52 am Post subject: |
|
|
Master
Joined: 20 Oct 2008
Posts: 200
|
| Quote: |
| amqrmppa can have up to 64 threads running. So for every 64 active channels you have, you will have at least 1 amqrmppa process. |
Even with one channel running, I can see more than 1 amqrmppa process running. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Feb 06, 2009 2:44 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| masteringmq wrote: |
| Quote: |
| amqrmppa can have up to 64 threads running. So for every 64 active channels you have, you will have at least 1 amqrmppa process. |
Even with one channel running, I can see more than 1 amqrmppa process running. |
Hence "at least 1" 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| SAFraser |
| Posted: Fri Feb 06, 2009 12:12 pm Post subject: |
|
|
Shaman
Joined: 22 Oct 2003
Posts: 742
Location: Austin, Texas, USA
|
| I am curious to know how you arrived at your channel count of 1200.... There are several ways to to so, just wondering what you did? |
|
| Back to top |
|
|
| jeevan |
| Posted: Fri Feb 06, 2009 1:22 pm Post subject: |
|
|
Grand Master
Joined: 12 Nov 2005
Posts: 1432
|
| SAFraser wrote: |
| I am curious to know how you arrived at your channel count of 1200.... There are several ways to to so, just wondering what you did? |
It was my colleague who was on call. He was reported by recovery team for an alert on patrol for channel full.
When my colleage asked me to look at, first i checked the activecount for the channel in Patrol Operator and it was showing only 1201 (exact number) but the alert message was saying channel full.
When I than start checking other stuff such as resources cpu, and memory, I got cpu taken 100%, and when I chekced the processes, I saw many many amqrmppa processes.
The alert was false anyway as the actual channel could not be used up as the max in our long observation would go upto 2500 per queue amanger. In case of them go down, it may go upto 3000 at max. the process of connecting to these queue maangers from sales registers are searching randomly. So, I could not think that the channel at any point could have reached 5000 but that the patrol alert and the log also tell us.
2/5/2009 10:03:50 - Process(16328.5566) User(xxxxxxxx) Program(amqrmppa.exe)
AMQ9573: Maximum number of active channels reached.
EXPLANATION:
There are too many channels active to start another. The current defined
maximum number of active channels is 5000.
ACTION:
Either wait for some of the operating channels to close or use the stop channel
command to close some channels. Retry the operation when some channels are
available. The maximum number of active channels is a configurable parameter in
the queue manager configuration file.
That is why I was confused and still am confused.
The channel is running now fine with a connection around 2200.
What I did was disabled the nav, and also restarted patrol agent which was taking a lot of cpu for some reasons which I need to investigate later on.
please feel free to ask more questions if you like |
|
| Back to top |
|
|
| SAFraser |
| Posted: Fri Feb 06, 2009 1:37 pm Post subject: |
|
|
Shaman
Joined: 22 Oct 2003
Posts: 742
Location: Austin, Texas, USA
|
The max active channel parameter applies to the aggregate of all channel connections (all types of channels) for that queue manager, not just the one channel in question.
In such a case, I would check (directly on the MQ server, not through my monitoring tool) the total number of channel connections on the queue manager. And as a previous poster suggested, I would then look at the output of netstat to see if the numbers made sense.
Not sure what you can do at this point, but perhaps others will have some additional ideas for you! |
|
| Back to top |
|
|
| jeevan |
| Posted: Fri Feb 06, 2009 1:43 pm Post subject: |
|
|
Grand Master
Joined: 12 Nov 2005
Posts: 1432
|
| SAFraser wrote: |
The max active channel parameter applies to the aggregate of all channel connections (all types of channels) for that queue manager, not just the one channel in question.
In such a case, I would check (directly on the MQ server, not through my monitoring tool) the total number of channel connections on the queue manager. And as a previous poster suggested, I would then look at the output of netstat to see if the numbers made sense.
Not sure what you can do at this point, but perhaps others will have some additional ideas for you! |
That's right. I could not do anything now. I in was more focused on fixing the resoruces side which fixed the problem. |
|
| Back to top |
|
|
|
|
