| Author |
Message
|
| nagabhb |
 Posted: Wed Jul 27, 2011 5:40 pm Post subject: Channel Program ended abnormally |
 |
|
Apprentice
Joined: 20 Dec 2010
Posts: 42
|
Hi,
Channels went in Retry status and below are the errros i could see in logs.
Sender side :
07/27/11 12:50:27 - Process(1249400.1) User(mqm) Program(runmqchl_nd)
Host()
AMQ9002: Channel 'QM1.QM2' is starting.
EXPLANATION:
Channel 'QM1.QM2' is starting.
ACTION:
None.
-------------------------------------------------------------------------------
07/27/11 12:50:40 - Process(1249400.1) User(mqm) Program(runmqchl_nd)
Host('')AMQ9527: Cannot send message through channel 'QM1.QM2' .
EXPLANATION:
The channel has closed because the remote queue manager cannot receive a
message.
ACTION:
Contact the systems administrator who should examine the error files of the
remote queue manager, to determine why the message cannot be received, and then
restart the channel.
----- cmqxrfpt.c : 457 --------------------------------------------------------
07/27/11 12:50:40 - Process(1249400.1) User(mqm) Program(runmqchl_nd)
Host()
AMQ9506: Message receipt confirmation failed.
EXPLANATION:
Channel 'QM1.QM2' has ended because the remote queue manager did
not accept the last batch of messages.
ACTION:
The error log for the channel at the remote site will contain an explanation of
the failure. Contact the remote Systems Administrator to resolve the problem.
----- amqrmrca.c : 863 --------------------------------------------------------
07/27/11 12:50:40 - Process(1249400.1) User(mqm) Program(runmqchl_nd)
Host()
AMQ9999: Channel program ended abnormally.
EXPLANATION:
Channel program 'QM1.QM2' ended abnormally.
ACTION:
Look at previous error messages for channel program 'QM1.QM2' in
the error files to determine the cause of the failure.
----- amqrccca.c : 921 --------------------------------------------------------
Receiver side :
7/27/2011 14:22:32 - Process(2904.5863) User(svc.mqNTFWID2) Program(amqrmppa.exe)
AMQ9002: Channel 'QM1.QM2' is starting.
EXPLANATION:
Channel 'QM1.QM2' is starting.
ACTION:
None.
-------------------------------------------------------------------------------
7/27/2011 14:22:38 - Process(2904.5863) User('') Program(amqrmppa.exe)
AMQ9511: Messages cannot be put to a queue.
EXPLANATION:
The attempt to put messages to queue 'SYSTEM.DEAD.LETTER.QUEUE' on queue
manager 'QM2' failed with reason code 2003.
ACTION:
Ensure that the required queue is available and operational.
----- amqrmmqa.c : 1671 -------------------------------------------------------
7/27/2011 14:22:39 - Process(1704.3) User('') Program(amqzmuc0.exe)
AMQ7469: Transactions rolled back to release log space.
EXPLANATION:
The log space for the queue manager is becoming full. One or more long-running
transactions have been rolled back to release log space so that the queue
manager can continue to process requests.
ACTION:
Try to ensure that the duration of your transactions is not excessive. Consider
increasing the size of the log to allow transactions to last longer before the
log starts to become full.
-------------------------------------------------------------------------------
7/27/2011 14:22:43 - Process(2904.5863) User('') Program(amqrmppa.exe)
AMQ9511: Messages cannot be put to a queue.
EXPLANATION:
The attempt to put messages to queue 'REPLY' on queue manager 'QM2'
failed with reason code 2003.
ACTION:
Ensure that the required queue is available and operational.
----- amqrmmqa.c : 1671 -------------------------------------------------------
7/27/2011 14:22:43 - Process(2904.5863) User('') Program(amqrmppa.exe)
AMQ9999: Channel program ended abnormally.
EXPLANATION:
Channel program 'QM1.QM2' ended abnormally.
ACTION:
Look at previous error messages for channel program 'QM1.QM2' in
the error files to determine the cause of the failure.
----- amqrmrsa.c : 468 --------------------------------------------------------
7/27/2011 14:23:08 - Process(2904.5864) User('') Program(amqrmppa.exe)
AMQ9207: The data received from host ('')' is not
valid.
EXPLANATION:
Incorrect data format received from host ' ('')' over
TCP/IP. It may be that an unknown host is attempting to send data. An FFST file
has been generated containing the invalid data received.
ACTION:
Tell the systems administrator.
----- amqccita.c : 3413 -------------------------------------------------------
7/27/2011 14:23:08 - Process(2904.5864) User('') Program(amqrmppa.exe)
AMQ9492: The TCP/IP responder program encountered an error.
EXPLANATION:
The responder program was started but detected an error.
ACTION:
Look at previous error messages in the error files to determine the error
encountered by the responder program.
----- amqrmrsa.c : 459 --------------------------------------------------------
Looks like long running transactions are causing this and i can see messages are piling up in SYSTEM.CLUSTER.TRANSMIT.QUEUE at sender end.
Please let me know how to proceed.
Thanks in Advance
Last edited by nagabhb on Thu Jul 28, 2011 4:05 am; edited 2 times in total |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Wed Jul 27, 2011 8:12 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
So you have 2 action paths
| Quote: |
The attempt to put messages to queue 'SYSTEM.DEAD.LETTER.QUEUE' on queue
manager 'IDCPRD02' failed with reason code 2003.
ACTION:
Ensure that the required queue is available and operational.
----- amqrmmqa.c : 1671 -------------------------------------------------------
7/27/2011 14:22:39 - Process(1704.3) User(svc.mqNTFWID2) Program(amqzmuc0.exe)
AMQ7469: Transactions rolled back to release log space.
EXPLANATION:
The log space for the queue manager is becoming full. One or more long-running
transactions have been rolled back to release log space so that the queue
manager can continue to process requests.
ACTION:
Try to ensure that the duration of your transactions is not excessive.
Consider increasing the size of the log to allow transactions to last longer before the log starts to become full. |
What did you do of the 2 actions suggested?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| nagabhb |
| Posted: Thu Jul 28, 2011 4:00 am Post subject: |
|
|
Apprentice
Joined: 20 Dec 2010
Posts: 42
|
| As of now i did not take any action. I'm just looking is there any option we can proceed with out increasing active log space. Why because this was running fine for quite some time, and all of sudden since yesterday we are noticing these errors. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jul 28, 2011 4:19 am Post subject: Re: Channel Program ended abnormally |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| nagabhb wrote: |
| Looks like long running transactions are causing this |
| nagabhb wrote: |
| I'm just looking is there any option we can proceed with out increasing active log space |
Have the applications use smaller units of work.
Also:
| nagabhb wrote: |
| The attempt to put messages to queue 'SYSTEM.DEAD.LETTER.QUEUE' |
Don't use SYSTEM objects. Define a local queue and use that as the DLQ.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Jul 28, 2011 4:59 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Moved to General WebSphere MQ Support forum.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| nagabhb |
| Posted: Thu Jul 28, 2011 5:38 am Post subject: |
|
|
Apprentice
Joined: 20 Dec 2010
Posts: 42
|
forgot to add one more note. We have similar setup in another servers (I mean to say same object definitons, same messages) which is working fine, surprisingly only in one server we are facing this issue.
Thanks |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jul 28, 2011 5:46 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| nagabhb wrote: |
| We have similar setup in another servers (I mean to say same object definitons, same messages) which is working fine |
How similar? The object definitions and messages may be the same, but it's queue manager log space that seems to be at issue here.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| nagabhb |
| Posted: Thu Jul 28, 2011 5:52 am Post subject: |
|
|
Apprentice
Joined: 20 Dec 2010
Posts: 42
|
I'm sorry, the thing which i did not understand is how only in one server active logs are getting filled up , what additional things are happening here compared to other active server.
However looks like increasing logs space is the solution, we will proceed in that direction.
Thaks for you Assistance !!! |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jul 28, 2011 6:07 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| nagabhb wrote: |
| I'm sorry, the thing which i did not understand is how only in one server active logs are getting filled up , what additional things are happening here compared to other active server. |
Because one server has more or larger log files than the other? You said the object definitions are the same across the 2, and the messages are the same across the 2, but did you check the queue managers themselves?
It's not outside the realm of possibility that the working one was defined with correctly sized logs while the other was accidently defined using the defaults.
This would give you some confidence in your solution if it transpires you're bringing the non-working one into line with the working one, rather than just increasing the logs to see if more space helps.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Jul 28, 2011 6:07 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| nagabhb wrote: |
I'm sorry, the thing which i did not understand is how only in one server active logs are getting filled up , what additional things are happening here compared to other active server.
|
Log space (primary and secondary) is set as part of qmgr creation. Logs (log space) are used based on workload; so, a busy qmgr will experience more log usage than, say, a test qmgr.
Are the log space definitions exactly the same for the two qmgrs?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Jul 28, 2011 6:34 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| nagabhb wrote: |
I'm sorry, the thing which i did not understand is how only in one server active logs are getting filled up , what additional things are happening here compared to other active server.
However looks like increasing logs space is the solution, we will proceed in that direction.
Thaks for you Assistance !!! |
One possibility is that for that lone server, you have an application that is not working anymore...
Results:- queue full
- DLQ overflow
- log overflow because the messages that do not get consumed use log space too...
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| nagabhb |
| Posted: Thu Jul 28, 2011 2:04 pm Post subject: |
|
|
Apprentice
Joined: 20 Dec 2010
Posts: 42
|
| Code: |
| It's not outside the realm of possibility that the working one was defined with correctly sized logs while the other was accidently defined using the defaults. |
Yes vitor what ever you said was correct. I have overlooked the definition.In one queue manager primary log files count is 5 and in another queue manager it is 3.
Thanks for your help. |
|
| Back to top |
|
|
|
|
