|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
|
|
CLUSRCVR channel in paused state |
« View previous topic :: View next topic » |
Author |
Message
|
yanaK |
Posted: Thu Jun 11, 2020 2:54 am Post subject: CLUSRCVR channel in paused state |
|
|
Acolyte
Joined: 28 May 2020 Posts: 69
|
Our cluster setup for a request response flow is like this:
Client ->
PS.SVR (Channel) -> SYSTEM.CLUSTER.TRANSMIT.QUEUE (QLOCAL) -> TO.PS.B (CLUSSDR) ->
TO.PS.B (CLUSRCVR) -> PS.REQ (QLOCAL) ->
[Application] ->
SYSTEM.DEF.SVRCONN (CHANNEL) -> SYSTEM.CLUSTER.TRANSMIT.QUEUE (QLOCAL) -> TO.PS.F (CLUSSDR) ->
TO.PS.F (CLUSRCVR) -> PS.RSP (QLOCAL) -> PS.SVR (Channel) ->
Client
Now suddenly I see TO.PS.B (CLUSRCVR) is in paused state. Everything else is in running state.
CLUSRCVER error shows this:
Code: |
----- amqrrmfa.c : 30286 ------------------------------------------------------
06/11/2020 03:12:46 AM - Process(11713.1) User(mqm) Program(amqrrmfa)
Host(t4231) Installation(Installation1)
VRMF(7.1.0.1) QMgr(PS.B)
AMQ9435: Unable to put repository manager message.
EXPLANATION:
The repository manager tried to send a message to the
SYSTEM.CLUSTER.COMMAND.QUEUE on another queue manager whose identifier is
MB_2012-07-23_11.24.30, but the MQPUT call was unsuccessful. MQCC=2,
MQRC=2085. Processing continues, but the repository information may be out of
date.
ACTION:
Refer to the Application Programming Reference manual for information about
MQCC 2 and MQRC 2085. Check the channel and cluster definitions on the local
and target queue managers, and ensure that the channels between them are
running. When the problem is corrected, the repository information will
normally be updated automatically. The REFRESH CLUSTER command can be used to
ensure that the repository information is up to date.
----- amqrrmfa.c : 30286 ------------------------------------------------------
06/11/2020 03:12:54 AM - Process(11738.58) User(mqm) Program(amqzlaa0)
Host(t4231) Installation(Installation1)
VRMF(7.1.0.1) QMgr(PS.B)
AMQ7234: 10000 messages from queue 'SYSTEM.DEAD.LETTER.QUEUE' loaded on queue
manager 'PS.B'.
EXPLANATION:
10000 messages from queue SYSTEM.DEAD.LETTER.QUEUE have been loaded on queue
manager PS.B.
ACTION:
None.
-------------------------------------------------------------------------------
06/11/2020 03:12:54 AM - Process(11738.58) User(mqm) Program(amqzlaa0)
Host(t4231) Installation(Installation1)
VRMF(7.1.0.1) QMgr(PS.B)
AMQ7234: 20000 messages from queue 'SYSTEM.DEAD.LETTER.QUEUE' loaded on queue
manager 'PS.B'.
ACTION:
None.
----- amqccita.c : 3688 -------------------------------------------------------
06/11/2020 03:15:35 AM - Process(11758.57) User(mqm) Program(amqrmppa)
Host(t4231) Installation(Installation1)
VRMF(7.1.0.1) QMgr(PS.B)
AMQ9544: Messages not put to destination queue.
EXPLANATION:
During the processing of channel 'TO.PS.B' one or more messages could not
be put to the destination queue and attempts were made to put them to a
dead-letter queue. The location of the queue is 1, where 1 is the local
dead-letter queue and 2 is the remote dead-letter queue.
ACTION:
Examine the contents of the dead-letter queue. Each message is contained in a
structure that describes why the message was put to the queue, and to where it
was originally addressed. Also look at previous error messages to see if the
attempt to put messages to a dead-letter queue failed. The program identifier
(PID) of the processing program was '11758'. |
On the 1st error one question is that queue manager is not in use - how to remove it? (I tried to strmqm but it won't even start)
On the CLUSSDR error I see this:
Code: |
----- amqrccca.c : 956 --------------------------------------------------------
06/11/2020 03:34:08 AM - Process(9143.13573) User(mqm) Program(amqrmppa)
Host(t4232) Installation(Installation1)
VRMF(7.1.0.1) QMgr(PS.F)
AMQ9544: Messages not put to destination queue.
EXPLANATION:
During the processing of channel 'TO.PS.B' one or more messages could not
be put to the destination queue and attempts were made to put them to a
dead-letter queue. The location of the queue is 2, where 1 is the local
dead-letter queue and 2 is the remote dead-letter queue.
ACTION:
Examine the contents of the dead-letter queue. Each message is contained in a
structure that describes why the message was put to the queue, and to where it
was originally addressed. Also look at previous error messages to see if the
attempt to put messages to a dead-letter queue failed. The program identifier
(PID) of the processing program was '9143'.
|
The Application is throwing "No message available (timeout) for queue PS.RSP".
I tried restarting the PS.B channel - no luck - then the queue manager - still no luck.
I saw the SYSTEM.CLUSTER.TRANSMIT.QUEUE (the 1st one) had a huge CURDEPTH - cleared it - still no luck.
Channel details:
Code: |
1 : dis chl (TO.PS.B)
AMQ8414: Display Channel details.
CHANNEL(TO.PS.B) CHLTYPE(CLUSRCVR)
ALTDATE(2019-12-19) ALTTIME(15.01.04)
BATCHHB(0) BATCHINT(0)
BATCHLIM(5000) BATCHSZ(40)
CLUSNL( ) CLUSTER(REPPST)
CLWLPRTY(0) CLWLRANK(0)
CLWLWGHT(50) COMPHDR(NONE)
COMPMSG(NONE) CONNAME(t4232(1416))
CONVERT(NO) DESCR( )
DISCINT(600) HBINT(120)
KAINT(AUTO) LOCLADDR( )
LONGRTY(999999999) LONGTMR(1200)
MAXMSGL(4194304) MCANAME( )
MCATYPE(THREAD) MCAUSER( )
MODENAME( ) MONCHL(QMGR)
MRDATA( ) MREXIT( )
MRRTY(10) MRTMR(1000)
MSGDATA( ) MSGEXIT( )
NETPRTY(0) NPMSPEED(FAST)
PROPCTL(COMPAT) PUTAUT(DEF)
RCVDATA( ) RCVEXIT( )
RESETSEQ(NO) SCYDATA( )
SCYEXIT( ) SENDDATA( )
SENDEXIT( ) SEQWRAP(999999999)
SHORTRTY(10) SHORTTMR(60)
SSLCAUTH(REQUIRED) SSLCIPH( )
SSLPEER( ) STATCHL(QMGR)
TPNAME( ) TRPTYPE(TCP)
USEDLQ(YES) |
My questions are 1. why did it enter this state ? 2. Is there a way to know which channel links to which queues ? 3. Is there a way to force it to running state ?
Thanks.[/code] |
|
Back to top |
|
|
Vitor |
Posted: Thu Jun 11, 2020 5:02 am Post subject: Re: CLUSRCVR channel in paused state |
|
|
Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
yanaK wrote: |
My questions are 1. why did it enter this state ? |
The queue manager is not in use to the point you can't even start it.
yanaK wrote: |
2. Is there a way to know which channel links to which queues ? |
Channels don't link to queues, channel link to queue managers. T
yanaK wrote: |
3. Is there a way to force it to running state ? |
If there was a way to fix a bad configuration by just forcing it to work, we'd all be out of a job. _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
|
fjb_saper |
Posted: Thu Jun 11, 2020 5:18 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003 Posts: 20729 Location: LI,NY
|
Code: |
MB_2012-07-23_11.24.30 |
This seems to be a qmid. Do you know what the qmgr name is and why are you not able to start it?
Does it have damaged queues? Do you need to start it with the -c option?
What did the qmgr log say after the attempt to start the qmgr failed? Why did the qmgr not start?
Was there a error message when you ran strmqm ? _________________ MQ & Broker admin |
|
Back to top |
|
|
bruce2359 |
Posted: Thu Jun 11, 2020 5:20 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008 Posts: 9442 Location: US: west coast, almost. Otherwise, enroute.
|
The errors you posted describe an ACTION:.
Did you take the actions? What were the results? _________________ I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
Back to top |
|
|
bruce2359 |
Posted: Thu Jun 11, 2020 5:55 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008 Posts: 9442 Location: US: west coast, almost. Otherwise, enroute.
|
Moved to Clustering forum. _________________ I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
Back to top |
|
|
hughson |
Posted: Thu Jun 11, 2020 7:57 pm Post subject: Re: CLUSRCVR channel in paused state |
|
|
Padawan
Joined: 09 May 2013 Posts: 1948 Location: Bay of Plenty, New Zealand
|
yanaK wrote: |
CLUSRCVER error shows this:
[code]06/11/2020 03:12:46 AM - Process(11713.1) User(mqm) Program(amqrrmfa)
Host(t4231) Installation(Installation1)
VRMF(7.1.0.1) QMgr(PS.B)
AMQ9435: Unable to put repository manager message.
EXPLANATION:
The repository manager tried to send a message to the
SYSTEM.CLUSTER.COMMAND.QUEUE on another queue manager whose identifier is
MB_2012-07-23_11.24.30, but the MQPUT call was unsuccessful. MQCC=2,
MQRC=2085. Processing continues, but the repository information may be out of
date.
ACTION:
Refer to the Application Programming Reference manual for information about
MQCC 2 and MQRC 2085. Check the channel and cluster definitions on the local
and target queue managers, and ensure that the channels between them are
running. When the problem is corrected, the repository information will
normally be updated automatically. The REFRESH CLUSTER command can be used to
ensure that the repository information is up to date.
----- amqrrmfa.c : 30286 ------------------------------------------------------
On the 1st error one question is that queue manager is not in use - how to remove it? (I tried to strmqm but it won't even start) |
If the queue manager is decommissioned, you should remove it from the cluster so that other members of the cluster don't try to continue using it. This error messages shows that they are still trying to use it.
This link Removing a queue manager from a cluster is how you should normally remove a queue manager from a cluster before you decommission it. It also contains another link within it to remove a queue manager when it hasn't been done correctly. I suspect you need the alternative method since you cannot start the queue manager.
yanaK wrote: |
Now suddenly I see TO.PS.B (CLUSRCVR) is in paused state. Everything else is in running state. |
STATUS(PAUSED) shows that a receiving type channel is failing to a MQPUT a message to a target queue, and has paused to see if the MQPUT will succeed if it tries again in a moment.
Given that you have just loaded 1000s of messages from the Dead Letter Queue, I suspect, they were on the DLQ because they couldn't be MQPUT to their target queue before and perhaps that problem has not been fixed, so you have to go through these 1000s messages, with the Message Retry timer for each one, and then they will end up on the DLQ again. Do you know why the messages were on the DLQ, and was the problem shown in the DL Header fixed before they were loaded again?
yanaK wrote: |
06/11/2020 03:12:54 AM - Process(11738.58) User(mqm) Program(amqzlaa0)
Host(t4231) Installation(Installation1)
VRMF(7.1.0.1) QMgr(PS.B)
AMQ7234: 10000 messages from queue 'SYSTEM.DEAD.LETTER.QUEUE' loaded on queue
manager 'PS.B'.
EXPLANATION:
10000 messages from queue SYSTEM.DEAD.LETTER.QUEUE have been loaded on queue
manager PS.B.
ACTION:
None.
-------------------------------------------------------------------------------
06/11/2020 03:12:54 AM - Process(11738.58) User(mqm) Program(amqzlaa0)
Host(t4231) Installation(Installation1)
VRMF(7.1.0.1) QMgr(PS.B)
AMQ7234: 20000 messages from queue 'SYSTEM.DEAD.LETTER.QUEUE' loaded on queue
manager 'PS.B'.
ACTION:
None.
----- amqccita.c : 3688 -------------------------------------------------------
Channel details:
1 : dis chl (TO.PS.B)
AMQ8414: Display Channel details.
CHANNEL(TO.PS.B) CHLTYPE(CLUSRCVR)
MRRTY(10) MRTMR(1000) |
yanaK wrote: |
The Application is throwing "No message available (timeout) for queue PS.RSP". |
This is not a surprise given the above - the messages this application is waiting on will be queued up behind all the re-queued messages from the DLQ.
yanaK wrote: |
I tried restarting the PS.B channel - no luck - then the queue manager - still no luck. |
This will have only delayed the processing of the 1000s of re-queued messages for a time while the channel restarted.
yanaK wrote: |
I saw the SYSTEM.CLUSTER.TRANSMIT.QUEUE (the 1st one) had a huge CURDEPTH - cleared it - still no luck. |
This does not seem like a good idea. You could have deleted both application and cluster messages.
yanaK wrote: |
My questions are 1. why did it enter this state ? |
The channel entered STATUS(PAUSED) because it is having problems putting messages to target queues.
yanaK wrote: |
2. Is there a way to know which channel links to which queues ? |
Channels can deliver messages to any queues on the queue manager. Normally there is only one cluster channel between any two queue managers, so that one channel will be delivering messages to all the cluster queues.
yanaK wrote: |
3. Is there a way to force it to running state ? |
It will go into running state when it gets through the backlog of failing puts. If you don't want to retry puts, you can remove the MRRTY and MRTMR attributes.
I would suggest that the root cause will be that messages from the DLQ were reloaded without the issue reported in the DLQ-ed messages being fixed first. You should look into that.
Cheers,
Morag _________________ Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
Back to top |
|
|
|
|
|
|
Page 1 of 1 |
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|
|
|