ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General IBM MQ Support » MQ Channel won't automatically go to Running from Retrying

Post new topic  Reply to topic Goto page 1, 2  Next
 MQ Channel won't automatically go to Running from Retrying « View previous topic :: View next topic » 
Author Message
krypton
PostPosted: Wed Aug 22, 2018 5:05 am    Post subject: MQ Channel won't automatically go to Running from Retrying Reply with quote

Disciple

Joined: 14 Mar 2010
Posts: 186

Hi , we have sender channel on our LINUX system to mainframe, (the channel configuration is defined below)

We sometimes have incident when network connectivity to mainframe failed and our sender channel goes into retrying mode. but once the network connectivity restores, the sender channel doesn't automatically goes into "Running". It should go back into "Running" , isn't it?
Is there any configuration are we missing?



CHANNEL(IBMQMGR.A1A1.SENDER) CHLTYPE(SDR)
ALTDATE(2015-06-07) ALTTIME(01.54.53)
BATCHHB(0) BATCHINT(0)
BATCHLIM(5000) BATCHSZ(50)
CERTLABL( ) COMPHDR(NONE)
COMPMSG(NONE) CONNAME(12.12.12.12(1415)))
CONVERT(NO) DESCR( )
DISCINT(6000) HBINT(300)
KAINT(AUTO) LOCLADDR( )
LONGRTY(999999999) LONGTMR(1200)
MAXMSGL(4194304) MCANAME( )
MCATYPE(PROCESS) MCAUSER( )
MODENAME( ) MONCHL(QMGR)
MSGDATA( ) MSGEXIT( )
NPMSPEED(FAST) PASSWORD( )
PROPCTL(COMPAT) RCVDATA( )
RCVEXIT( ) RESETSEQ(NO)
SCYDATA( ) SCYEXIT( )
SENDDATA( ) SENDEXIT( )
SEQWRAP(999999999) SHORTRTY(10)
SHORTTMR(60)
SSLCIPH(TLS_RSA_WITH_AES_256_CBC_SHA)
SSLPEER( ) STATCHL(QMGR)
TPNAME( ) TRPTYPE(TCP)
USEDLQ(YES) USERID( )
XMITQ(A1A1)

_________________
Dreams are not something which you watch when you are asleep,it is something which doesn't let you sleep.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Aug 22, 2018 5:16 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

Yes, after n retries, the channel should go to RUNNING state.

Is this a new channel? Has this sender channel ever worked?

Have any errors been logged on the sender side? Receiver side?

What is the channel status on the receiver end of the channel?

Is a listener running on the receiver end of the channel?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
krypton
PostPosted: Wed Aug 22, 2018 5:36 am    Post subject: Reply with quote

Disciple

Joined: 14 Mar 2010
Posts: 186

1. yes in normal circumstances the channel keep running.

2. It is not a new channel , been there for few years and yes it works in normal conditions.

3. Error message on the Sender channel (LINUX) side is given below.

4. channel status on receiver side is inactive.

5. Yes, the listener is running on recever side, everything is good on receiver side, only thing is network connectivity get lost and after that the sender on our side doesn't switch automatically from "Retry" to "Running".


Below is the MQ QMGR error logs on sender side

08/12/2018 03:39:06 AM - Process(4681.1) User(mqm) Program(runmqchl)
Host(tdaf-eaipl01.dcd.tdaf.com) Installation(Installation1)
VRMF(8.0.0.2) QMgr(IBMQMGR1)

AMQ9202: Remote host 'mainframe (12.12.12.12) (1415)' not available,
retry later.

EXPLANATION:
The attempt to allocate a conversation using TCP/IP to host 'mainframe
(12.12.12.12) (1415)' for channel IBMQMGR.A1A1.SENDER was not successful.
However the error may be a transitory one and it may be possible to
successfully allocate a TCP/IP conversation later.

In some cases the remote host cannot be determined and so is shown as '????'.
ACTION:
Try the connection again later. If the failure persists, record the error
values and contact your systems administrator. The return code from TCP/IP is
111 (X'6F'). The reason for the failure may be that this host cannot reach the
destination host. It may also be possible that the listening program at host
'prod-mainframe (12.12.12.12) (1415)' was not running. If this is the case,
perform the relevant operations to start the TCP/IP listening program, and try
again.
----- amqccita.c : 1212 -------------------------------------------------------
08/12/2018 03:39:06 AM - Process(4681.1) User(mqm) Program(runmqchl)
Host(abcd01.abcd.com) Installation(Installation1)
VRMF(8.0.0.2) QMgr(IBMQMGR1)

AMQ9999: Channel 'IBMQMGR.A1A1.SENDER' to host '12.12.12.12(1415))' ended
abnormally.

EXPLANATION:
The channel program running under process ID 4681 for channel
'IBMQMGR.A1A1.SENDER' ended abnormally. The host name is '12.12.12.12(1415))';
in some cases the host name cannot be determined and so is shown as '????'.
ACTION:
Look at previous error messages for the channel program in the error logs to
determine the cause of the failure. Note that this message can be excluded
completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage"
attributes under the "QMErrorLog" stanza in qm.ini. Further information can be
_________________
Dreams are not something which you watch when you are asleep,it is something which doesn't let you sleep.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Aug 22, 2018 5:53 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

krypton wrote:


... but once the network connectivity restores, the sender channel doesn't automatically goes into "Running".

If it doesn't go into RUNNING, what state does it go into?

krypton wrote:
AMQ9202: Remote host 'mainframe (12.12.12.12) (1415)' not available,
retry later.



So, what's keeping the sender end from making contact with the receiver end?
Possibilities: a firewall, a listener, a new/modified receiver channel definition, receiver end in STOPPED or RETRY state.

Contact the folks at the receiver end of the channel. Ask them what's in their error logs.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Aug 22, 2018 6:08 am    Post subject: Re: MQ Channel won't automatically go to Running from Retryi Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

krypton wrote:
s there any configuration are we missing?


What's missing is a step in the network resolution.

When they fix the problem, what do the network people do? If you say this channel has been around a few years, I would postulate that whatever MQ is looking at has been "improved" with additional caching so even when the problem is resolved, MQ makes a connection request and is told by the network layer that the IP address doesn't work.

If you can fix the problem by stopping and then immediately restarting the channel, then this is not a smoking gun but it is a gun where the ballistics match. Whatever they're doing to "restore network connectivity" isn't flushing the old connections out of the IP caches
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
krypton
PostPosted: Wed Aug 22, 2018 6:46 am    Post subject: Reply with quote

Disciple

Joined: 14 Mar 2010
Posts: 186

My apologies for not making it clear.

"I think I created the impression in this post that channel is currently not running".
The channel do gets into running state every time once we stop and start it.
So, it is not a problem.
But what we are looking for a solution where someone doesn't have to be manually present, suppose admin is traveling and doesn't have access to servers and if this network issue happens, then the channel should go automatically into Running from Retrying.
_________________
Dreams are not something which you watch when you are asleep,it is something which doesn't let you sleep.
Back to top
View user's profile Send private message
exerk
PostPosted: Wed Aug 22, 2018 7:00 am    Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 6339

As the man said: "If you can fix the problem by stopping and then immediately restarting the channel, then this is not a smoking gun but it is a gun where the ballistics match. Whatever they're doing to "restore network connectivity" isn't flushing the old connections out of the IP caches."
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Aug 22, 2018 7:01 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

krypton wrote:
My apologies for not making it clear.


I too seem to have failed to make myself clear.

krypton wrote:
"I think I created the impression in this post that channel is currently not running".


Nope

krypton wrote:
The channel do gets into running state every time once we stop and start it.


Which I refer to above.

krypton wrote:
But what we are looking for a solution where someone doesn't have to be manually present, suppose admin is traveling and doesn't have access to servers and if this network issue happens, then the channel should go automatically into Running from Retrying.


And as I outlined above, if the network people properly complete the resolution of the network problem that's exactly what will happen.

But I doubt there's any other action than a stop/start you can take from the MQ side. The channel is correctly reacting to the information the network layer is giving it.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
krypton
PostPosted: Wed Aug 22, 2018 7:28 am    Post subject: Reply with quote

Disciple

Joined: 14 Mar 2010
Posts: 186

Quote:
And as I outlined above, if the network people properly complete the resolution of the network problem that's exactly what will happen.

But I doubt there's any other action than a stop/start you can take from the MQ side. The channel is correctly reacting to the information the network layer is giving it.


Thank you for providing this information, this clarifies my doubt why the channel is not going back automatically into Running.
and if we can't do anything at channel configuration now, we will probably put a cron script which looks for channel status and try to stop start it if it is in "Retrying" mode.
_________________
Dreams are not something which you watch when you are asleep,it is something which doesn't let you sleep.
Back to top
View user's profile Send private message
exerk
PostPosted: Wed Aug 22, 2018 7:30 am    Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 6339

krypton wrote:
...we will probably put a cron script which looks for channel status and try to stop start it if it is in "Retrying" mode.

A kludge to mitigate someone else's problem - get 'them' to fix it or this will always be your problem when Incident/Fault/Failure audit people do their reports.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Aug 22, 2018 7:54 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

exerk wrote:
krypton wrote:
...we will probably put a cron script which looks for channel status and try to stop start it if it is in "Retrying" mode.

A kludge to mitigate someone else's problem - get 'them' to fix it or this will always be your problem when Incident/Fault/Failure audit people do their reports.


Now, now my young apprentice; let the cold light of reality shine upon you.

Before the network people here figured out how to do this, we had the monitoring software set up to restart channels / WAS servers / database clients automatically in the face of this kind of problem, and only escalate the ticket to a human if the problem persisted. The tradeoff was the ticket (raised automatically to document the restart) was opened against the network team. I grinned so much at the quarter end bar chart showing number of incidents by team that my jaw nearly fell off.

So yes, the cron job described is a kludge, but it's a workable kludge. My only comment is the job should notify every time it has to do this, so the OP's network team looks bad.

Even if that doesn't help the OP's underlying problem, it's fun.......
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Aug 22, 2018 9:12 am    Post subject: Re: MQ Channel won't automatically go to Running from Retryi Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

krypton wrote:
We sometimes have incident when network connectivity to mainframe failed and our sender channel goes into retrying mode

You have described a symptom. Your task is to find the underlying problem that causes the symptom to appear.

Stopping/restarting the channel is a temporary circumvention - clearly it does not fix the underlying problem.

Not surprised that networks folks say that there is no problem with the network, and subsequently discovering firewall, router and/or nic card issues.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Wed Aug 22, 2018 4:35 pm    Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2492
Location: Melbourne, Australia

Quote:
LONGTMR(1200)

The channel will automatically retry every 20 minutes.
If you know the n/w issue has been fixed, you don't need to wait for the next retry, you can do a START CHANNEL and it should quickly go into RUNNING status.
_________________
Glenn
Back to top
View user's profile Send private message
krypton
PostPosted: Thu Aug 23, 2018 12:16 am    Post subject: Reply with quote

Disciple

Joined: 14 Mar 2010
Posts: 186

Quote:
minutes.
If you know the n/w issue has been fixed, you don't need to wait for the next retry, you can do a START CHANNEL and it should quickly go into RUNNING status


Do you mean we still need to do manual start of channel?

Doesn't the above parameter makes the channel change automatically from retrying to running state as soon as the network issue is over?
_________________
Dreams are not something which you watch when you are asleep,it is something which doesn't let you sleep.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Thu Aug 23, 2018 3:21 am    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

krypton wrote:
Quote:
minutes.
If you know the n/w issue has been fixed, you don't need to wait for the next retry, you can do a START CHANNEL and it should quickly go into RUNNING status


Do you mean we still need to do manual start of channel?

Doesn't the above parameter makes the channel change automatically from retrying to running state as soon as the network issue is over?


As soon as? No.

You channel is set up to retry 10 times one minute apart. If the network issue is resolved within the first 10 minutes, the channel will reconnect on its own but it could be up to 59.9999999 seconds after the network outage is over before the channel attempts the restart on its on.

If the outage is longer than 10 minutes, your channel settings say from that point forward try once every 20 minutes. At that point it could be almost 20 minutes after a network outage is over before your channel retries on its own.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » General IBM MQ Support » MQ Channel won't automatically go to Running from Retrying
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.