| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| SVRCONN channels showing status INACTIVE and don't go away |
« View previous topic :: View next topic » |
| Author |
Message
|
| royce |
 Posted: Sat Jul 24, 2021 4:14 pm Post subject: SVRCONN channels showing status INACTIVE and don't go away |
 |
|
Newbie
Joined: 08 Jan 2009
Posts: 3
|
Hello,
I did search on SVRCONN INACTIVE and didn't find anything on this. But I am recently noticing a lot of server connection channels , approximately 150, in INACTIVE status. I haven't noticed this before They are in status RECEIVE. I would expect that the app has dropped the connection but I don't understand why the SVRCONN channels don't go away from view in Channel status instead of showing INACTIVE.
IBM MQ v 9.1.2.0
OS Linux VM running RHEL 7.1
App is Java running from PCF.
I am running one way TLS 1.2 on the channels. With shared conversations set to 10.
There are about 2000 SVRCONN channels (max channels set to 3000) showing in chs and about 8000 connections to queue manager.
dis chl
CHANNEL(OMS.SVRCONN) CHLTYPE(SVRCONN)
ALTDATE(2020-10-13) ALTTIME(19.35.29)
CERTLABL( ) COMPHDR(NONE)
COMPMSG(NONE) DESCR( )
DISCINT(0) HBINT(300)
KAINT(AUTO) MAXINST(999999999)
MAXINSTC(999999999) MAXMSGL(104857600)
MCAUSER(*no-show*) MONCHL(QMGR)
RCVDATA( ) RCVEXIT( )
SCYDATA( ) SCYEXIT( )
SENDDATA( ) SENDEXIT( )
SHARECNV(10) SSLCAUTH(OPTIONAL)
SSLCIPH(TLS_RSA_WITH_AES_128_CBC_SHA256)
SSLPEER( ) TRPTYPE(TCP)
dis chs
CHANNEL(OMS.SVRCONN) CHLTYPE(SVRCONN)
BUFSRCVD(318144) BUFSSENT(206184)
BYTSRCVD(31933088) BYTSSENT(28978552)
CHSTADA(2021-07-12) CHSTATI(03.27.5
COMPHDR(NONE,NONE) COMPMSG(NONE,NONE)
COMPRATE(0,0) COMPTIME(0,0)
CONNAME(10.119.16.220) CURRENT
EXITTIME(0,0) HBINT(300)
JOBNAME(0000000000000000) LOCLADDR(::ffff:10.119.30.9(30011))
LSTMSGDA(2021-07-22) LSTMSGTI(08.03.56)
MCASTAT(NOT RUNNING) MCAUSER(*no-show*)
MONCHL(HIGH) MSGS(206123)
RAPPLTAG(WebSphere MQ Client for Java)
SECPROT(TLSV12) SSLCERTI( )
SSLCIPH(TLS_RSA_WITH_AES_128_CBC_SHA256)
SSLKEYDA( ) SSLKEYTI( )
SSLPEER( ) SSLRKEYS(0)
STATUS(INACTIVE) STOPREQ(NO)
SUBSTATE(RECEIVE) CURSHCNV(7)
MAXSHCNV(10) RVERSION(07010007)
RPRODUCT(MQJM)
I find it interesting that it says MCASTAT NOT RUNNING. I am wondering if adppt new MCA would help.
Thanks for any suggestions or responses. |
|
| Back to top |
|
 |
| hughson |
| Posted: Mon Jul 26, 2021 3:10 am Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
Normally you never see channels that have STATUS(INACTIVE). They just don't get returned.
You say that you have about 8000 connections to the queue manager from these 2000 SVRCONNs. If each one has around 7 connections, as the one example you show us does, then that seems OK. Only an average of 4 connections per SVRCONN.
Do the client machines think they have a current working connection to the queue manager?
Is the number shown in MSGS changing (going up)? This would imply that the connection is still working.
Or to ask another way, do you have reason to believe that these are stale connections, or is the only thing that looks wrong the STATUS(INACTIVE)?
What does netstat have to say?
They have been running for quite a long time - since the 12th July. Is that normal for your system?
P.S. Very old version of MQ at the client end!
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| royce |
| Posted: Mon Jul 26, 2021 2:59 pm Post subject: |
|
|
Newbie
Joined: 08 Jan 2009
Posts: 3
|
Hello Morag,
There have been recent application connectivity issues related to this queue manager. This is a long running application at for us, as you noted by the old client, but was migrated from on premise datacenter running on Linux VMWare to Azure cloud IaaS. We added Multi-Instance and a load balancer to provide HA in cloud. Recently the app team started monitoring their channel status and called this out.
I did check netstat and do see similar number of ESTABILISHED connections to the number of RUNNING server connection channels. So I don't think the INACTIVE channels are showing in netstat.
I do see lots of AMQ9209E and AMQ9999E errors, which makes me wonder if between app system, load balancer, network and MQ there are some network settings that could be optimized better. I have seen some documentation suggesting using shared conversations has some limitations, do you think setting to 1 could help with the stability, or maybe the verions of MQ client could be contributing?
AMQ9209E: Connection to host 'IP removed' for channel 'OMS.SVRCONN' closed.
EXPLANATION:
An error occurred receiving data from 'IP removed' over TCP/IP. The
connection to the remote host has unexpectedly terminated.
AMQ9999E: Channel 'OMS.SVRCONN' to host 'IP removed' ended abnormally.
EXPLANATION:
The channel program running under process ID 28376 for channel 'OMS.SVRCONN'
ended abnormally. The host name is 'IP removed'; in some cases the host name
cannot be determined and so is shown as '????'.
Good point about the old client. I will mention that to app team that they should plan to update in the future to give best stability and performance. |
|
| Back to top |
|
|
| hughson |
| Posted: Mon Jul 26, 2021 10:43 pm Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
| royce wrote: |
AMQ9209E: Connection to host 'IP removed' for channel 'OMS.SVRCONN' closed.
EXPLANATION:
An error occurred receiving data from 'IP removed' over TCP/IP. The
connection to the remote host has unexpectedly terminated. |
This certainly sounds like a break in connectivity. Hard to say what caused it just from this error message though. If netstat shows that there are not established sockets associated with the STATUS(INACTIVE) SVRCONNs then they certainly sound like stale state. Given that, you should certainly consider opening a case with IBM. Regardless of how flaky your network is, you should never get state like that.
| royce wrote: |
I have seen some documentation suggesting using shared conversations has some limitations, do you think setting to 1 could help with the stability, or maybe the verions of MQ client could be contributing?
Good point about the old client. I will mention that to app team that they should plan to update in the future to give best stability and performance. |
It is true that early versions of shared conversations could cause performance issues. Never heard of them causing stale state like this though. It shouldn't functionally change the way the JMS applications work if you change to 1, it'll just mean you will have 8000 SVRCONNs instead of 2000 SVRCONNs with 8000 connections. You'll want to up your MAXCHLS if you decide to make that change of course!
Could be better to get a more modern version of the client. Who knows how many defects have been fixed in the over a dozen major releases that have come since the version they are using 
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| royce |
| Posted: Thu Jul 29, 2021 1:42 pm Post subject: |
|
|
Newbie
Joined: 08 Jan 2009
Posts: 3
|
Yes, agreed that max channels will need a bump if change shared conversations to 1. Do you think there is anything to gain in channel stability or performance if I change from 10 to 1?
Is it safe to assume I might see more system resource usage if I have lots more serverconn channels running?
Will also pass on the details to app team to upgrade their client libraries. Would you say it is possible the client version has more impact on channel stability and performance than the shared conversation parameter?
I am on base version 9.1.2.0 so probably due for MQ server upgrade as well. If the INACTIVE channel status bugs me enough I will open case but otherwise might wait until after I upgrade to see if it goes away.
Thanks for the input Morag!
Last edited by royce on Wed Aug 04, 2021 8:49 am; edited 2 times in total |
|
| Back to top |
|
|
| hughson |
| Posted: Thu Jul 29, 2021 2:10 pm Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
I suspect an IBMer would give a more valid opinion than me, but I think the client version would have the biggest impact on the stability.
If you are able to get them to upgrade first and see if that sorts it then you might not have to change the queue manager side of things.
You're right though, if you have more SVRCONNs running, you will see more system resource usage.
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Thu Jul 29, 2021 3:53 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| royce wrote: |
AMQ9209E: Connection to host 'IP removed' for channel 'OMS.SVRCONN' closed.
EXPLANATION:
An error occurred receiving data from 'IP removed' over TCP/IP. The
connection to the remote host has unexpectedly terminated.
AMQ9999E: Channel 'OMS.SVRCONN' to host 'IP removed' ended abnormally.
EXPLANATION:
The channel program running under process ID 28376 for channel 'OMS.SVRCONN'
ended abnormally. The host name is 'IP removed'; in some cases the host name cannot be determined and so is shown as '????'. |
This usually happens if there is a network interruption on the TCP socket session, or the MQ Client application is terminated or killed without a proper MQ disconnect occurring. Monitor it carefully. Look for patterns. It is probably not MQ that is causing the issue.
_________________
Glenn |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
