| Author |
Message
|
| czaszek |
 Posted: Wed Nov 22, 2006 7:30 am Post subject: Too many SVRCONN channels |
 |
|
Apprentice
Joined: 30 May 2006
Posts: 36
Location: Poland
|
HI All
We maintain qmgr to which clients connects. After few days I observe few hundreds of opened SVRCONN channels (of the same name). This makes the machine not available after some time.
Any idea how to make them close automatically after the client disconnets?
The parameters of the SVRCONN channel:
1 : dis chl(CHANNEL1)
AMQ8414: Display Channel details.
CHANNEL(CHANNEL1) CHLTYPE(SVRCONN)
TRPTYPE(TCP) DESCR( )
SCYEXIT( ) MAXMSGL(4194304)
SCYDATA( ) HBINT(10)
SSLCIPH( ) SSLCAUTH(REQUIRED)
KAINT(10) MCAUSER(mqm)
ALTDATE(2006-07-28) ALTTIME(10.03.57)
SSLPEER()
SENDEXIT( )
RCVEXIT( )
SENDDATA( )
RCVDATA( )
in mqsi.ini we have:
CHANNELS:
MaxActiveChannels=2000
MaxChannels=2000
TCP:
KeepAlive=Yes
Thanks in advance for any help.
Best Regards
Lukasz |
|
| Back to top |
|
 |
| Vitor |
| Posted: Wed Nov 22, 2006 7:35 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Are the clients using the Java / JMS interface or C / C++ native interface? Or XMS?
Are you certain that the clients are performing a proper disconnect & clean-up before ending or are they just dropping out?
Can you also please post details of platform & MQ version you're using?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| czaszek |
| Posted: Wed Nov 22, 2006 9:56 am Post subject: |
|
|
Apprentice
Joined: 30 May 2006
Posts: 36
Location: Poland
|
We are using Aix 5.1 (or 5.2) and MQ 5.3.
Regarding the client I can not say much (Most probably the client uses the JMS.), because I am not aware of this application, I thought that it is possible to manage this without knowledge about the client site (I am working and responsible only for the server site). |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Nov 22, 2006 1:46 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| czaszek wrote: |
We are using Aix 5.1 (or 5.2) and MQ 5.3.
Regarding the client I can not say much (Most probably the client uses the JMS.), because I am not aware of this application, I thought that it is possible to manage this without knowledge about the client site (I am working and responsible only for the server site). |
If you are certain that this situation is not due to network problems you can force them to shutdown the communications by shutting down the offending channel.
Of course this will also disconnect legitimately connected clients but it should clear the problem and if your clients don't bother to disconnect properly they should suffer the aggravation of you periodically stopping the svrconn channel and starting it again... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Nov 23, 2006 12:16 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
I concur with fjb_saper and think that a medium term solution is to report this as a possible issue with the client code. Slighty more friendly and professional than just shutting down the channel. It's more than likely the disconnect is not happening properly their end.
My preferred solution would be to schedule a cron job to bounce the channel at regular intervals.... 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| czaszek |
| Posted: Thu Nov 23, 2006 1:41 am Post subject: |
|
|
Apprentice
Joined: 30 May 2006
Posts: 36
Location: Poland
|
I wondered whether the MQ automatic mechanisms: heartbeat and KAINT could (should) solve the problem. I have found that the heartbeats are being sent only when a GET with a wait is invoked against the MQI channel, but in this case should not the KAINT solve the problem?
Does the KAINT work with MQI channels? |
|
| Back to top |
|
|
| Nigelg |
| Posted: Thu Nov 23, 2006 2:03 am Post subject: |
|
|
Grand Master
Joined: 02 Aug 2004
Posts: 1046
|
HBINT only applies on client channels during an MQGET, as you found out.
KAINT is mainframe only.
Your best bet would be to enable KeepAlive on the channels, and set a low keep alive interval in TCP.
Set it on AIX using these commands:
| Code: |
AIX:
get:
no -a tcp_keepintvl
no -a tcp_keepidle
set:
no -o tcp_keepintvl=20
no -o tcp_keepidle=120
The interval is in half-seconds
The parameter takes effect immediately. If the machine is rebooted the parameter is reset to the default value.
To make the change permanent, add the no commands to the /etc/rc.net script.
|
Enable KeepAlive in WMQ:
| Code: |
Enable KeepAlive in WMQ config file
On AIX, keep alive has to be enabled in a WMQ config file.
On a server
Add these lines to /var/mqm/qmgrs/QMNAME/qm.ini
TCP:
KeepAlive=Yes
On a client
Create a file called /var/mqm/qm.ini, and add the same lines.
|
_________________
MQSeries.net helps those who help themselves.. |
|
| Back to top |
|
|
| czaszek |
| Posted: Thu Nov 23, 2006 4:58 am Post subject: |
|
|
Apprentice
Joined: 30 May 2006
Posts: 36
Location: Poland
|
Most probably our client application uses JMS, according to IBM, these kind of applications do not use the qm.ini files, so can not gather the info regarding the KeepAlive.
If it was a client application using MQ API written in C, the qm.ini would be used. If then the client disconnected incorrectly (without using MQ_DISCONNECT or whatever is needed), the "netstat -an" would show several entries "CLOSE_WAIT", which would be closed according to the system tcp keep alive setting.
Unfortunately in case of JMS application if client disconnect incorrectly, the only way to close the channels is a qmgr restart.
My case is slightly different from the above. I see in the "netstat -an" lots of entries ESTABLISHED, so in my opinion the client application not only do not close (or try to close) the opened channels, but every time establish a new connection (maybe within a new application thread).
I guess the best way to release all the channel instances would be to stop the SVRCONN channel using MODE(FORCE) and start it again manually, at least the qmgr restart will not be necessary.
What do you think? |
|
| Back to top |
|
|
| Sartin |
| Posted: Thu Nov 23, 2006 6:24 am Post subject: |
|
|
Newbie
Joined: 15 May 2006
Posts: 9
Location: Poland
|
We've got similiar problem with lots of open channels and connections with ESTABLISHED state. The appliaction that's causing problems is written in Java and deployed on WebSphere Appliation Server that acts as MQ client.
Releasing channels by stoping channel with FORCEis not an option becouse it will disrupt application behaviour. Is there any other way? |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Nov 23, 2006 7:57 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
If the application has failed to release the client channel properly then you either need to bounce the channel or the queue manager; both of which will terminate applications currently in action. It's down to the requirements of your site how you schedule this for minimum disruption.
The only other course of action open (and it's not a short term solution) is to raise a PMR with IBM & see if there's any more subtle way of forcing these connections to close. Or see if something can be built into the next release...... 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Nov 23, 2006 12:53 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
The alternate option is to do nothing on the qmgr but TERMINATE the jvm responsible for the stale connections. This should free the connections as well. However this might just be unacceptable for the app. The best course of action still seems to force close the channel and start it again. If the offending application then reacquires a connection you should be good to go for a least a little while...
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Nov 24, 2006 12:18 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
In theory at least, a running client application should be coded on the assumption the connection will drop unexpectedly and try to recover from it; so any valid application bounced by the force close should reconnect leaving just the stale connections gone.
Of course, the same theory states that a client application will clear up after itself and not leave connections hanging around.... 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
