| Author |
Message
|
| RikBaeten |
 Posted: Thu Dec 09, 2010 6:30 am Post subject: [SOLVED] stop conn(*) not fully cleaning up the connection |
 |
|
Novice
Joined: 26 Feb 2007
Posts: 19
|
After I issue the
command, the connection is not visible anymore via
. However via the netstat command I see the corresponding sockets are still open and via the
command I see the corresponding channels are still RUNNING (mcastate).
So I suppose the stop conn command doesn't totally cleanup the connection. I suppose a
command is still required.
Here's a little more background on my situtation:
By default WMQ never time-outs connections and I'm not willing to change this setting (TCPKeepalive in qm.ini) at this moment, since it will impact too much applications not reconnecting automatically. Sometimes abrupt network outages or in my case firewalls can cause stale connections on the WMQ server, which cause occupied connection resources on the queue manager. After a while this makes the number of active connections increase until the max channel instances threshold is reached.
Last edited by RikBaeten on Tue Jan 04, 2011 4:56 am; edited 1 time in total |
|
| Back to top |
|
 |
| RikBaeten |
| Posted: Thu Dec 09, 2010 8:51 am Post subject: |
|
|
Novice
Joined: 26 Feb 2007
Posts: 19
|
|
| Back to top |
|
|
| Mr Butcher |
| Posted: Thu Dec 09, 2010 8:55 pm Post subject: |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
if you want to stop a channel, use the "stop channel" command. if you want to break an application - mq connection use the "stop conn" command.
although an mca is like an application to mq, i would not stop a channel using a "stop conn" command, as there are more things to do when a channel has to be stopped then just to break the (local) mq connection.....
_________________
Regards, Butcher |
|
| Back to top |
|
|
| HubertKleinmanns |
| Posted: Thu Dec 09, 2010 10:56 pm Post subject: |
|
|
Shaman
Joined: 24 Feb 2004
Posts: 732
Location: Germany
|
I have the same problem with "dead" channel. In my case an application connects to a QMgr through a firewall. The application puts or gets several messages. Then it does not use the channel for a long time (let's say an hour or so) but leaves the connection open. Afterwards the application tries to put or get some more message. In the meantime the firewall has interrupted the IP connection. The application recognizes this interruption, opens a new connection and is happy.
BUT the QMgr itself does still see the old connections. After a while - may take days or weeks - all connections which are allowed on the QMgr (100 by default) are in use. Now the application is not able to open a new channel.
There are several solution for the problem:
1. Tell the application developers, to correct their application. It should close an unneeded QMgr connection, before the firewall time-out occurs.
2. Use the "ClientIdle" parameter in the "Channels" stanza. This would close inactive channels after a number of seconds specified with this parameter.
3. Upgrade to Version 7. In Version 7 has a heart beat mechanism for SVRCONN channels. In version 6 such a heartbeat only exists for message channel. To activate this heartbeat you have to set the new attribute SHARECNV at least to "1" (default is "10").
Tell us, if this helps.
_________________
Regards
Hubert |
|
| Back to top |
|
|
| RikBaeten |
| Posted: Fri Dec 10, 2010 4:31 am Post subject: |
|
|
Novice
Joined: 26 Feb 2007
Posts: 19
|
Thank you very much both for your comments.
@Mr Butcher:
Interesting info. So if I understand it correctly:
- on one side the MCA accepts TCP/IP connection from client applications => this corresponds with the result from dis conn and also with the tcp/ip sockets open on the queue manager server
- on the other side it connects in bindings mode to the queue manager itself (and forwards all calls) => this corresponds with the result from dis chstatus
So: if I close the conn, the MCA keeps the channel to the queue manager open. If I stop the channel, the conn stays open. So, I need to both stop the conn and stop the channel in order to liberate all resources.
Now I have two questions:
- Is it possible to stop only the orphaned channel instances without impacting the running channel instances?
- Would it be possible to adapt the MCA, so that it would automatically clean up the occupied channels if no connections are open to it anymore (and vice versa)? Either how the channels cannot be used anymore, so to me it would seem to be a logical implementation and easier for admins. 
@HubertKleinmanns: interesting proposals:
4. I see a fourth solution being the Keepalive setting in the TCP stanza of the qm.ini file. The infocenter states this "causes TCP/IP to check periodically that the other end of the connection is still available. If it is not, the channel is closed." I've tested this by provoking an orphaned channel instance with an abrupt network connection and it works: the channel instance is correctly cleaned up after a timeout set inside my unix server kernel config. => However I don't want to implement this now, since for the moment some of my client applications don't reconnect automatically and this would force us to restart these applications very often (every time after a timeout the channel has been closed).
3. The v7 feature is no option at this moment, since both client and server are not yet upgraded yet.
2. I will check-out and test the ClientIdle parameter in the Channels stanza. It seems similar to the Keepalive setting. |
|
| Back to top |
|
|
| RikBaeten |
| Posted: Fri Dec 10, 2010 5:04 am Post subject: |
|
|
Novice
Joined: 26 Feb 2007
Posts: 19
|
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Dec 10, 2010 8:24 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
I'd say you are going strong with
| Quote: |
| advise against its use |
I'd say that the thread gives you some elements of caution to ponder on.
Like it affects all client connections equally...
On the other hand if your client apps are well behaved and will reconnect on their own, running periodically:
| Code: |
| STOP CHANNEL XYZ STATUS(INACTIVE) |
should help clean up most of your wayward connections. Worst case scenario, you may have to add MODE(FORCE).
However this is an extreme workaround and you should strive to a more gentle solution 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Dec 10, 2010 4:24 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| HubertKleinmanns wrote: |
I have the same problem with "dead" channel. In my case an application connects to a QMgr through a firewall. The application puts or gets several messages. Then it does not use the channel for a long time (let's say an hour or so) but leaves the connection open. Afterwards the application tries to put or get some more message. In the meantime the firewall has interrupted the IP connection. The application recognizes this interruption, opens a new connection and is happy.
BUT the QMgr itself does still see the old connections. After a while - may take days or weeks - all connections which are allowed on the QMgr (100 by default) are in use. Now the application is not able to open a new channel.
There are several solution for the problem:
1. Tell the application developers, to correct their application. It should close an unneeded QMgr connection, before the firewall time-out occurs.
2. Use the "ClientIdle" parameter in the "Channels" stanza. This would close inactive channels after a number of seconds specified with this parameter.
3. Upgrade to Version 7. In Version 7 has a heart beat mechanism for SVRCONN channels. In version 6 such a heartbeat only exists for message channel. To activate this heartbeat you have to set the new attribute SHARECNV at least to "1" (default is "10").
Tell us, if this helps. |
Hubert, good ol' fashioned TCP Keep Alive will clean up these orphaned channels if the firewall truly severed the connection.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Dec 10, 2010 4:27 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| RikBaeten wrote: |
2. I will check-out and test the ClientIdle parameter in the Channels stanza. It seems similar to the Keepalive setting. |
TCP Keep Alive only cleans up orphaned connections where there is nothing left on the client side on that socket. True orphans.
ClientIdle kills both orphaned and dormant but otherwise perfectly valid connections as well. 
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| Mr Butcher |
| Posted: Sun Dec 12, 2010 11:26 pm Post subject: |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
if i want to get rid of an old svrconn connection, i use
stop channel(..) conname(...) mode(force)
then its gone from the list of "active" channels.
i'd not use the stop conn for CHIN connections that reflect channel connections.
_________________
Regards, Butcher |
|
| Back to top |
|
|
| RikBaeten |
| Posted: Wed Dec 15, 2010 3:13 am Post subject: |
|
|
Novice
Joined: 26 Feb 2007
Posts: 19
|
| Mr Butcher wrote: |
if i want to get rid of an old svrconn connection, i use
stop channel(..) conname(...) mode(force)
then its gone from the list of "active" channels. |
The problem is that there are both orphaned connections and valid connections from that specific conname. So I think issuing that command also will break valid connections. In the ideal case the application will rollback the transaction and try to reestablish the connection, but we're not always sure if the concerned application has that degree of robustness implemented. So it's not ideal.
I am looking for a safer manual cleanup that only closes orphaned channel instances.
| Mr Butcher wrote: |
i'd not use the stop conn for CHIN connections that reflect channel connections. |
I'm not sure what you mean with CHIN connections, but I suppose it's not relevant for me. Maybe related to the channel initiator, but in the infocenter I only find Z/OS specific references. |
|
| Back to top |
|
|
| RikBaeten |
| Posted: Wed Dec 15, 2010 3:19 am Post subject: |
|
|
Novice
Joined: 26 Feb 2007
Posts: 19
|
| PeterPotkay wrote: |
TCP Keep Alive only cleans up orphaned connections where there is nothing left on the client side on that socket. True orphans.
ClientIdle kills both orphaned and dormant but otherwise perfectly valid connections as well. |
Interesting!
I don't like the name TCPKeepAlive in this context though, since it suggests keepalive packets are sent. I'd prefer a name like AutocleanOrphanedConns, TimeoutOrphanedConns or something like that. But that's a more philosophic pondering. |
|
| Back to top |
|
|
| HubertKleinmanns |
| Posted: Wed Dec 15, 2010 3:50 am Post subject: |
|
|
Shaman
Joined: 24 Feb 2004
Posts: 732
Location: Germany
|
| RikBaeten wrote: |
...
Interesting!
I don't like the name TCPKeepAlive in this context though, since it suggests keepalive packets are sent. I'd prefer a name like AutocleanOrphanedConns, TimeoutOrphanedConns or something like that. But that's a more philosophic pondering. |
The KeepAlive parameter means: Advise the operating system, to send IP packages through an IP connection - and so "keep it alive".
If the sending of this package fails, the operating system recognizes this and cleans up the IP socket.
So cleaning up an orphaned IP connection is more or less only a "side effect" of keeping the connection active.
And therefore the name does make sense (at least a little bit )
_________________
Regards
Hubert |
|
| Back to top |
|
|
| RikBaeten |
| Posted: Wed Dec 15, 2010 4:31 am Post subject: |
|
|
Novice
Joined: 26 Feb 2007
Posts: 19
|
| HubertKleinmanns wrote: |
The KeepAlive parameter means: Advise the operating system, to send IP packages through an IP connection - and so "keep it alive".
If the sending of this package fails, the operating system recognizes this and cleans up the IP socket.
So cleaning up an orphaned IP connection is more or less only a "side effect" of keeping the connection active.
And therefore the name does make sense (at least a little bit ) |
Once again, interesting! 
But I'm quite sure there must be an additional thing done by the KeepAlive parameter, namely stopping the orphaned channel instances, since:
a manual stop conn cleans up connections and sockets, but not channel instances
the keepalive setting cleans up connections, sockets AND channel instances
|
|
| Back to top |
|
|
| HubertKleinmanns |
| Posted: Wed Dec 15, 2010 5:25 am Post subject: |
|
|
Shaman
Joined: 24 Feb 2004
Posts: 732
Location: Germany
|
| RikBaeten wrote: |
...
But I'm quite sure there must be an additional thing done by the KeepAlive parameter, namely stopping the orphaned channel instances, since:
a manual stop conn cleans up connections and sockets, but not channel instances
the keepalive setting cleans up connections, sockets AND channel instances
|
No, MQ does nothing, the operating system sends keep alive packages and cleans up dead IP connections. This has nothing to do with the channel itself.
Th behaviour of STOP CONN is as follows:
1. You have a running channel agent (MCA) wich on one side is connected to the QMgr, on the other side to the network.
2. STOP CONN stops the connection to the QMgr, but not to the network.
3. The result is a running MCA without connection to the QMgr.
_________________
Regards
Hubert |
|
| Back to top |
|
|
|
|
