Author |
Message
|
rickwatsonb |
Posted: Wed Jun 02, 2010 7:15 am Post subject: Orphan Channels on Unix MQ server |
|
|
 Voyager
Joined: 15 Aug 2006 Posts: 87 Location: USA: Mid-West
|
Hi,
I have researched the information on orphan channels previously, but I am still uncertain about some of the details/basics.
We have a batch application using JMS to connect to a TRN queue manager on a UNIX server via a SVRCONN channel. The batch team just started running batch on the TRN queue manager. They say the same batch jobs have been running fine in another environment.
The SVRCONN channel is maxing out; the Max Active channels is set to 500 but netstat is showing 100 channels for the port, and the TRN batch jobs will no longer run. Thus, I cannot connect via MQ Explorer. It appears that there are 400 orphan channels.
I know that recycling the channel will free up the orphaned connections. Before I do that, I would like to obtain more information if possible about the orphaned channels.
My questions are:
(1) Is it true that the netstat listing on the Unix MQ server only shows the two-way connections (app to MQ server and MQ server to app)?
(2) Is there a way to display a listing of the orphaned channels?
(3) Are the orphaned channels connections from the Unix MQ server to the application server that do not have a corresponding connection from the application server to the Unix MQ server?
We have not implemented TCP KeepAlive yet; although I would prefer to do so. Perhaps this will provide the needed leverage.
Thanks for your time and help. |
|
Back to top |
|
 |
PeterPotkay |
Posted: Wed Jun 02, 2010 10:06 am Post subject: Re: Orphan Channels on Unix MQ server |
|
|
 Poobah
Joined: 15 May 2001 Posts: 7722
|
Make sure you understand the difference between orphaned channels versus stale or inactive channels.
Orphaned channels exist on the QM side, but no longer have a valid socket connection on the client side.
Stale or inactive channels may appear orphaned to you, but they are 100% valid channels from the QM's perspective. The application has simply stopped using them but otherwise not closed them.
rickwatsonb wrote: |
(1) Is it true that the netstat listing on the Unix MQ server only shows the two-way connections (app to MQ server and MQ server to app)? |
I think so, not 100% sure.
rickwatsonb wrote: |
(2) Is there a way to display a listing of the orphaned channels? |
No, but doing a display of all channels will allow you to look at the last activity on an particular instance of a channel. If it hasn't sent any data in a long time, it could be orphaned. Or merely stale / inactive.
rickwatsonb wrote: |
(3) Are the orphaned channels connections from the Unix MQ server to the application server that do not have a corresponding connection from the application server to the Unix MQ server?
|
Yes.
rickwatsonb wrote: |
We have not implemented TCP KeepAlive yet; although I would prefer to do so. |
You should. But it will only help with orphaned channels, not stale / inactive ones.
A system variable called ClientIdle will help with stale / inactive channels, but I don't like it because when it kicks in and drops otherwise perfectly valid connections, it doesn't report any unique error code. You ar eleft wondering why the connection broke. Instead of using ClientIdle you should focus on getting the apps to use client chnanels correctly - only open as many as you need, close them when you are truly done, make sure your close logic gets called in ALL cases and don't skip it when your app goes down an error path, etc. _________________ Peter Potkay
Keep Calm and MQ On |
|
Back to top |
|
 |
rickwatsonb |
Posted: Wed Jun 02, 2010 11:15 am Post subject: |
|
|
 Voyager
Joined: 15 Aug 2006 Posts: 87 Location: USA: Mid-West
|
Thank you Peter for the quick reply and for adding the information about inactive channels.
Is it then correct to surmise that netstat on the MQ Unix server will include inactive channels along with the active ones in the listing?
Thanks! |
|
Back to top |
|
 |
PeterPotkay |
Posted: Wed Jun 02, 2010 11:33 am Post subject: |
|
|
 Poobah
Joined: 15 May 2001 Posts: 7722
|
rickwatsonb wrote: |
Is it then correct to surmise that netstat on the MQ Unix server will include inactive channels along with the active ones in the listing? |
Haven't tested it, but I would assume yes, as stale / inactive channels are valid connections into the MQ server. _________________ Peter Potkay
Keep Calm and MQ On |
|
Back to top |
|
 |
gbaddeley |
Posted: Wed Jun 02, 2010 3:31 pm Post subject: |
|
|
 Jedi Knight
Joined: 25 Mar 2003 Posts: 2538 Location: Melbourne, Australia
|
I don't think its possible to see a SVRCONN channel instance in "inactive" status. Possibly if the client end has died or nothing useful has been done for a long time, the channel status will be shown as "running" on the queue manager side, but looking at the channel status details it will be in a substate of "receive" and its last date/time will be old. Keep alive or heartbeat should be able to detect these "stale" connections and MQ or TCP/IP will then clean them up. _________________ Glenn |
|
Back to top |
|
 |
PeterPotkay |
Posted: Wed Jun 02, 2010 3:53 pm Post subject: |
|
|
 Poobah
Joined: 15 May 2001 Posts: 7722
|
Inactive was a poor choice of words, since that is an actual MQ state of a channel that is not running at all.
Replace "inactive" with "running, but doing no application work" in my previous comments.
TCP Keep Alive and/or Heartbeats will not stop these "running, but doing no application work" client channels, since they are otherwise valid connections. ClientIdle will, but with the word of warning I mentioned earlier.
TCP Keep Alive will kill a running SVRCONN channels on the MQ Server side of the MQ Client side TCP/IP socket is gone, i.e. a truly orphaned channel. _________________ Peter Potkay
Keep Calm and MQ On |
|
Back to top |
|
 |
mvic |
Posted: Wed Jun 02, 2010 3:55 pm Post subject: Re: Orphan Channels on Unix MQ server |
|
|
 Jedi
Joined: 09 Mar 2004 Posts: 2080
|
rickwatsonb wrote: |
We have not implemented TCP KeepAlive yet; although I would prefer to do so. |
It's a simple edit to qm.ini and a qm restart.. let us know if it helps. |
|
Back to top |
|
 |
PeterPotkay |
Posted: Wed Jun 02, 2010 3:56 pm Post subject: Re: Orphan Channels on Unix MQ server |
|
|
 Poobah
Joined: 15 May 2001 Posts: 7722
|
mvic wrote: |
rickwatsonb wrote: |
We have not implemented TCP KeepAlive yet; although I would prefer to do so. |
It's a simple edit to qm.ini and a qm restart.. let us know if it helps. |
Assuming the O/S has TCP/IP Keep Alive turned on. _________________ Peter Potkay
Keep Calm and MQ On |
|
Back to top |
|
 |
mvic |
Posted: Wed Jun 02, 2010 4:10 pm Post subject: Re: Orphan Channels on Unix MQ server |
|
|
 Jedi
Joined: 09 Mar 2004 Posts: 2080
|
|
Back to top |
|
 |
mvic |
Posted: Wed Jun 02, 2010 4:11 pm Post subject: Re: Orphan Channels on Unix MQ server |
|
|
 Jedi
Joined: 09 Mar 2004 Posts: 2080
|
PeterPotkay wrote: |
Assuming the O/S has TCP/IP Keep Alive turned on. |
Fair point. I had indeed made that assumption. |
|
Back to top |
|
 |
SAFraser |
Posted: Thu Jun 03, 2010 9:37 am Post subject: |
|
|
 Shaman
Joined: 22 Oct 2003 Posts: 742 Location: Austin, Texas, USA
|
And, that the TCP keep alive on the OS is configured to a useful duration (depending upon your requirements).
For our environment, we reduce the Solaris keepalive parameter from its default of two hours to 15 minutes. |
|
Back to top |
|
 |
George Carey |
Posted: Tue Jul 13, 2010 11:05 am Post subject: netstat usage |
|
|
Knight
Joined: 29 Jan 2007 Posts: 500 Location: DC
|
Good posts !!
On the subject of determining orphaned channels.
netstat -n | grep {clientport} | sort
On linux gives a nice eyeball checker way to see socket session pairs with state declared as ESTABLISHED.
Assumedly if an orphaned connection existed the state would be something other than ESTABLISHED and there would be an odd number if piped to wc -l (e.g. 8 = 4 client connections, 7 = 3 connections , one failed/orphaned)
Any way with MQ Explorer to kill a specific client connection? I have 4 running clients but only 1 appears 'truly active' the other three appear to be 'stale' as PP refers to them.
The MQ Explorer shows different MCA job name for each {hexnumber}7, 8, 9, c. Three in 'Receiving' channel substate and one in 'In MQGet' channel substate. Where is best place to read up on substate meaning of channels? _________________ "Truth is ... grasping the virtually unconditioned",
Bernard F. Lonergan S.J.
(from book titled "Insight" subtitled "A Study of Human Understanding") |
|
Back to top |
|
 |
George Carey |
Posted: Tue Jul 13, 2010 11:32 am Post subject: oops |
|
|
Knight
Joined: 29 Jan 2007 Posts: 500 Location: DC
|
Well apparently that isn't exactly the way to interpret the netstat -n output as I can do the same command on my MQ Explorer port connection and I get a count of 1 ... no return socket connection pairing ... yet channel is active and running of course.
... anyhow ...
Bruce ... I believe that was 'best place' as in a document / manual ??
Would like to try to map, mca channel substates to what I know of the different tcp/ip socket connection states if possible.
Doco could be same place all MQ documentation is just haven't really seached for it in a while. Also I am talking about a client channel.
My my your testy today! ... did you get out of wrong side of bed? _________________ "Truth is ... grasping the virtually unconditioned",
Bernard F. Lonergan S.J.
(from book titled "Insight" subtitled "A Study of Human Understanding") |
|
Back to top |
|
 |
George Carey |
Posted: Tue Jul 13, 2010 11:49 am Post subject: for instance |
|
|
Knight
Joined: 29 Jan 2007 Posts: 500 Location: DC
|
maybe you could map the channel substates for me into this:
http://support.microsoft.com/kb/137984
Quote: |
This article describes TCP connection states and how to read Netstat (NETSTAT.EXE) output.
Before data transfer takes place in TCP, a connection must be established. TCP employs a three-way handshake (the details of this can be found in RFC793, Chapter 3: "Functional Specification").
Back to the top
MORE INFORMATIONTCP Connection StatesFollowing is a brief explanation of this handshake. In this...TCP Connection States
Following is a brief explanation of this handshake. In this context the "client" is the peer requesting a connection and the "server" is the peer accepting a connection. Note that this notation does not reflect Client/Server relationships as an architectural principal.
Connection Establishment
The client sends a SYN message which contains the server's port and the client's Initial Sequence Number (ISN) to the server (active open).
The server sends back its own SYN and ACK (which consists of the client's ISN + 1).
The Client sends an ACK (which consists of the server's ISN + 1).
Connection Tear-down (modified three way handshake).
The client sends a FIN (active close). This is a now a half-closed connection. The client no longer sends data, but is still able to receive data from the server. Upon receiving this FIN, the server enters a passive close state.
The server sends an ACK (which is the clients FIN sequence + 1)
The server sends its own FIN.
The client sends an ACK (which is server's FIN sequence + 1). Upon receiving this ACK, the server closes the connection.
A half-closed connection can be used to terminate sending data while sill receiving data. Socket applications can call shutdown with the second argument set to 1 to enter this state.
Back to the top
Netstat Output
The above TCP connection states can be monitored in a network trace under the TCP flags. It is also possible to determine the status of the connection by running the Netstat utility and looking at the State column. Netstat is shipped with Windows NT, Windows 95, and TCP/IP-32 for Windows for Workgroups.
State explanations as shown in Netstat:
State Explanation
------------ --------------------------------------------------------
SYN_SEND Indicates active open.
SYN_RECEIVED Server just received SYN from the client.
ESTABLISHED Client received server's SYN and session is established.
LISTEN Server is ready to accept connection.
NOTE: See documentation for listen() socket call. TCP sockets in listening state are not shown - this is a limitation of NETSTAT. For additional information, please see the following article in the Microsoft Knowledge Base:
134404 (http://support.microsoft.com/kb/134404/EN-US/ ) NETSTAT.EXE Does Not Show TCP Listen Sockets
FIN_WAIT_1 Indicates active close.
TIMED_WAIT Client enters this state after active close.
CLOSE_WAIT Indicates passive close. Server just received first FIN from a client.
FIN_WAIT_2 Client just received acknowledgment of its first FIN from the server.
LAST_ACK Server is in this state when it sends its own FIN.
CLOSED Server received ACK from client and connection is closed.
As an example, consider the following scenario:
A socket application has been terminated, but Netstat reports the socket in a CLOSE_WAIT state. This could indicate that the client properly closed the connection (FIN has been sent), but the server still has its socket open. This could be the result of one instance (among all threads or processes) of the socket not being closed.
NOTE: It is normal to have a socket in the TIME_WAIT state for a long period of time. The time is specified in RFC793 as twice the Maximum Segment Lifetime (MSL). MSL is specified to be 2 minutes. So, a socket could be in a TIME_WAIT state for as long as 4 minutes. Some systems implement different values (less than 2 minutes) for the MSL.
Additional references:
"Internetworking with TCP/IP, Volume 1" by Douglas Comer
"TCP/IP Illustrated, Volume 1" by Richard Stevens.
"Computer Networks" by Andrew Tanenbaum
Back to the top |
_________________ "Truth is ... grasping the virtually unconditioned",
Bernard F. Lonergan S.J.
(from book titled "Insight" subtitled "A Study of Human Understanding") |
|
Back to top |
|
 |
bruce2359 |
Posted: Tue Jul 13, 2010 11:50 am Post subject: |
|
|
 Poobah
Joined: 05 Jan 2008 Posts: 9469 Location: US: west coast, almost. Otherwise, enroute.
|
Oooooooops. And I apologize. Wrong side of bed, indeed. And misplaced reading glasses. In summary, no excuse. I will forthwith smack myself with a dead fish. _________________ I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
Back to top |
|
 |
|