| Author |
Message
|
| terlisimo |
 Posted: Wed Dec 23, 2009 11:49 pm Post subject: Failed connection attempts leak memory? |
 |
|
Newbie
Joined: 09 Nov 2009
Posts: 8
|
Hello,
If the queue manager is unavailable due to service being offline or network connectivity problems, the application should be attempting to reconnect forever.
I found that MQ classes leak about 4kb of memory per connection attempt. Some internal MQ class keeps a RemoteTCPConnection object referenced, see snapshot data (8640 connect attempts):
http://imgur.com/3W6dA
The application works like this (simplified):
| Code: |
while (!bConnected) {
try {
conn = connectionFactory.createConnection();
bConnected = true;
}
catch(Exception e) {
LOG.error("Unable to connect, sleeping 10 seconds.");
}
Thread.sleep(10000);
}
|
So, attempting to reconnect every 10 seconds leaks about 32MB of memory per day which is not really nice for a server application.
Increasing the reconnect interval to 5 minutes makes this 32MB per month which is not that bad but is only delaying memory problems.
I made a workaround by attempting to create a TCP socket connection to the MQ server host/port before attempting to connect to a server via MQ classes.
It seems to work fine.
Am I doing something wrong here? Is there a better way to handle Websphere MQ connection attempts?
I'm running WMQ 7.0.1, using JMS and MQTopicConnectionFactory. |
|
| Back to top |
|
 |
| rekarm01 |
| Posted: Thu Dec 24, 2009 3:01 am Post subject: Re: Failed connection attempts leak memory? |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
How about trying to close the connection?
| Code: |
while (!bConnected) {
try {
conn = connectionFactory.createConnection();
bConnected = true;
}
catch(JMSException e) {
if (conn != null)
try { conn.close(); }
catch(JMSException e) {}
LOG.error("Unable to connect, sleeping 10 seconds.");
Thread.sleep(10000);
}
} |
|
|
| Back to top |
|
|
| terlisimo |
| Posted: Thu Dec 24, 2009 3:56 am Post subject: |
|
|
Newbie
Joined: 09 Nov 2009
Posts: 8
|
| Tried that but in practice if connectionFactory.createConnection() throws an Exception then the connection object is never really created so it remains remains null and there's nothing to close. |
|
| Back to top |
|
|
| terlisimo |
| Posted: Thu Dec 24, 2009 5:41 am Post subject: |
|
|
Newbie
Joined: 09 Nov 2009
Posts: 8
|
I am stupid.
My app leaked memory because every time it tried to connect it would create a new ConnectionFactory object. Simplified code is too simplified 
The old ConnectionFactory would be gracefully GC'd but it would add the "failed" connection object to the internal TCP Connection Pool so those "failed" connection objects were never released/GC'd nor reused.
The solution was to create a singleton connectionFactory factory which ensured that there can be only one CF object instantiated and the leak went away.
EDIT:
The solution still leaks 
Last edited by terlisimo on Sun Dec 27, 2009 6:10 am; edited 1 time in total |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Dec 24, 2009 5:51 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| terlisimo wrote: |
| I am stupid. |
It happens to us all. 
I found it stopped bothering me after the first 100 or so times... 
As a slight aside to your originial post, I would question the wisdom of an application that attempts to reconnect "forever". Absolutely it should try and reconnect, and absolutely it should make a number of attempts over a significant period of time (both time and number calculated according to local considerations).
But at some point it should give up, so as to highlight the fact there's a problem. Granted that whatever you've got monitoring queue manager and network should spot the problem, but nothing says "page the on-call person" like a downed application.
Unless of course the application, while attempting reconnection, has some means of pinging the monitoring software directly.
My 2 cents, other views probably valid, Merry Xmas to all.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| terlisimo |
| Posted: Sun Dec 27, 2009 6:08 am Post subject: |
|
|
Newbie
Joined: 09 Nov 2009
Posts: 8
|
There is a leak after all, even when using only one connection factory.
If ConnectionFactory.createConnection method is called and the remote system refuses TCP connection, the application keeps referencing com.ibm.mq.jmqi.remote.internal.RemoteTCPConnection for each failed connection attempt.
Here's a simple app that demonstrates this behavior. Just point it to a host/port that is not listening. Requires WMQ client 7.0+.
| Code: |
import javax.jms.JMSException;
import com.ibm.mq.jms.MQTopicConnectionFactory;
import com.ibm.msg.client.wmq.WMQConstants;
public class MQLeak {
public static void main(final String[] p_args) throws JMSException {
MQTopicConnectionFactory cf = null;
cf = new MQTopicConnectionFactory();
int i = 0;
while (i < 2000) {
cf.setPort(1414);
cf.setHostName("localhost");
cf.setIntProperty(WMQConstants.WMQ_CONNECTION_MODE, WMQConstants.WMQ_CM_CLIENT);
cf.setQueueManager("TEST");
cf.setChannel("TEST");
cf.setUseConnectionPooling(true);
try {
cf.createConnection();
}
catch (JMSException e) {
if (i % 100 == 0) {
System.out.println(i);
}
}
i++;
}
}
}
|
The workaround:
before calling connect, try opening a socket connection to the destination broker, like so:
| Code: |
private static boolean pingHostPort(String p_host, int p_port) {
Socket socket;
try {
socket = new Socket(p_host, p_port);
socket.close();
socket = null;
}
catch (Exception e) {
return false;
}
return true;
} |
I guess I'll open a PMR and see what the big boys say. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Dec 27, 2009 2:56 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Don't know about your test, but I wonder if keeping the CF initialization in the loop would have something to do with it...
Try the same with moving the CF initialization outside of the loop.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| terlisimo |
| Posted: Sun Jan 31, 2010 11:22 pm Post subject: |
|
|
Newbie
Joined: 09 Nov 2009
Posts: 8
|
I can confirm that this was a real memory leak and that it is fixed in Websphere Client 7.0.1.1
 |
|
| Back to top |
|
|
| Michael Dag |
| Posted: Mon Feb 01, 2010 12:46 am Post subject: |
|
|
Jedi Knight
Joined: 13 Jun 2002
Posts: 2607
Location: The Netherlands (Amsterdam)
|
|
| Back to top |
|
|
| terlisimo |
| Posted: Mon Feb 01, 2010 1:01 am Post subject: |
|
|
Newbie
Joined: 09 Nov 2009
Posts: 8
|
|
| Back to top |
|
|
| Michael Dag |
| Posted: Mon Feb 01, 2010 1:06 am Post subject: |
|
|
Jedi Knight
Joined: 13 Jun 2002
Posts: 2607
Location: The Netherlands (Amsterdam)
|
|
| Back to top |
|
|
|
|
