| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Program stops picking up messages under load |
« View previous topic :: View next topic » |
| Author |
Message
|
| yalmasri |
 Posted: Wed Jun 18, 2008 3:58 am Post subject: Program stops picking up messages under load |
 |
|
Centurion
Joined: 18 Jun 2008
Posts: 110
|
Hi,
I have a simple Java program that fetches messages from a queue, with a simple pooling technique to manage connections. The program works fine for a considerable amount of time under load, but eventually starts slowing down reading messages until it blacks out, and the queue starts piling up messages consequently. When the program is restarted, it proceeds working very normal, draining all the piled messages, and continuing operation for some time, then the cycle is repeated.
Here is a snippet of the getMessage method of the program:
synchronized public MQMessage getMQMessage(MQQueueManager qMgr, String qName, byte[] corrId) {
MQMessage message = new MQMessage();
MQQueue queue = null;
int openOptions = MQC.MQOO_INPUT_AS_Q_DEF ;
log.debug("trying to access queue: " + qName);
queue = qMgr.accessQueue(qName, openOptions, null, null, null);
log.debug("queue accessed successfully.");
MQGetMessageOptions gmo = new MQGetMessageOptions();
gmo.matchOptions = MQC.MQMO_MATCH_CORREL_ID;
gmo.options= MQC.MQGMO_WAIT;
gmo.waitInterval = MQConnectionManager.QUEUE_WAIT_TIME_OUT; // this is constant in my MQConnectionManager class
message.correlationId = corrId;
queue.get(message, gmo);
log.debug("got message successfully.");
qMgr.commit();
queue.close();
log.debug("closed queue successfully.");
...
}
The qMgr passed to the method is retrieved from this method:
public synchronized MQQueueManager getConnection() {
for(int i =0 ; i < connectionsMap.size() ; i++) {
Boolean inUse = (Boolean)connectionsMap.getValueAt(i);
MQQueueManager key = (MQQueueManager)connectionsMap.getKeyAt(i);
if(!inUse.booleanValue()) {
if (!key.isConnected()) {
log.debug("Connection is broken and will be replaced");
connectionsMap.remove(key);
MQQueueManager newConnection = createNewConnnection(); // this method will create a new MQQueueManager and add it to the pool
if(newConnection != null)
connectionsMap.put(newConnection , new Boolean(true));
log.debug("Connection replaced and will be returned");
return newConnection;
}
connectionsMap.setValueAt(i, new Boolean(true));
MQQueueManager con = (MQQueueManager)connectionsMap.getKeyAt(i);
return con;
}
}
// all connections in the pool are in use going to create new one
MQQueueManager newConnection = createNewConnnection();
if(newConnection != null)
connectionsMap.put(newConnection , new Boolean(true));
return newConnection;
}
I exhausted everything I could possibly think of that's behind this behavior, but to no avail. Does anyone have any hints or thoughts that may help me pinpointing the problem?
Thanks. |
|
| Back to top |
|
 |
| RogerLacroix |
| Posted: Wed Jun 18, 2008 8:03 pm Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
Hi,
In getMQMessage method, you do: open, get, commit & close - bad design. Stop opening & closing the queue over and over again.
Secondly, you are using commit, but you never set GMO option of MQC.MQGMO_SYNCPOINT, so the commit is pointless.
Third, you wrote a connection manager? Why? And why didn't you use the built in connection manager called: MQSimpleConnectionManager
Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jun 18, 2008 8:10 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
A few questions:
A)Is there a specific reason for a synchronized getMessage method? Obviously all the relevant variables are instance variables for the method... so there should be no data collision and synchronization can only slow you down.
Needed synchronization is done for you by the MQ implemetation anyway...
B) Have you checked the qmgr logs and more specifically the number of concurrent connections (dis chs(*) ). Maybe you need to allow for more than the default...
C) Before discarding a broken connection, you might want to try and close it anyhow. You know it is broken but the QMGR might not. To avoid using up all connections trying to close it might get through to the qmgr and allow to free resources.
D) Have you considered allocating an application specific channel that you can "force close" and restart so as not to have to bounce the app or the qmgr?
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| yalmasri |
| Posted: Sat Jun 21, 2008 5:44 am Post subject: |
|
|
Centurion
Joined: 18 Jun 2008
Posts: 110
|
Thanks for the help guys.
Roger,
OK you're right, I think I should instead be looping around the get method of the queue, somewhere between the accessQueue and close.
Secondly, you're right again ... but only because I'm not operating on z/OS 
As per the MQSimpleConnectionManager, I really was not sure if it guarantees returning me healthy connections, it just did not provide methods like testOnBorrow or testWhileIdle so I was doubtful about how much I could rely on it, and I preferred off doing this myself.
But anyway, do you think any of what you mentioned might be causing this hang?
fjb,
Thank you for the valuable suggestions. Let me clarify more my program. As you have noticed, my pooler is creating as many connections as the concurrent working threads and leave them open forever. I'm printing a statement whenever createNewConnnection() is called, and I'm able to count the same number of threads I have.
Now when I reach the maximum number of connections I don't change it, and my program keeps reusing those connections, so if I'm going to face a problem due to inability to handle connections, that'd be early when I reach that maximum, which is not the case. I'm working for a whole day without any trouble and suddenly things start slowing down (over 10 minutes), and eventually die. The weird thing here is that I also never printed "Connection is broken and will be replaced" at all, which means all the connections were and are still healthy!
I hope I find a solution to this problem where I don't have to deal with option D
 |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Jun 21, 2008 9:39 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
You will still need to handle the problem of a broken connection.
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| yalmasri |
| Posted: Sat Jun 21, 2008 10:30 pm Post subject: |
|
|
Centurion
Joined: 18 Jun 2008
Posts: 110
|
I do, but only when I have a broken connection. As I said, if I never print "Connection is broken and will be replaced" then I assume I never faced a disconnection.
What perplexes me is that the same code is used somewhere else in many applications and is working fine, it's only in three of the heavily loaded ones we face this.
To rearrange my thoughts, what I'm ending up with is:
- A bunch of connections that are valid and tested against "isConnected" before used - The number of connections is fixed (nothing is added or removed) and equal to the number of threads in the application - The application runs for around 20 hours under load then it starts slowing down (no abrupt disconnection occurred) - The application eventually stops picking up messages although I can see a good number of them in the queue
My feeling is that there is something wrong going on in the MQ itself and not an application problem.  |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Jun 21, 2008 10:36 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| yalmasri wrote: |
I do, but only when I have a broken connection. As I said, if I never print "Connection is broken and will be replaced" then I assume I never faced a disconnection.
What perplexes me is that the same code is used somewhere else in many applications and is working fine, it's only in three of the heavily loaded ones we face this.
To rearrange my thoughts, what I'm ending up with is:
- A bunch of connections that are valid and tested against "isConnected" before used - The number of connections is fixed (nothing is added or removed) and equal to the number of threads in the application - The application runs for around 20 hours under load then it starts slowing down (no abrupt disconnection occurred) - The application eventually stops picking up messages although I can see a good number of them in the queue
My feeling is that there is something wrong going on in the MQ itself and not an application problem. |
MQ has been working fine for some time. My assumption here would be that for whatever reason the connection has gone stale on you and you did not catch it. What you would need to do is release (close) the connections and refresh the pool.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| yalmasri |
| Posted: Sat Jun 21, 2008 10:47 pm Post subject: |
|
|
Centurion
Joined: 18 Jun 2008
Posts: 110
|
I got you.
What would be the test upon which this refreshment should occur in your opinion? I'm not comfortable to make it timely based. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Jun 22, 2008 5:51 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| yalmasri wrote: |
I got you.
What would be the test upon which this refreshment should occur in your opinion? I'm not comfortable to make it timely based. |
Whenever for any operation(queue, message, qmgr, etc...) throws an Exception with reason code 2009. In fact you will always need to print the exception regardless of the reason code and analyze the cause given the reason code. You could just get a wrong state exception instead of a connection broken etc...
Also use a tool. Having messages in a queue (qdepth) does not mean that they are in a gettable state. If reason code is not 2033 (no more messages matching criteria) or 0, I'd say you have a problem.
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
