| Author |
Message
|
| abhay09 |
 Posted: Tue Oct 17, 2017 3:29 am Post subject: Stuck Thread on new MQQueueManager(qMName, Props). |
 |
|
Acolyte
Joined: 31 May 2016
Posts: 66
|
Hi Experts,
Recently I found a Stuck Thread issue in my running server and it is on MQQueueManager qm= new MQQueueManager(qMName, props);
Stack trace:
<XXX 22:25:20> <Error> <XXX> <XXX> <[STUCK] ExecuteThread: '40' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "608" seconds working on the request "Workmanager: XXX, Version: 0, Scheduled=false, Started=true, Started time: 608268 ms
", which is more than the configured time (StuckThreadMaxTime) of "600" seconds in "server-failure-trigger". Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
com.ibm.mq.jmqi.remote.impl.RemoteConnection.requestSendLock(RemoteConnection.java:2731)
com.ibm.mq.jmqi.remote.impl.RemoteConnection.sendTSH(RemoteConnection.java:1755)
com.ibm.mq.jmqi.remote.impl.RemoteSession.startConversation(RemoteSession.java:1123)
com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.sessionFromEligible(RemoteConnectionSpecification.java:449)
com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSessionFromEligibleConnection(RemoteConnectionSpecification.java:392)
com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSession(RemoteConnectionSpecification.java:250)
com.ibm.mq.jmqi.remote.impl.RemoteConnectionPool.getSession(RemoteConnectionPool.java:144)
com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1491)
com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1144)
com.ibm.mq.MQSESSION.MQCONNX_j(MQSESSION.java:924)
com.ibm.mq.MQManagedConnectionJ11.<init>(MQManagedConnectionJ11.java:222)
com.ibm.mq.MQClientManagedConnectionFactoryJ11._createManagedConnection(MQClientManagedConnectionFactoryJ11.java:553)
com.ibm.mq.MQClientManagedConnectionFactoryJ11.createManagedConnection(MQClientManagedConnectionFactoryJ11.java:593)
com.ibm.mq.StoredManagedConnection.<init>(StoredManagedConnection.java:95)
com.ibm.mq.MQSimpleConnectionManager.allocateConnection(MQSimpleConnectionManager.java:198)
com.ibm.mq.MQQueueManagerFactory.obtainBaseMQQueueManager(MQQueueManagerFactory.java:882)
com.ibm.mq.MQQueueManagerFactory.procure(MQQueueManagerFactory.java:770)
com.ibm.mq.MQQueueManagerFactory.constructQueueManager(MQQueueManagerFactory.java:719)
com.ibm.mq.MQQueueManagerFactory.createQueueManager(MQQueueManagerFactory.java:175)
com.ibm.mq.MQQueueManager.<init>(MQQueueManager.java:727)
Now, I am looking for an api to handle this. Does IBM MQ Java API doesn't provide any method of connectionMaxWait or ConnectionTimedOut when trying to create new MQQueueManager connection?
I have gone through the IBM docs and couldn't find any. How to wait for sometime and throw exception after that time elapsed.
Please suggest how to handle this stuck thread.
Any help will be appreciated.
Thanks! |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Tue Oct 17, 2017 4:16 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Make sure you are using the MQ Resource Adapter in your weblogic server config.
Also talk to your MQ admin about this issue. After you confirm the above.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Oct 17, 2017 5:16 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Did you make sure you're not having a dead lock somewhere? Looks to be some concurrency issue...
Why are you using MQ Base in a WebLogic server and not JMS??? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| abhay09 |
| Posted: Tue Oct 17, 2017 5:20 am Post subject: |
|
|
Acolyte
Joined: 31 May 2016
Posts: 66
|
| mqjeff wrote: |
Make sure you are using the MQ Resource Adapter in your weblogic server config.
Also talk to your MQ admin about this issue. After you confirm the above. |
I am using plain MQ client libraries not MQ Resource Adapter.
We can ask MQ Admin but need some info if MQ provides any connection time out or max wait to handle these situations.
Please let me know if we can provide connection time out or max wait in MQQueueManager object.
Thanks! |
|
| Back to top |
|
|
| abhay09 |
| Posted: Tue Oct 17, 2017 5:22 am Post subject: |
|
|
Acolyte
Joined: 31 May 2016
Posts: 66
|
| fjb_saper wrote: |
Did you make sure you're not having a dead lock somewhere? Looks to be some concurrency issue...
Why are you using MQ Base in a WebLogic server and not JMS??? |
It is like this only we are using MQ java client jars to connect to MQ.
I need some info if MQ provides any connection time out or max wait to handle these situations.
Please let me know if we can provide connection time out or max wait in MQQueueManager object because I couldn't find any. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Oct 17, 2017 6:12 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
If you are running in a JEE server (i.e. weblogic) then you should be using JMS and the MQ Resource Adapter.
If you are using the plain MQ classes, then you run into a lot of different problems - among other things creating a very very very large number of connections on the queue manager.
The MQ Admin can provide timeouts on the channel you are using.
Talk to them. Talk to your WebLogic admin. Talk to your team lead or senior engineer and get them to explain why you shouldn't be using JMS... which you should be using.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| tomleend |
| Posted: Tue Oct 17, 2017 6:13 am Post subject: |
|
|
Acolyte
Joined: 24 Jan 2014
Posts: 51
|
There is a connection timeout configuration property you can set for the MQ classes for JMS and classes for Java when TCP/IP sockets are created and an attempt to connect them are made. Namely:
Connect_Timeout
in the TCP stanza of the MQ client ini file, which accepts a timeout value in seconds. (This can also be set as a Java System Property -Dcom.ibm.mq.cfg.TCP.Connect_Timeout=<value_in_seconds>).
*But* this won't help you in your case.
Just as fjb_saper mentions above, this looks like a concurrency issue where you thread is waiting for an internal lock ("SendLock") in order to then be able to write some data to an pre-established socket in order to start a new conversation (hConn) with a queue manager.
When you observe this issue, you should take a Javacore. You will probably find that another thread that is writing to the socket that your blocked thread wants to use and it is this thread that is holding the "SendLock" being requested. The question then is why is the thread holding the "SendLock" not progressing - which may require you to investigate the health of your network.
Note that there is no timeout configuration property that can be set to instruct the thread in the call to com.ibm.mq.jmqi.remote.impl.RemoteConnection.requestSendLock() to give up waiting after a certain period of time. It is an indefinite blocking wait to obtain the lock.
One workaround though could be to set the sharing conversations (SHARECNV) property on the client/server connection channel used by the application to the value 1 (rather than the default of 10). With this setup, each conversation (hConn) to the queue manager from your application will use its own TCP/IP connection rather than multiplex multiple conversations over a single socket. Note this will increase the number of channel instances (one channel instance = one TCP/IP connection) on your queue manager. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Oct 17, 2017 11:34 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Using the same connection for a get and a put and having a get with unlimited wait may do that to you. So your put process is waiting for socket time that is being monopolized by your get thread...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| tomleend |
| Posted: Tue Oct 17, 2017 12:34 pm Post subject: |
|
|
Acolyte
Joined: 24 Jan 2014
Posts: 51
|
Luckily, that's not the case. Imagine how awful the SHARECNV multiplexing capabilities would be if that were true 
If you want to do a get with unlimited wait (perhaps a JMS doing a MessageConsumer.receive() call) then the application thread making that API call only needs the "SendLock" for the connection to write the small MQGET request flow to the queue manager before it drops it. So that should be super quick.
There's a dedicate "Remote Receive Thread" that is always listening on the socket to pass the response from queue manager (the message in this case) to the waiting application thread. The Remote Receive Thread doesn't need the SendLock. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Oct 17, 2017 1:08 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
You mean to say that the client waiting on an MQGET does not lock the socket on the client machine?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| tomleend |
| Posted: Tue Oct 17, 2017 1:33 pm Post subject: |
|
|
Acolyte
Joined: 24 Jan 2014
Posts: 51
|
Yes, that's correct. A thread will only lock the socket (actually just its InputStream) while it writes data. It can then drop the lock and let another application thread write some data to the same socket, in order to request whatever it wants from the queue manager. Each thread does not need to lock the socket and prevent others writing to it until a response is received.
The requests from the client and their corresponding responses from the queue manager are tied together using a conversation identifier. That's how multiple conversations (sometimes called connection handles or hconns) can independently coexist over a single socket without getting in the way of each other too much.
Obviously, with a higher SHARECNV value more conversations can be allocated per socket which increases contention in the client for the SendLock. This may be important when sending large messages over slow networks as each send takes longer to complete causing threads to wait. |
|
| Back to top |
|
|
|
|
