| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| The Dreaded "Enlist" Exception |
« View previous topic :: View next topic » |
| Author |
Message
|
| amit_talpade_star |
 Posted: Wed Jan 11, 2006 3:12 am Post subject: The Dreaded "Enlist" Exception |
 |
|
Apprentice
Joined: 14 Dec 2004
Posts: 39
|
I am having a harrowing time with this intermittently occuring error.
I have tried everything from MQ (CSD11) and WAS upgrades(5.1.1.5) to debugging the application code and checking the firewall rules (>60 min inactivity timeout) between our WAS server and our MQ server.
The connection and session pools for the connection factories purge and recreate all connections every 900 seconds (15 minutes)
BOTHRESH is also set to less than the retries for MDB Listener port (poison message scenario)
I end up seeing this error from time to time and do not know the trigger for this. From IBM documentation I understand that this could be due to some dead lock with MQ resources, however the queue manager logs at the same time do not show any errors.
Please help with your valuable suggestions and comments so that this error can be fixed.
[1/11/06 6:50:38:308 GMT] 4e34d9cd XATransaction E J2CA0027E: An exception occurred while invoking start on an XA Resource Adapter from dataSource JMS$BRKQCF$JMSManagedConnection@157440496, within transaction ID {XID: formatId(57415344), gtrid_length(36), bqual_length(2 , data(000000000000000400000001e1a113f4383299c549b6232c2ffe36b5c7e3438543505331e1a113f4383299c549b6232c2ffe36b5c7e343850000000100000000)}: javax.transaction.xa.XAException: XA operation failed, see errorCode
at com.ibm.mq.MQXAResource.start(MQXAResource.java:443)
at com.ibm.ejs.jms.JMSManagedSession$JMSXAResource.start(JMSManagedSession.java:1254)
at com.ibm.ejs.j2c.XATransactionWrapper.start(XATransactionWrapper.java:1287)
at com.ibm.ws.Transaction.JTA.JTAResourceBase.start(JTAResourceBase.java:164)
at com.ibm.ws.Transaction.JTA.RegisteredResources.startRes(RegisteredResources.java:391)
at com.ibm.ws.Transaction.JTA.TransactionImpl.enlistResource(TransactionImpl.java:1999)
at com.ibm.ws.Transaction.JTA.TranManagerSet.enlist(TranManagerSet.java:491)
at com.ibm.ejs.j2c.XATransactionWrapper.enlist(XATransactionWrapper.java:612)
at com.ibm.ejs.j2c.ConnectionEventListener.interactionPending(ConnectionEventListener.java:753)
at com.ibm.ejs.jms.JMSManagedSession.interactionPending(JMSManagedSession.java:1010)
at com.ibm.ejs.jms.JMSManagedSession.enlist(JMSManagedSession.java:853)
at com.ibm.ejs.jms.JMSSessionHandle.enlist(JMSSessionHandle.java:954)
at com.ibm.ejs.jms.listener.ServerSession.onMessage(ServerSession.java:509)
at com.ibm.ejs.jms.listener.ServerSession.dispatch(ServerSession.java:492)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:5
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:60)
at java.lang.reflect.Method.invoke(Method.java:391)
at com.ibm.ejs.jms.listener.ServerSessionDispatcher.dispatch(ServerSessionDispatcher.java:37)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:91)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:127)
at com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:373)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:912)
.
[1/11/06 6:50:38:319 GMT] 4e34d9cd RegisteredRes E WTRN0078E: An attempt by the transaction manager to call start on a transactional resource has resulted in an error. The error code was XAER_RMERR. The exception stack trace follows: javax.transaction.xa.XAException: XA operation failed, see errorCode
at com.ibm.mq.MQXAResource.start(MQXAResource.java:443)
at com.ibm.ejs.jms.JMSManagedSession$JMSXAResource.start(JMSManagedSession.java:1254)
at com.ibm.ejs.j2c.XATransactionWrapper.start(XATransactionWrapper.java:1287)
at com.ibm.ws.Transaction.JTA.JTAResourceBase.start(JTAResourceBase.java:164)
at com.ibm.ws.Transaction.JTA.RegisteredResources.startRes(RegisteredResources.java:391)
at com.ibm.ws.Transaction.JTA.TransactionImpl.enlistResource(TransactionImpl.java:1999)
at com.ibm.ws.Transaction.JTA.TranManagerSet.enlist(TranManagerSet.java:491)
at com.ibm.ejs.j2c.XATransactionWrapper.enlist(XATransactionWrapper.java:612)
at com.ibm.ejs.j2c.ConnectionEventListener.interactionPending(ConnectionEventListener.java:753)
at com.ibm.ejs.jms.JMSManagedSession.interactionPending(JMSManagedSession.java:1010)
at com.ibm.ejs.jms.JMSManagedSession.enlist(JMSManagedSession.java:853)
at com.ibm.ejs.jms.JMSSessionHandle.enlist(JMSSessionHandle.java:954)
at com.ibm.ejs.jms.listener.ServerSession.onMessage(ServerSession.java:509)
at com.ibm.ejs.jms.listener.ServerSession.dispatch(ServerSession.java:492)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:5
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:60)
at java.lang.reflect.Method.invoke(Method.java:391)
at com.ibm.ejs.jms.listener.ServerSessionDispatcher.dispatch(ServerSessionDispatcher.java:37)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:91)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:127)
at com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:373)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:912)
[1/11/06 6:50:38:328 GMT] 4e34d9cd XATransaction E J2CA0030E: Method enlist caught javax.transaction.SystemException: Failed to start the transaction association.
at com.ibm.ws.Transaction.JTA.TransactionImpl.enlistResource(TransactionImpl.java:2013)
at com.ibm.ws.Transaction.JTA.TranManagerSet.enlist(TranManagerSet.java:491)
at com.ibm.ejs.j2c.XATransactionWrapper.enlist(XATransactionWrapper.java:612)
at com.ibm.ejs.j2c.ConnectionEventListener.interactionPending(ConnectionEventListener.java:753)
at com.ibm.ejs.jms.JMSManagedSession.interactionPending(JMSManagedSession.java:1010)
at com.ibm.ejs.jms.JMSManagedSession.enlist(JMSManagedSession.java:853)
at com.ibm.ejs.jms.JMSSessionHandle.enlist(JMSSessionHandle.java:954)
at com.ibm.ejs.jms.listener.ServerSession.onMessage(ServerSession.java:509)
at com.ibm.ejs.jms.listener.ServerSession.dispatch(ServerSession.java:492)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:5
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:60)
at java.lang.reflect.Method.invoke(Method.java:391)
at com.ibm.ejs.jms.listener.ServerSessionDispatcher.dispatch(ServerSessionDispatcher.java:37)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:91)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:127)
at com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:373)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:912)
Caused by: javax.transaction.xa.XAException: XA operation failed, see errorCode
at com.ibm.mq.MQXAResource.start(MQXAResource.java:443)
at com.ibm.ejs.jms.JMSManagedSession$JMSXAResource.start(JMSManagedSession.java:1254)
at com.ibm.ejs.j2c.XATransactionWrapper.start(XATransactionWrapper.java:1287)
at com.ibm.ws.Transaction.JTA.JTAResourceBase.start(JTAResourceBase.java:164)
at com.ibm.ws.Transaction.JTA.RegisteredResources.startRes(RegisteredResources.java:391)
at com.ibm.ws.Transaction.JTA.TransactionImpl.enlistResource(TransactionImpl.java:1999)
... 18 more
while trying to enlist resources from datasource JMS$BRKQCF$JMSManagedConnection@157440496 with the Transaction Manager for the current transaction, and threw a ResourceException.
[1/11/06 6:50:38:339 GMT] 4e34d9cd ConnectionEve A J2CA0056I: The Connection Manager received a fatal connection error from the Resource Adaptor for resource JMS$BRKQCF$JMSManagedConnection@157440496. The exception which was received is javax.jms.JMSException: Enlist failed
[1/11/06 6:50:38:343 GMT] 4e34d9cd ConnectionEve A J2CA0056I: The Connection Manager received a fatal connection error from the Resource Adaptor for resource jms/BRKQCF. The exception which was received is javax.jms.JMSException: Enlist failed
[1/11/06 6:50:38:347 GMT] 4e34d9cd JMSExceptionL E WMSG0018E: Error on JMSConnection for MDB HTHRequestFailOverMDB , JMSDestination jms/MUA_MB2CA_REQ : javax.jms.JMSException: Enlist failed
at com.ibm.ejs.jms.JMSCMUtils.mapToJMSException(JMSCMUtils.java:107)
at com.ibm.ejs.jms.JMSManagedSession.enlist(JMSManagedSession.java:871)
at com.ibm.ejs.jms.JMSSessionHandle.enlist(JMSSessionHandle.java:954)
at com.ibm.ejs.jms.listener.ServerSession.onMessage(ServerSession.java:509)
at com.ibm.ejs.jms.listener.ServerSession.dispatch(ServerSession.java:492)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:5
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:60)
at java.lang.reflect.Method.invoke(Method.java:391)
at com.ibm.ejs.jms.listener.ServerSessionDispatcher.dispatch(ServerSessionDispatcher.java:37)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:91)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:127)
at com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:373)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:912)
---- Begin backtrace for Nested Throwables
javax.resource.ResourceException: enlist: caught Exception
at com.ibm.ejs.j2c.XATransactionWrapper.enlist(XATransactionWrapper.java:713)
at com.ibm.ejs.j2c.ConnectionEventListener.interactionPending(ConnectionEventListener.java:753)
at com.ibm.ejs.jms.JMSManagedSession.interactionPending(JMSManagedSession.java:1010)
at com.ibm.ejs.jms.JMSManagedSession.enlist(JMSManagedSession.java:853)
at com.ibm.ejs.jms.JMSSessionHandle.enlist(JMSSessionHandle.java:954)
at com.ibm.ejs.jms.listener.ServerSession.onMessage(ServerSession.java:509)
at com.ibm.ejs.jms.listener.ServerSession.dispatch(ServerSession.java:492)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:5
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:60)
at java.lang.reflect.Method.invoke(Method.java:391)
at com.ibm.ejs.jms.listener.ServerSessionDispatcher.dispatch(ServerSessionDispatcher.java:37)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:91)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:127)
at com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:373)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:912)
---- Begin backtrace for Nested Throwables
javax.transaction.SystemException: Failed to start the transaction association.
at com.ibm.ws.Transaction.JTA.TransactionImpl.enlistResource(TransactionImpl.java:2013)
at com.ibm.ws.Transaction.JTA.TranManagerSet.enlist(TranManagerSet.java:491)
at com.ibm.ejs.j2c.XATransactionWrapper.enlist(XATransactionWrapper.java:612)
at com.ibm.ejs.j2c.ConnectionEventListener.interactionPending(ConnectionEventListener.java:753)
at com.ibm.ejs.jms.JMSManagedSession.interactionPending(JMSManagedSession.java:1010)
at com.ibm.ejs.jms.JMSManagedSession.enlist(JMSManagedSession.java:853)
at com.ibm.ejs.jms.JMSSessionHandle.enlist(JMSSessionHandle.java:954)
at com.ibm.ejs.jms.listener.ServerSession.onMessage(ServerSession.java:509)
at com.ibm.ejs.jms.listener.ServerSession.dispatch(ServerSession.java:492)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:5
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:60)
at java.lang.reflect.Method.invoke(Method.java:391)
at com.ibm.ejs.jms.listener.ServerSessionDispatcher.dispatch(ServerSessionDispatcher.java:37)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:91)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:127)
at com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:373)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:912)
Caused by: javax.transaction.xa.XAException: XA operation failed, see errorCode
at com.ibm.mq.MQXAResource.start(MQXAResource.java:443)
at com.ibm.ejs.jms.JMSManagedSession$JMSXAResource.start(JMSManagedSession.java:1254)
at com.ibm.ejs.j2c.XATransactionWrapper.start(XATransactionWrapper.java:1287)
at com.ibm.ws.Transaction.JTA.JTAResourceBase.start(JTAResourceBase.java:164)
at com.ibm.ws.Transaction.JTA.RegisteredResources.startRes(RegisteredResources.java:391)
at com.ibm.ws.Transaction.JTA.TransactionImpl.enlistResource(TransactionImpl.java:1999)
... 18 more
[1/11/06 6:50:38:363 GMT] 4e34d9cd JMSExceptionL E WMSG0057E: Error on JMSConnection for MDB HTHRequestFailOverMDB , JMSDestination jms/MUA_MB2CA_REQ , JMS Linked Exception : javax.resource.ResourceException: enlist: caught Exception
at com.ibm.ejs.j2c.XATransactionWrapper.enlist(XATransactionWrapper.java:713)
at com.ibm.ejs.j2c.ConnectionEventListener.interactionPending(ConnectionEventListener.java:753)
at com.ibm.ejs.jms.JMSManagedSession.interactionPending(JMSManagedSession.java:1010)
at com.ibm.ejs.jms.JMSManagedSession.enlist(JMSManagedSession.java:853)
at com.ibm.ejs.jms.JMSSessionHandle.enlist(JMSSessionHandle.java:954)
at com.ibm.ejs.jms.listener.ServerSession.onMessage(ServerSession.java:509)
at com.ibm.ejs.jms.listener.ServerSession.dispatch(ServerSession.java:492)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:5
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:60)
at java.lang.reflect.Method.invoke(Method.java:391)
at com.ibm.ejs.jms.listener.ServerSessionDispatcher.dispatch(ServerSessionDispatcher.java:37)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:91)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:127)
at com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:373)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:912)
---- Begin backtrace for Nested Throwables
javax.transaction.SystemException: Failed to start the transaction association.
at com.ibm.ws.Transaction.JTA.TransactionImpl.enlistResource(TransactionImpl.java:2013)
at com.ibm.ws.Transaction.JTA.TranManagerSet.enlist(TranManagerSet.java:491)
at com.ibm.ejs.j2c.XATransactionWrapper.enlist(XATransactionWrapper.java:612)
at com.ibm.ejs.j2c.ConnectionEventListener.interactionPending(ConnectionEventListener.java:753)
at com.ibm.ejs.jms.JMSManagedSession.interactionPending(JMSManagedSession.java:1010)
at com.ibm.ejs.jms.JMSManagedSession.enlist(JMSManagedSession.java:853)
at com.ibm.ejs.jms.JMSSessionHandle.enlist(JMSSessionHandle.java:954)
at com.ibm.ejs.jms.listener.ServerSession.onMessage(ServerSession.java:509)
at com.ibm.ejs.jms.listener.ServerSession.dispatch(ServerSession.java:492)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:5
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:60)
at java.lang.reflect.Method.invoke(Method.java:391)
at com.ibm.ejs.jms.listener.ServerSessionDispatcher.dispatch(ServerSessionDispatcher.java:37)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:91)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:127)
at com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:373)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:912)
Caused by: javax.transaction.xa.XAException: XA operation failed, see errorCode
at com.ibm.mq.MQXAResource.start(MQXAResource.java:443)
at com.ibm.ejs.jms.JMSManagedSession$JMSXAResource.start(JMSManagedSession.java:1254)
at com.ibm.ejs.j2c.XATransactionWrapper.start(XATransactionWrapper.java:1287)
at com.ibm.ws.Transaction.JTA.JTAResourceBase.start(JTAResourceBase.java:164)
at com.ibm.ws.Transaction.JTA.RegisteredResources.startRes(RegisteredResources.java:391)
at com.ibm.ws.Transaction.JTA.TransactionImpl.enlistResource(TransactionImpl.java:1999)
... 18 more
[1/11/06 6:50:38:585 GMT] 4e34d9cd MDBListenerIm I WMSG0043I: MDB Listener Above2HTHRequestLPT2 stopped for JMSDestination jms/MUA_MB2CA_REQ
[1/11/06 6:50:38:588 GMT] 4e34d9cd MDBListenerIm I WMSG0058I: Listener Port Above2HTHRequestLPT2 will attempt to restart in 60 seconds
[1/11/06 6:50:38:598 GMT] 4e34d9cd ExceptionUtil E CNTR0020E: Non-application exception occurred while processing method "onMessage" on bean "BeanId(Adapter#HTHEJB.jar#HTHRequestFailOverMDB, null)". Exception data: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:5
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:60)
at java.lang.reflect.Method.invoke(Method.java:391)
at com.ibm.ejs.jms.listener.ServerSessionDispatcher.dispatch(ServerSessionDispatcher.java:37)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:91)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:127)
at com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:373)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:912)
Caused by: javax.jms.JMSException: Enlist failed
at com.ibm.ejs.jms.JMSCMUtils.mapToJMSException(JMSCMUtils.java:107)
at com.ibm.ejs.jms.JMSManagedSession.enlist(JMSManagedSession.java:871)
at com.ibm.ejs.jms.JMSSessionHandle.enlist(JMSSessionHandle.java:954)
at com.ibm.ejs.jms.listener.ServerSession.onMessage(ServerSession.java:509)
at com.ibm.ejs.jms.listener.ServerSession.dispatch(ServerSession.java:492)
... 10 more
---- Begin backtrace for Nested Throwables
javax.resource.ResourceException: enlist: caught Exception
at com.ibm.ejs.j2c.XATransactionWrapper.enlist(XATransactionWrapper.java:713)
at com.ibm.ejs.j2c.ConnectionEventListener.interactionPending(ConnectionEventListener.java:753)
at com.ibm.ejs.jms.JMSManagedSession.interactionPending(JMSManagedSession.java:1010)
at com.ibm.ejs.jms.JMSManagedSession.enlist(JMSManagedSession.java:853)
at com.ibm.ejs.jms.JMSSessionHandle.enlist(JMSSessionHandle.java:954)
at com.ibm.ejs.jms.listener.ServerSession.onMessage(ServerSession.java:509)
at com.ibm.ejs.jms.listener.ServerSession.dispatch(ServerSession.java:492)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:5
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:60)
at java.lang.reflect.Method.invoke(Method.java:391)
at com.ibm.ejs.jms.listener.ServerSessionDispatcher.dispatch(ServerSessionDispatcher.java:37)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:91)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:127)
at com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:373)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:912)
---- Begin backtrace for Nested Throwables
javax.transaction.SystemException: Failed to start the transaction association.
at com.ibm.ws.Transaction.JTA.TransactionImpl.enlistResource(TransactionImpl.java:2013)
at com.ibm.ws.Transaction.JTA.TranManagerSet.enlist(TranManagerSet.java:491)
at com.ibm.ejs.j2c.XATransactionWrapper.enlist(XATransactionWrapper.java:612)
at com.ibm.ejs.j2c.ConnectionEventListener.interactionPending(ConnectionEventListener.java:753)
at com.ibm.ejs.jms.JMSManagedSession.interactionPending(JMSManagedSession.java:1010)
at com.ibm.ejs.jms.JMSManagedSession.enlist(JMSManagedSession.java:853)
at com.ibm.ejs.jms.JMSSessionHandle.enlist(JMSSessionHandle.java:954)
at com.ibm.ejs.jms.listener.ServerSession.onMessage(ServerSession.java:509)
at com.ibm.ejs.jms.listener.ServerSession.dispatch(ServerSession.java:492)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:5
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:60)
at java.lang.reflect.Method.invoke(Method.java:391)
at com.ibm.ejs.jms.listener.ServerSessionDispatcher.dispatch(ServerSessionDispatcher.java:37)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:91)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:127)
at com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:373)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:912)
Caused by: javax.transaction.xa.XAException: XA operation failed, see errorCode
at com.ibm.mq.MQXAResource.start(MQXAResource.java:443)
at com.ibm.ejs.jms.JMSManagedSession$JMSXAResource.start(JMSManagedSession.java:1254)
at com.ibm.ejs.j2c.XATransactionWrapper.start(XATransactionWrapper.java:1287)
at com.ibm.ws.Transaction.JTA.JTAResourceBase.start(JTAResourceBase.java:164)
at com.ibm.ws.Transaction.JTA.RegisteredResources.startRes(RegisteredResources.java:391)
at com.ibm.ws.Transaction.JTA.TransactionImpl.enlistResource(TransactionImpl.java:1999)
... 18 more
[1/11/06 6:50:38:669 GMT] 4e34d9cd ServerSession W WMSG0031E: Exception processing JMS Message for MDB HTHRequestFailOverMDB, JMSDestination jms/MUA_MB2CA_REQ : com.ibm.ejs.container.UnknownLocalException: ; nested exception is: javax.jms.JMSException: Enlist failed
at com.ibm.ejs.container.LocalExceptionMappingStrategy.mapException(LocalExceptionMappingStrategy.java:198)
at com.ibm.ejs.container.LocalExceptionMappingStrategy.setUncheckedException(LocalExceptionMappingStrategy.java:266)
at com.ibm.ejs.container.EJSDeployedSupport.setUncheckedLocalException(EJSDeployedSupport.java:318)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:9
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:127)
at com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:373)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:912)
Caused by: javax.jms.JMSException: Enlist failed
at com.ibm.ejs.jms.JMSCMUtils.mapToJMSException(JMSCMUtils.java:107)
at com.ibm.ejs.jms.JMSManagedSession.enlist(JMSManagedSession.java:871)
at com.ibm.ejs.jms.JMSSessionHandle.enlist(JMSSessionHandle.java:954)
at com.ibm.ejs.jms.listener.ServerSession.onMessage(ServerSession.java:509)
at com.ibm.ejs.jms.listener.ServerSession.dispatch(ServerSession.java:492)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:5
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:60)
at java.lang.reflect.Method.invoke(Method.java:391)
at com.ibm.ejs.jms.listener.ServerSessionDispatcher.dispatch(ServerSessionDispatcher.java:37)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:91)
... 3 more
---- Begin backtrace for nested exception
javax.jms.JMSException: Enlist failed
at com.ibm.ejs.jms.JMSCMUtils.mapToJMSException(JMSCMUtils.java:107)
at com.ibm.ejs.jms.JMSManagedSession.enlist(JMSManagedSession.java:871)
at com.ibm.ejs.jms.JMSSessionHandle.enlist(JMSSessionHandle.java:954)
at com.ibm.ejs.jms.listener.ServerSession.onMessage(ServerSession.java:509)
at com.ibm.ejs.jms.listener.ServerSession.dispatch(ServerSession.java:492)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:5
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:60)
at java.lang.reflect.Method.invoke(Method.java:391)
at com.ibm.ejs.jms.listener.ServerSessionDispatcher.dispatch(ServerSessionDispatcher.java:37)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:91)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:127)
at com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:373)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:912)
---- Begin backtrace for Nested Throwables
javax.resource.ResourceException: enlist: caught Exception
at com.ibm.ejs.j2c.XATransactionWrapper.enlist(XATransactionWrapper.java:713)
at com.ibm.ejs.j2c.ConnectionEventListener.interactionPending(ConnectionEventListener.java:753)
at com.ibm.ejs.jms.JMSManagedSession.interactionPending(JMSManagedSession.java:1010)
at com.ibm.ejs.jms.JMSManagedSession.enlist(JMSManagedSession.java:853)
at com.ibm.ejs.jms.JMSSessionHandle.enlist(JMSSessionHandle.java:954)
at com.ibm.ejs.jms.listener.ServerSession.onMessage(ServerSession.java:509)
at com.ibm.ejs.jms.listener.ServerSession.dispatch(ServerSession.java:492)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:5
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:60)
at java.lang.reflect.Method.invoke(Method.java:391)
at com.ibm.ejs.jms.listener.ServerSessionDispatcher.dispatch(ServerSessionDispatcher.java:37)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:91)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:127)
at com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:373)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:912)
---- Begin backtrace for Nested Throwables
javax.transaction.SystemException: Failed to start the transaction association.
at com.ibm.ws.Transaction.JTA.TransactionImpl.enlistResource(TransactionImpl.java:2013)
at com.ibm.ws.Transaction.JTA.TranManagerSet.enlist(TranManagerSet.java:491)
at com.ibm.ejs.j2c.XATransactionWrapper.enlist(XATransactionWrapper.java:612)
at com.ibm.ejs.j2c.ConnectionEventListener.interactionPending(ConnectionEventListener.java:753)
at com.ibm.ejs.jms.JMSManagedSession.interactionPending(JMSManagedSession.java:1010)
at com.ibm.ejs.jms.JMSManagedSession.enlist(JMSManagedSession.java:853)
at com.ibm.ejs.jms.JMSSessionHandle.enlist(JMSSessionHandle.java:954)
at com.ibm.ejs.jms.listener.ServerSession.onMessage(ServerSession.java:509)
at com.ibm.ejs.jms.listener.ServerSession.dispatch(ServerSession.java:492)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:5
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:60)
at java.lang.reflect.Method.invoke(Method.java:391)
at com.ibm.ejs.jms.listener.ServerSessionDispatcher.dispatch(ServerSessionDispatcher.java:37)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:91)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:127)
at com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:373)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:912)
Caused by: javax.transaction.xa.XAException: XA operation failed, see errorCode
at com.ibm.mq.MQXAResource.start(MQXAResource.java:443)
at com.ibm.ejs.jms.JMSManagedSession$JMSXAResource.start(JMSManagedSession.java:1254)
at com.ibm.ejs.j2c.XATransactionWrapper.start(XATransactionWrapper.java:1287)
at com.ibm.ws.Transaction.JTA.JTAResourceBase.start(JTAResourceBase.java:164)
at com.ibm.ws.Transaction.JTA.RegisteredResources.startRes(RegisteredResources.java:391)
at com.ibm.ws.Transaction.JTA.TransactionImpl.enlistResource(TransactionImpl.java:1999)
... 18 more |
|
| Back to top |
|
 |
| mvic |
| Posted: Wed Jan 11, 2006 3:54 am Post subject: Re: The Dreaded "Enlist" Exception |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| amit_talpade_star wrote: |
I have tried everything from MQ (CSD11) and WAS upgrades(5.1.1.5) to debugging the application code and checking the firewall rules (>60 min inactivity timeout) between our WAS server and our MQ server.
The connection and session pools for the connection factories purge and recreate all connections every 900 seconds (15 minutes)
...snip...
I end up seeing this error from time to time and do not know the trigger for this. From IBM documentation I understand that this could be due to some dead lock with MQ resources |
Could you refer us directly to the piece of documentation you're quoting please?
I saw enlist errors with a big WAS/MQ system a couple of years ago now - so I'm a bit doubtful it is the same, unless it was not fixed.
The root cause at that time was an active piece of network hardware that was cancelling all sockets after 60 minutes "inactivity".
MQ makes heavy use of "persistent" network connections. That means they can go inactive for any length of time, and this is not a problem.
With the WAS/MQ system I referred to, their system had built a pool of MQ connections by the end of the day, and workload was light during the night. The pool did not serve the connections out of the pool in a round-robin manner, so one went quiet for over an hour. The active network hardware cancelled the connection. Sometime in the night, the work picked up, and the pool served the quiet (now cancelled) connection. WAS tried to send an XA request to MQ. But the connection was dead. We saw an "enlist" error as a direct result of this.
(I don't have ready access to my notes from the time, so can't compare the exception you're seeing. In any case the code is likely to have been altered somewhat in the 2 years elapsed).
My advice, therefore, is to tune all active parts of your network between WAS and MQ (including firewalls - this may be difficult) so that they do not cancel any active connections. |
|
| Back to top |
|
|
| amit_talpade_star |
| Posted: Wed Jan 11, 2006 4:19 am Post subject: |
|
|
Apprentice
Joined: 14 Dec 2004
Posts: 39
|
The IBM support page which I was referring to is
http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg1PK00582
However this does not state how the deadlock can be prevented from happening.
Do you have any specific suggestions or may be recollections on what were the steps undertaken in tuning the firewalls? That would be most helpful
What we have already done is to analyze firewall logs to see for connection drops to port 1414 on our MQ Server. And to keep all connections to port 1414 on MQ Server active for 24 hours. |
|
| Back to top |
|
|
| mvic |
| Posted: Wed Jan 11, 2006 4:50 am Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
There's a difference which seems important to me : you have a javax.transaction.xa.XAException but the document you reference has not got this. This probably means that your issue is not the same as the deadlock issue at that page. The XA exception also matches my memory of the WAS/MQ situation I described.
| Quote: |
Do you have any specific suggestions or may be recollections on what were the steps undertaken in tuning the firewalls? That would be most helpful
What we have already done is to analyze firewall logs to see for connection drops to port 1414 on our MQ Server. And to keep all connections to port 1414 on MQ Server active for 24 hours. |
I'm sorry but at that time I was supporting the WAS/MQ customer remotely, and was not directly involved with looking at their network infrastructure. In fact for a long time in the problem, it was thought the active hardware was NOT unilaterally breaking connections, but eventually it emerged that that was exactly what it was doing. The network admins did all of the necessary changes, but I never got to hear any precise details.
Is it possible to get the firewall to leave MQ connections alone altogether? It is quite normal for them to stay up as long as the queue manager is up. Indeed, cancelling them at any time - even after 24 hours of inactivity - is bound to cause some problems of the sort I described before.
If really necessary you might like to look into the MQIPT SupportPac - but this takes time and effort to learn and administer, and I am not aware of how well it works in a WAS system, if at all.
Also, have you raised a call with IBM WAS or MQ Support?
HTH. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Jan 11, 2006 4:52 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
I'm confused.
How could an external network firewall interfer with the bindings connection required for XA? Are we certain that amit_talpade_star is using the Extended Transactional Client?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| mvic |
| Posted: Wed Jan 11, 2006 5:40 am Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| jefflowrey wrote: |
| How could an external network firewall interfer with the bindings connection required for XA? Are we certain that amit_talpade_star is using the Extended Transactional Client? |
I assumed the ETC was in use because of this part of the description : "checking the firewall rules (>60 min inactivity timeout) between our WAS server and our MQ server." |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jan 11, 2006 2:44 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
I would suggest checking the JMS Topology paper and making sure the pool size is big enough. At the same time you need to check if the max handles set on the qmgr will support the pool size you are gunning for.
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
