| Author |
Message
|
| Tibor |
 Posted: Wed Nov 16, 2005 9:26 am Post subject: Random XA problems with Weblogic + ETC [SOLVED] |
 |
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
We are testing a new application and sometimes arrives this error:
| Quote: |
javax.ejb.TransactionRolledbackLocalException: EJB Exception: ... javax.jms.JMSException: [J2EE:160049]An error occurred while registering the JMS connection with JTA: javax.transaction.SystemException: start() failed on resource 'GPS_DOMAIN.GPS_SERVER_1.JMSXASessionPool.jms/GPSOutputQueueConnectionFactory': XAER_RMERR : A resource manager error has occured in the transaction branch
javax.transaction.xa.XAException: XA operation failed, see errorCode
at com.ibm.mq.MQXAResource.start(MQXAResource.java:443)
at weblogic.deployment.jms.WrappedXAResource_com_ibm_mq_MQXAResource.start(Unknown Source)
at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1140)
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1072)
...
|
Yesterday was a mass test with thousands of messages without any errors. But today the admin had to restart twice the weblogic application, however there was less than 100 messages 
Weblogic: v8.1 sp4
MQ client: v5.3 csd11 (with hotfix) + com.ibm.mqetclient.jar
Oracle: v9.2
Thanks in advance,
Tibor
Last edited by Tibor on Wed Feb 15, 2006 8:43 am; edited 1 time in total |
|
| Back to top |
|
 |
| Tibor |
| Posted: Thu Feb 02, 2006 5:59 pm Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
Firstly, it depends on mq fixpak level: our problem occurs on CSD11 but client works fine on CSD08 (server and client on same level).
After tracing on client side (60MB+) it seems the root of the problem is next: the jms client fails after a network problem (firewall, stop channel, etc). Had anyone seen a similar? There are tons of RC 2019 (MQRC_HOBJ_ERROR):
| Quote: |
00:05:08:[1138705508847] Thread: ExecuteThread: '1' for queue: 'weblogic.ejb20.internal.JMSMessagePoller', Object: com.ibm.mq.MQSPIQueueManager@40096c queue = GPS_ERROR
queue manager = FSQT
dynamic queue name = AMQ.*
alternate user id =
options = 8208
00:05:08:[1138705508847] Thread: ExecuteThread: '1' for queue: 'weblogic.ejb20.internal.JMSMessagePoller' ==> MQSESSIONClient::MQOPEN() entry
00:05:08:[1138705508847] Thread: ExecuteThread: '1' for queue: 'weblogic.ejb20.internal.JMSMessagePoller' Class: MQSESSIONClient Invalid connection handle: 21
00:05:08:[1138705508847] Thread: ExecuteThread: '1' for queue: 'weblogic.ejb20.internal.JMSMessagePoller', Object: com.ibm.mq.MQException: MQJE001: Completion Code 2, Reason 2019 ==> MQException constructor(cc, rc, source)() entry
00:05:08:[1138705508848] Thread: ExecuteThread: '1' for queue: 'weblogic.ejb20.internal.JMSMessagePoller', Object: com.ibm.mq.MQException: MQJE001: Completion Code 2, Reason 2019 javabase/com/ibm/mq/MQException.java, java, j5306, j5306-11-050701 04/04/30 13:57:41 @(#) 1.50.1.4
00:05:08:[1138705508848] Thread: ExecuteThread: '1' for queue: 'weblogic.ejb20.internal.JMSMessagePoller', Object: com.ibm.mq.MQException: MQJE001: Completion Code 2, Reason 2019 cc = 2
...
com.ibm.mq.jms.MQQueueSession@193d23b Throwing javax.jms.JMSException: MQJMS2008: failed to open MQ queue GPS_ERROR |
...and so on.
Tibor |
|
| Back to top |
|
|
| mvic |
| Posted: Fri Feb 03, 2006 1:31 am Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
The title of this thread is now quite different to the latest symptoms.
However, a very cursory glance at the description makes me think that a queue cannot be opened, and then subsequent attempts to use the queue are (understandably) failing with HOBJ_ERROR. Question is, why does the queue fail to be opened?
What code levels are you at on client and server? (I've read your notes on CSD levels, but can't tell what is in place at the time of the latest error). |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Feb 03, 2006 3:21 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Tibor,
I think you might have to dig a little bit deeper into either the reconnect logic of the app or the reconnect logic of Weblogic for the MDB. Especially if the error only appears after a firewall / network problem.
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Tibor |
| Posted: Fri Feb 03, 2006 5:40 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
mvic,
(1) You are right, but this is still same error, just from another point of view. When I wrote my first post it was a high level error. Yesterday we was tracing with setting MQJMS_TRACE_LEVEL=base and finding the real mqrc codes in the output mqjms.trc file.
(2) My second post was ambiguous about CSD level. Server and clients are on CSD11 now, but 3 days ago it was CSD08.
fjb_saber,
I can reproduce the error with this simple command: stop channel(chlname) mode(force) so it didn't bound to a network error.
Tibor |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Feb 03, 2006 11:31 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Tibor wrote: |
| I can reproduce the error with this simple command: stop channel(chlname) mode(force) icon_frown.gif so it didn't bound to a network error. |
I expect you are talking about a svrconn type channel. Your command has the same action as a hickup in communications: The original communication is cut and the handle is no longer valid....
Of course I expect you to have already filtered out the CSD08/09 problem...
I hit it yesterday after using some old CSD08 jar files. Replaced with the jar files from 6 RP1 and had no problems in the client connection.
As I said this has to do with the handling of connection errors that are always more prone to creep up in a client connection. You want to get rid of them... get a server... or write some reconnection logic if you loose the connection...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Tibor |
| Posted: Mon Feb 06, 2006 12:49 pm Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
fjb_saber,
The problem was re-analyzed by application developers and me, and there are new informations:
- the message consumer components (an MDB) re-connects correctly
- only the message sender component fails. As developers told me, they got a connection and a session, handled by jms (thru jndi) and only the last step, the message sender went wrong without correct error message, namely a JMS exception.
Tibor |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Feb 06, 2006 2:17 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Tibor wrote: |
fjb_saber,
without correct error message, namely a JMS exception.
Tibor |
Tibor, 2 words: bizarre, bizarre.
At this point I would suggest PMR time. However have your developers check first that the JMSException is not handled in the code and suppressed (not rethrown)
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Tibor |
| Posted: Wed Feb 08, 2006 2:45 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
Hi fjb,
I'm very angry with developers, because they got the JMS exception!!! 
I've just written a stroppy mail for them. However their last reply asserts the linkedException is missing, but I don't credit at this...
Developers, developers! 
Thanks all,
Tibor |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Feb 08, 2006 6:08 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Tibor wrote: |
Hi fjb,
I'm very angry with developers, because they got the JMS exception!!!
I've just written a stroppy mail for them. However their last reply asserts the linkedException is missing, but I don't credit at this...
Developers, developers!
Thanks all,
Tibor |
Tibor,
Not all JMS Exceptions carry a linked exception! In that case the LinkedException is null.
Anyway this was a good catch. Your Developers need to be more careful and make sure that when catching an exception it is handled properly (which means some need to be rethrown).
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Tibor |
| Posted: Mon Feb 13, 2006 6:47 am Post subject: [Solved] Random XA problems with Weblogic + ETC |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
For everyone who interests in the topic: problem solved!
In details: the developers discovered a special feature in weblogic appserver: caching of connection and object handlers. Message consumer is re-connecting correctly but message sender holding the handlers. That's why after the exception is raising this cache have to be cleaned.
Tibor |
|
| Back to top |
|
|
| mfinkels |
| Posted: Mon Feb 13, 2006 11:49 am Post subject: How did you clean the cache ? |
|
|
Newbie
Joined: 20 Sep 2004
Posts: 5
Location: Toronto, ON Canada
|
Tibor,
How to clean this cache?
Thank you,
Michael |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Feb 13, 2006 1:17 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Just have the guy holding the handle close it before he lets it go.
At the same time you might want to check and maybe force people off the channel after a hickup. (stop channel mode force...)
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mfinkels |
| Posted: Mon Feb 13, 2006 1:25 pm Post subject: |
|
|
Newbie
Joined: 20 Sep 2004
Posts: 5
Location: Toronto, ON Canada
|
Thanks for the reply.
For the case of WebLogic it is the Weblogic pool and the MDBs, which are holding the connections.
It would be nice to have something to refresh the connection pool.
Thanks,
Michael |
|
| Back to top |
|
|
| Tibor |
| Posted: Wed Feb 15, 2006 8:32 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
|
| Back to top |
|
|
|
|
