MQSeries.net :: View topic

fredand44 · Posted: Sat May 31, 2008 8:07 am Post subject:

Hello guys!

I really need som comment for this problem, and I need it fast.
We are suppose to deliver a system upgrade into our production environment tomorrow!
Yesterday we experience a very odd behavior in our production_like_environment that we have not experience before.
All other test has been fine!

First, please take a look at our system architecture.
The system look

System A
Weblogic 8.1.5
In this we got an MDB registred for listening for messages at a queue at MQ called MQQUEUE_A.
It finds the queue at MQ through a .bindings jndi

MQ-server A
This is where the queue MQQUEUE_A resides.

MQ-broker routes messages
Routes messages between queue MQQUEUE_A and queue MQQUEUE_B.

Mq-server B
This is where the queue MQQUEUE_B resides.
This server is our connection point to our collaborate partners.

Firewall

Mq-server C
This is where the queue MQQUEUE_C resides.
This server is the server at our collaborate partners.

So the line looks like:
MQQUEUE_C -> (Firewall) -> MQQUEUE_B -> MQ-broker -> MQQUEUE_A -> MDB.

At thursday night everything was working fine.

At friday morning our collaborate partners posted two messages at MQQUEUE_C as normal, just for a simple last test.
But they did ot arrive!!!

After some investigation our collaborate partners find out that the channel between MQQUEUE_C -> MQQUEUE_B was in retrying mode. (The two messages was still at the MQQUEUE_C).

Our Firwall-department confirmed that Mq-server C and Mq-server B was talking with each other so nothing was wrong in the Firewall.

Our collaborate partners tried to restart the channel. Voila the messages was sent away.
... but they did not go all the way to our MDB.

After some more investigation we saw that the queue depth was 1 at MQQUEUE_A. (But 2 messages was sent!!!)
In our serverlog at our Weblogic we now was getting logging, every approx. 20 sec, that said something like: "The MDB has reconnect to MQQUEUE_A". In our MQ-server A we was getting logging saying something like "This error might have to due because of connection error".

We tried to restart our Weblogic. During the time the Weblogic was down the queue depth was 2 at MQQUEUE_A. This looks like then that the two messages was luckely sent all the way to our queue MQQUEUE_A!! We gues that the first message was held in the RAM then at MQ-server A.

When our Weblogic was restarted we again get the same error message in the logs, both at Weblogic and and in MQ-server A.
We then tried to temporary remove the 2 messages and sent a new message from our collaborate partner . That message vent all the way through to our MDB!!!

We then put back the 2 messages, which gave us the same error message again in log files.

Our conclusion is that the 2 messages was in some way corrupted! But we really need to find out why and how to prevent this.

In Weblogic we are pretty sure that the MDB itself was never getting the messages. If I understand it correct our weblogic first get the message from MQ and then hands them over to our MDB. Below is our config for our MDB.

ejb-jar.xml:
<message-driven>
<ejb-name>MqToJms</ejb-name>
<ejb-class>com.ourcompany.MqToJmsMDB</ejb-class>
<transaction-type>Bean</transaction-type>
<acknowledge-mode>Auto-acknowledge</acknowledge-mode>
<message-driven-destination>
<destination-type>javax.jms.Queue</destination-type>
<subscription-durability>Durable</subscription-durability>
</message-driven-destination>
</message-driven>

weblogic-ejb-jar.xml
<weblogic-enterprise-bean>
<ejb-name>MqToJms</ejb-name>
<message-driven-descriptor>
<pool>
<max-beans-in-free-pool>1</max-beans-in-free-pool>
<initial-beans-in-free-pool>1</initial-beans-in-free-pool>
</pool>
<destination-jndi-name>MQQUEUE_A</destination-jndi-name>
<initial-context-factory>com.sun.jndi.fscontext.RefFSContextFactory</initial-context-factory>
<provider-url>file:path/to/jndi</provider-url>
<connection-factory-jndi-name>queue.connection.factory</connection-factory-jndi-name>
<jms-polling-interval-seconds>10</jms-polling-interval-seconds>
</message-driven-descriptor>
<reference-descriptor>
</reference-descriptor>
</weblogic-enterprise-bean>

in folder file:path/to/jndi we got an .bindings file describing the jndi-tree perhaps the error is in that??

If any one got any idea of why we get this mysterious error we really would love to get that.
So please do not hasitate to guess!!
Best regards
Fredrik