| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Messages randomly ignored by consumers |
« View previous topic :: View next topic » |
| Author |
Message
|
| virgo47 |
 Posted: Wed Feb 04, 2009 8:24 am Post subject: Messages randomly ignored by consumers |
 |
|
Newbie
Joined: 04 Feb 2009
Posts: 2
|
Hello
I've searched if someone experienced something similar, but I can't find anything. This forum was recommended and so I try here. We have both standalone JVM applications and Java EE 5 applications running on top of Websphere MQ 6 (can't tell the exact version but I can find out) cluster with our Queue manager on a Solaris 10/Sparc machine. We use IBM's Java driver (/opt/mqm/java/lib/...) and JMS generic RA for Glassfish application server.
The trouble is that sometimes messages that comes to us are ignored. There are two scenarios when this happens and they occur pretty randomly:
1) "synchronous" communication from standalone JVM application
We obtain Q objects from naming (LDAP), init the whole stuff (connection, session, sender, receiver) and send the message waiting for a reply which has the same correlation-id like the msg-id of the request. It works most of the time, but sometimes the response is not read. We wait 30 s (or so) and then report reply timeout. But the customer can prove that the put time of the reply message into our QManager was in a matter of seconds after the request. It works most of the time and sometimes ... it doesn't.
2) Java EE 5, MDB, Glassfish, Generic JMS RA. Queues and ConnFactories are objects in Glassfish JNDI. MDB receives messages, it works most of the time and sometimes it simply ignores the message. Customer calls that some expected process didn't pass and we investigate. The results are: The messages are in the Q that is read by the MDB, application seems to work (it's actually just a few hours after night restart in most cases when the problem occurs but the trouble was the same without restart), but messages are ignored. After redeploying that particular application MDB starts to work. After restart of the AppServer, the same.
Trouble is that we don't know MQ Series into details and it's managed by the customer. But as of now the blame is ours. Did anyone experienced anything similar?
Thanks very much for any hint
Virgo |
|
| Back to top |
|
 |
| Vitor |
| Posted: Wed Feb 04, 2009 8:33 am Post subject: Re: Messages randomly ignored by consumers |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| virgo47 wrote: |
| Thanks very much for any hint |
If messages are not being read off a queue, check the correlation id matches the value you're expecting especially if a) it's being generated by something other than a queue manager or b) any application anywhere is pretending it's a string.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Feb 04, 2009 9:19 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
| Quote: |
| The messages are in the Q that is read by the MDB, application seems to work (it's actually just a few hours after night restart in most cases when the problem occurs but the trouble was the same without restart), but messages are ignored. After redeploying that particular application MDB starts to work. After restart of the AppServer, the same. |
To add to what Vitor said...
As you say you consume the messages after start up we will assume that there is no filter present... or that no message selector is being used.
In this case leave everything up and verify that you can browse the messages waiting to be consumed with a tool like MO71, RFHUtil, MQExplorer, etc...
It could be that a restart of the appserver commits messages (on shutdown) that are part of the queue depth, but not yet committed. And the time on the message is the put time, not the commit time...
Now if your system is busy and you have delivery set by priority make sure that the number of MDB instances is able to handle the load.
You might hit the rare case where messages with lower priority just wait on the queue because messages with higher priority get all the available cycles... Scaling the receiver (MDB instances) would take care of that by allowing a greater throughput. Be careful as message affinity prevents you from scaling the receiver.
The other parameter to look at is the JMS connection pool size. If it is too small you could be starved for connections and have a limited throughput due to connection starvation... Each MDB will need the number of (max) instances + 1 connections. Leave at least a margin to account for outgoing connections.
Hope it helps some 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| virgo47 |
| Posted: Wed Feb 04, 2009 9:49 am Post subject: |
|
|
Newbie
Joined: 04 Feb 2009
Posts: 2
|
Thank you very much for replies.
Vitor: correlation-id is definitely not an issue here, we've checked it many times. There are even cases when we do this synchronous communications for many various cases one after another and first dozens were ignored and then it suddenly started working. But in most cases it works fine. It's only a matter of some days and occurrence is completely random for us.
fjb_saper: maybe, just maybe it's a matter of connection. However the system is used to do it just a few hours after complete restart and the behavior is perfectly timed with the same problem on standalone JVM. The whole application bundle consists of a few JVM modules and few EARs on an application server - these are interconnected with imq (Sun's JMS, probably pretty close to mq.dev.java.net) - we never noticed any anomaly on this messaging.
But WSMQ is used to communicate with customer's other system and in some rare cases it has these ignorant manners.  When this occurs it occurs simultaneously on JVM (synchronous fashion with correlation-id selector) and appserver (MDB, no selector). Various reply queues. That's why I'm skeptic about connection pool relation.
You can imagine how this makes me crazy. It actually happens for nearly a year after customer moved from separated MQ Series 5 queue managers to WS MQ 6 cluster solution, but I doubt the problem is in the cluster because when customer calls us that something didn't start (MDB ignored the message) we can find the messages in the Q (browsing, runmqsc/dis q, probably any way you'd choose) - and they are perfectly consumed after our application restarts. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Feb 04, 2009 2:05 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
Could it be that the MDB gets a bad thread once in a while and that releasing that dead thread (recycling the app) will roll back the message to MQ to have it again in a gettable state. In any case this is going to be an arduous search and you are probably better off opening a PMR with IBM about it.
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
