| Author |
Message
|
| bgrimes |
 Posted: Tue Oct 24, 2006 7:58 am Post subject: Messages stuck on channel |
 |
|
Newbie
Joined: 15 Aug 2006
Posts: 8
|
Does anyone know why MQ does not reteive the messages in the following scenario?
System A:
1) Message PUT onto MQ Queue
System B:
1) Channel shows the message count has been incremented
2) But, MQ does not retreive the message
3) Some period of time later (minutes to hours) the message is retrieved by MQ.
 |
|
| Back to top |
|
 |
| Vitor |
| Posted: Tue Oct 24, 2006 8:07 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Unit of Work on System A not committed or the batching on the channel set a bit funny would be my guesses. Most likely the former than the latter.
Also note that MQ does not retrieve anything. I assume you mean that after x period of time it turns up on the queue because MQ "retrieves" it from the channel, when in fact MQ has just delivered it there.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| DTran |
| Posted: Tue Oct 24, 2006 9:01 pm Post subject: |
|
|
Acolyte
Joined: 11 May 2006
Posts: 62
Location: Amsterdam
|
| or the receiving side don't have enough diskspace, and the transaction is roll back. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Oct 24, 2006 11:43 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| DTran wrote: |
| or the receiving side don't have enough diskspace, and the transaction is roll back. |
So why is it turning up x minutes later? What do you see as the retry mechanism? 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| DTran |
| Posted: Tue Oct 24, 2006 11:56 pm Post subject: |
|
|
Acolyte
Joined: 11 May 2006
Posts: 62
Location: Amsterdam
|
| In this stage channel is running but in doubt state. If diskspace is free again, then message arrived at receiving side. I don't know if this is the case, if not, then the only reason i can think of is the UOW. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Oct 25, 2006 12:23 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Interesting - I'd have thought (if the receiving disk was full) the MCA would have got the equivalent of a 2056, rolled back as you suggest but I would have expected the message to end up on the sender DLQ as undeliverable rather than an indoubt transaction; from the MCA's point of view there's no doubt it hasn't worked.
But I can see what you're getting at. I still think the time delay bgrimes talks about is longer than an MCA would wait. You'd also expect the channel status to change if the MCA is having problems, so as to prevent more messages going down the channel and ending up in doubt.
bgrimes - is there a change in channel status during all of this, or any messages in the queue manager log associated with your problem?
Also, does anyone in the assembled multitude have more authoriative information on what a channel does if the recieving box is out of space? 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| DTran |
| Posted: Wed Oct 25, 2006 1:42 am Post subject: |
|
|
Acolyte
Joined: 11 May 2006
Posts: 62
Location: Amsterdam
|
to tell you the truth, it suprised me the first time it happened. In the sender MQ-log it would looks like this. And if you stop the channel and do the commit, the message can be lost  . This is also happened to me once
-------------------------------------------------------------------------------
10/25/06 07:00:51
AMQ7469: Transactions rolled back to release log space.
EXPLANATION:
The log space for the queue manager is becoming full. One or more long-running
transactions have been rolled back to release log space so that the queue
manager can continue to process requests.
ACTION:
Try to ensure that the duration of your transactions is not excessive. Consider
increasing the size of the log to allow transactions to last longer before the
log starts to become full.
------------------------------------------------------------------------------- |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Oct 25, 2006 1:55 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Ah - so not the machine running out of disk, but the log running out of disk!
Ok, I see where you're coming from now. I would have (perhaps pedantically) said that this is not actually a space problem but another UOW problem - as the action suggests the problem is not that the log has filled up, but that there's a unit of work large enough to fill it..... 
It's a subtle distinction I admit. IHMO unless you have a large number of simultanious applications you shouldn't fill the log; UOW should be small and tight, even in a long running transaction. Though I've seen my share of programs that start at 8am, run through the busness day then commit & close down at 6pm. You do wonder about how long the rollback would take if it hits a problem at 5:58.... 
(Rational is that "we can't commit until the business processing is complete". When there is a rational other than "well it was fine in testing"  )
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| JackOfAllTrades |
| Posted: Wed Oct 25, 2006 9:40 am Post subject: |
|
|
Newbie
Joined: 25 Oct 2006
Posts: 1
Location: Canada
|
| If I may I'll add further information to that supplied by bgrimes (I'm working on the same project). The implementation is on Solaris running MQ v5.3. There are no errors evident in any of the MQ logs and disk (logging or otherwise) doesn't seem to be a problem. What we have is a local-queue that is fed by a local Java-app AND a receive-channel (a remote-system is sending the queue data via a remote-queue definition). We see the Java-app populate the queue OK pretty much regardless of other activity. However, when the remote system sends data while the Java-app is busy the channel negotiates correctly, the messages-in counter for the channel increments BUT the received messages do not immediately appear on the queue. Once the Java-app has been inactive for a period of time (haven't determined exactly how long yet) OR if one were to shutdown the app the messages sent via the channel appear on the queue immediately. The intent, here, is to have the Java-app and the receive-channel interleave all messages in chronological order of receipt. Any and all thoughts or advice greatly appreciated! |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Oct 25, 2006 9:53 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
It is suggestive of the Java application failing to commit GETs.
Or opening the queue for exclusive access, preventing the channel from writing.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| bgrimes |
| Posted: Wed Oct 25, 2006 10:22 am Post subject: More info |
|
|
Newbie
Joined: 15 Aug 2006
Posts: 8
|
We have checked the error logs on both systems and there are no errors.
We checked diskspace and there are gigs available.
One of the first thing we thought of was that the work was not committed, but it was.
We have two feeds to the queue on System B. One is the previously mentioned System A and the other is an application internal to System B. The messages PUT to the queue by the application always makes it through. We verified that the application opens the queue, puts a message then closes the queue.
When a message is PUT to the queue by System B, a success message is returned. We also see the message count on the channel incremented. At this point we are trying to determine how, where and why MQ buffers the messages. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Oct 25, 2006 10:29 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
If a PUT has returned an RC 0, then the message has been accepted for delivery. If the message is not immediately available on the destination queue, and the destination queue is a qremote from the queue manager the PUTing system is connected to - then the message is on the transmission queue. It may be in an uncommitted state - but it *is* on the transmission queue.
Messages that can't be delivered from a transmission queue are either put to the Dead Letter Queue or they are discarded if they are non-persistant. If they are put to the DLQ, then will have an return code on the Dead Letter Header that tells you why the message was PUT to the DLQ.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| bgrimes |
| Posted: Wed Oct 25, 2006 11:06 am Post subject: more info |
|
|
Newbie
Joined: 15 Aug 2006
Posts: 8
|
The DLQ is empty on both machines. Also, the messages are delivered, just later.
We have verified that we can send data from mulitple applications to the queue; however, the information being sent via the channel is held until the other data is complete.
All queues/data are persistent.
We are verifying that the BATCHSZ=50 and batchint=0 on System A; however, we do not believe this to be a batch issue.
It is almost as if the channel has a lower priority than the applications??? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Oct 25, 2006 2:00 pm Post subject: Re: more info |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| bgrimes wrote: |
All queues/data are persistent.
We are verifying that the BATCHSZ=50 and batchint=0 on System A; however, we do not believe this to be a batch issue.
It is almost as if the channel has a lower priority than the applications??? |
Have you checked if the priority by default of the messages coming in through the channel is lower than the priority of the messages being directly put to the queue by apps?
You could as well switch the message delivery from priority to FIFO (queue attribute).
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| bgrimes |
| Posted: Wed Nov 01, 2006 6:02 am Post subject: Problem solved |
|
|
Newbie
Joined: 15 Aug 2006
Posts: 8
|
The problem has been solved. It was a priority issue. I wanted to share with everyone who helped the results of our investigation.
We had checked all of the MQ queues and verified that they had a default priority of zero. We verified that all of our applications, which fed the queues, were setup to use the priority as provided by the queues. All of this was as expected. They all sent messages with a priority of zero.
All systems, but the two systems in question, sent there messages to one queue. A transformation program would get messages from this queue, transform the messages then put the messages on the queue which is the same queue that the two systems in question were putting their messages. The two systems in question were defauting to the queue priority of zero. The problem was that the transformation app was written in JMS and the default priority for JMS is 4, so all transformed messages were receiving a priority of 4.
Thank you all very much for your help! |
|
| Back to top |
|
|
|
|
