| Author |
Message
|
| DuaneEBenson |
 Posted: Thu Jul 05, 2007 11:08 am Post subject: MQ "stuck" after a rollback in .net application |
 |
|
Newbie
Joined: 05 Jul 2007
Posts: 5
|
I'm a queue newbie with a problem... no, please... keep reading! 
I've been searching and reading for the past several days, but haven't found an answer yet, so I'm posting here to see if anyone can point me in the right direction.
First, my environment is like this:
5.3 queue is running on AIX (no app server, just a queue)
I am accessing the queue remotely from a .net app, reading JMS messages, using XMS and the 6.0 MQ Client.
My .net application is successfully reading messages from the queue most of the time. This app needs to guarantee that it has "taken responsibility" for processing the contents of the message before it removes the message from the queue, so I am following this sequence:
1. create a (remote?) connection to the queue (i.e. I don't use a local queue)...
2. do a connection.start()...
3. Create a session using autoacknowledge...
4. Create a consumer with a message selector...
5. Call consumer.ReceiveNoWait() to get the message...
6. store the message in my own database to ensure I can process it...
7. either commit or rollback the message on the queue, depending on the logic applied in step #6.
All of this works... most of the time... and this loop may execute every 1 minute at times.
My problem is this... approximately once or twice per week... the application will go into a cycle where it is constantly reading/rolling back the same message for a period of time (let's debate the appropriateness of this later).... sometimes after the rollback, the XMS layer gives me an exception, stating reason code 2036 (queue not open for browse). Then, my application cannot read anything from the queue unless I remove the message selector. Reason code 2036 says to specify MQOO_BROWSE, but I don't see that this is applicable when using XMS from .NET.
Another clue may be this... after I get the queue stuck... I need to do a get from the queue without using a message selector... this get operation tells me there was no message on the queue... but it clears the stuck condition.
The error message is below, the first two messages are mine, the last two are XMS exceptions.
Anyone have any ideas??? Any pointers would be very much appreciated.
2007-07-04 12:29:23.308 201 AppWorker.Worker Start- error moving message from queue to database rolling back
DatabaseManager.Exceptions.OracleNoResultException: No unused credentials available for the specified legacy type id = 2
at DatabaseManager.DBLayer.LockAndConsumeFirstAvailableLegacyCredential(Int32 legacyTypeId, Int32& legacyUserId, String& username, String& password)
at AppWorker.Worker.readNextMessage(MYContext myContext)
2007-07-04 12:29:23.308 201 AppWorker.Worker Start- No unused credentials available for the specified legacy type id = 2
DatabaseManager.Exceptions.OracleNoResultException: No unused credentials available for the specified legacy type id = 2
at DatabaseManager.DBLayer.LockAndConsumeFirstAvailableLegacyCredential(Int32 legacyTypeId, Int32& legacyUserId, String& username, String& password)
at AppWorker.Worker.readNextMessage(MYContext myContext)
at AppWorker.Worker.Start()
2007-07-04 12:29:28.605 (null) (null) ERROR AppWorker.Worker Start- error moving message from queue to database rolling back
IBM.XMS.IllegalStateException: CWSMQ0006E: An exception was received during the call to the method {1}: {0}. During execution of the specified method an exception was thrown by another component. See the linked exception for more information.
at IBM.XMS.WMQ.WmqMessageConsumer.GetAMessage(Int64 timeOut)
at IBM.XMS.WMQ.WmqMessageConsumer.ReceiveInternal(Int64 timeout)
at IBM.XMS.WMQ.WmqMessageConsumer.ReceiveNoWait()
at AppMQManager.MessageConsumer.Consume(String selector, String& messageContent)
at AppWorker.Worker.readNextMessage(MYContext myContext)
Linked Exception : CompCode: 2, Reason: 2036
2007-07-04 12:29:28.636 (null) (null) ERROR AppWorker.Worker Start- CWSMQ0006E: An exception was received during the call to the method {1}: {0}. During execution of the specified method an exception was thrown by another component. See the linked exception for more information.
IBM.XMS.IllegalStateException: CWSMQ0006E: An exception was received during the call to the method {1}: {0}. During execution of the specified method an exception was thrown by another component. See the linked exception for more information.
at IBM.XMS.WMQ.WmqMessageConsumer.GetAMessage(Int64 timeOut)
at IBM.XMS.WMQ.WmqMessageConsumer.ReceiveInternal(Int64 timeout)
at IBM.XMS.WMQ.WmqMessageConsumer.ReceiveNoWait()
at MQManager.MessageConsumer.Consume(String selector, String& messageContent)
at AppWorker.Worker.readNextMessage(MYContext myContext)
at AppWorker.Worker.Start()
Linked Exception : CompCode: 2, Reason: 2036 |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Thu Jul 05, 2007 11:13 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Well, you should verify that it's not an issue with your MQ connection somehow - by looking at the MQ logs in the MQ client install and on the server install as well (or at least ask the MQ Admin to confirm that there are no errors showing up at around the time you're seeing a problem).
But it kind of looks like an issue with your database connection... Perhaps the database is throwing an exception, and your code is trying to pick up after that but in the wrong place for how the MQ code is structured...
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| DuaneEBenson |
| Posted: Thu Jul 05, 2007 11:22 am Post subject: |
|
|
Newbie
Joined: 05 Jul 2007
Posts: 5
|
I should clarify.. the database exceptions that you see in "my" messages are not actual problems with the database... I throw these to indicate a condition in my busines logic that triggers the rollback of the message on the queue.
I don't have admin access on the queue, but I have already asked the mq admin to look at the logs... still waiting on that, it's a painful process.
I thought about looking at messages logged by the mq client on my machine, but I don't see any. Is there a way to enable logging on my local mq client?
Thanks!
Duane |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Jul 05, 2007 11:28 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
There are only certain kinds of things that would cause errors to show up in the AMQERR* logs on the client side.
It may not be significantly meaningful that you don't see anything.
There isn't, that I know of, any particular way to change what causes client side logging to occur.
You can, however, turn on client side TRACING. Look for instructions on tracing in the system admin guide.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jul 05, 2007 12:18 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
What level of CSD is the queue manager at? If you're using XMS, you should at least be at v5.3.10
Even if you're not using XMS, it's good practice to be patched to date
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| DuaneEBenson |
| Posted: Fri Jul 06, 2007 6:02 am Post subject: |
|
|
Newbie
Joined: 05 Jul 2007
Posts: 5
|
The xms tracing has helped a great deal... I can see a lot more detail about what is happening...
I have yet to fully interpret the whole series of events, but it looks like xms is failing to move messages to the dead letter queue.
What is still confusing me is that the message doesn't seem to go to the dead letter queue after a consistent number of reads... sometimes it happens after 10 or so read/rollbacks, and other times it may read/rollback 100 times or more before a problem occurs. It seems to be more closely related to how quickly we read/rollback, rather than how many times.
I'm still waiting to hear back on patch levels of my clients mq server.
I'm making some progress on this issue, but if anyone has any hints to speed up the process feel free to add your 2 cents!
Thanks!
Duane |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri Jul 06, 2007 6:06 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Ahhh haaa....
Sounds like you're running out of log space.
I.e. you're making your transactions too big (accidentally by forgetting to commit correctly, or "on purpose" by forgetting to size the logs to meet the demand).
These would be the transaction logs on the qmgr.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| DuaneEBenson |
| Posted: Tue Jul 10, 2007 6:44 am Post subject: |
|
|
Newbie
Joined: 05 Jul 2007
Posts: 5
|
I'm still churning on this problem...
Still waiting to find out the version number of MQ, but I am using the 6.0 client.
The log files are circular... so they will overwrite as needed... not sure what else to look for on log related problems.
The XMS trace log has revealed that the backout threshold has been reached, and it is attempting to move the message to the system.dead.letter.queue, but it fails in attempting that move, with the following error messages:
GetAMessage() Backout Threshold Reached
BackOutRequeue()Trying to backout to system.dead.letter.queue
BackOutRequeue()Unable to requeue message CC=2 Reason Code=2097
GetAMessage()Message requeue failed-treating it as a bad message
DeadLetter()Unable to commit requeued message as session is transacted or client ack
GetAMessage()-Get call failed for MD Version2 w/ CC=2 RC=2036
So... my queue manager had no backout queue defined and the default value of zero for the backout threshold... I tried configuring these... set backout threshold to 500 and backout queue to a specifically created backout queue modeled after the system.dead.letter.queue...
the results... XMS ignored the new settings... and used the JMS default of 20 for the backout threshold... and is still attempting to use system.dead.letter.queue.
Can anybody tell me where I can define the backout threshold??? I understand that it is JMS's responsibility to respect this threshold... the queue manager ignores it completely... but I want XMS(JMS) to use a specific backout threshold and a specific backout queue.
Can anyone tell me where to configure these?
remember, I'm not using websphere application server, this is a direct client connection, using XMS.NET and the MQ Client 6.0. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jul 10, 2007 7:18 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| DuaneEBenson wrote: |
The log files are circular... so they will overwrite as needed... not sure what else to look for on log related problems.
|
Yes, but if you have too many uncommitted transactions to fit in the maximum number of log files you'll run out of space!
| DuaneEBenson wrote: |
So... my queue manager had no backout queue defined and the default value of zero for the backout threshold... I tried configuring these... set backout threshold to 500 and backout queue to a specifically created backout queue modeled after the system.dead.letter.queue...
|
Where did you set these on the queue manager? These are queue attributes not queue manager ones. Dead letter queue is at queue manager level, backout queue is not.
| DuaneEBenson wrote: |
Can anybody tell me where I can define the backout threshold??? ... but I want XMS(JMS) to use a specific backout threshold and a specific backout queue.
|
I though XMS was wrapped round the C API, which doesn't honour this setting? 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| DuaneEBenson |
| Posted: Tue Jul 10, 2007 9:41 am Post subject: |
|
|
Newbie
Joined: 05 Jul 2007
Posts: 5
|
ok, just tried increasing the count and size of the logs... didn't affect the outcome at all.
And yes, I did set the backout threshold and the backout queue on the queue... not the queue manager, sorry for the confusion.
XMS does have a property for JMSX_DELIVERY_COUNT, which I assume is the count maintained by the MQ to determine if we have reached the backout threshold... so in desparation, I'm considering watching this value in my application, and when it gets to 18 or so, I'll just commit the read of the message from the queue and put it back there myself, which will effectively "reset" the number of failed attempts. Admittedly, this is a bad solution... but I don't know what else to try. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Jul 10, 2007 4:01 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
You can only actually increase the count of the logs (LogPrimary and LogSecondary).
You can increase the size of the logs, it just doesn't, um, count if you do so.
You may also have to restart the qmgr after increasing the # of logs - I've forgotten.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jul 11, 2007 12:42 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| DuaneEBenson wrote: |
| I'm considering watching this value in my application, and when it gets to 18 or so, I'll just commit the read of the message from the queue and put it back there myself, which will effectively "reset" the number of failed attempts. Admittedly, this is a bad solution... but I don't know what else to try. |
For what it's worth, it's how I'd code a traditional C API application. On read I'd check the backout count in the MQMD against the backout threshold of the queue, if it was higher I'd put the message unopened onto the backout queue associated with the queue, commit the entire unit of work and move on with my life. 
Hence my comment about XMS being wrapped round the C API not the JMS one, even if it contains JMS variable names for compatibility. I could be wrong of course.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
