| Author |
Message
|
| GaryGWood |
 Posted: Thu Nov 06, 2003 7:21 am Post subject: WMQI 2.1 Troubleshooting |
 |
|
Apprentice
Joined: 14 Oct 2003
Posts: 48
Location: Austin, TX
|
I inherited a WMQI 2.1 environment and am trying to come up to speed. It seems that troubleshooting WMQI is a bit of an art I have yet to aquire.
I have a relatively straightforward message flow that does the following (each line is a node in the message flow):
Pick up messages from an input queue
Add the RFH2 header
Perform a database lookup via ESQL and create a new message for each record in the result set (may be several hundred records)
Publish the messages to subscriptions
I get a list of error messages which are listed below in chronological order.
Message 1
( QM1.default ) Unable to write message to queue 'SYSTEM.DEAD.LETTER.QUEUE' on queue manager 'QM1'. MQPUT failed: MQCC=2; MQRC=2003; message flow node 'MessageFlow.Message Out.MQOutput'.
The message flow node 'MessageFlow.Message Out.MQOutput' was unable to write an MQSeries message as it failed to put it to the queue 'SYSTEM.DEAD.LETTER.QUEUE' belonging to queue manager 'QM1'. The completion and reason code from the MQPUT operation are as displayed.
Message 2
( QM1.default ) Unable to write message to queue 'SYSTEM.DEAD.LETTER.QUEUE' on queue manager 'QM1'. MQPUT failed: MQCC=2; MQRC=2003; message flow node 'MessageFlow.Message Out.MQOutput'.
The message flow node 'MessageFlow.Message Out.MQOutput' was unable to write an MQSeries message as it failed to put it to the queue 'SYSTEM.DEAD.LETTER.QUEUE' belonging to queue manager 'QM1'. The completion and reason code from the MQPUT operation are as displayed.
Message 3
( QM1.default ) Exception condition detected on input node 'MessageFlow.Message In'.
The input node 'MessageFlow.Message In' detected an error whilst processing a message. The message flow has been rolled-back and, if the message was being processed in a unit of work, it will remain on the input queue to be processed again. Following messages will indicate the cause of this exception.
Message 4
( QM1.default ) Error detected whilst processing a message in node 'MessageFlow.Extract Data from database'.
The message broker detected an error whilst processing a message in node 'MessageFlow.Extract Data from Pwdreg'. An exception has been thrown to cut short the processing of the message.
See the following messages for details of the error.
Message 5
( QM1.default ) Error detected whilst processing a message in node 'MessageFlow.Message Out.MQOutput'.
The message broker detected an error whilst processing a message in node 'MessageFlow.Message Out.MQOutput'. An exception has been thrown to cut short the processing of the message.
See the following messages for details of the error.
Message 6
( QM1.default ) Exception during destination list processing; output node 'MessageFlow.Message Out.MQOutput'.
While processing a destination list in the output node 'MessageFlow.Message Out.MQOutput' an error (or errors) occurred.
Message 7
( QM1.default ) Unable to write message to queue 'SUB_Q' on queue manager 'QM1'. MQPUT failed: MQCC=2; MQRC=2003; message flow node 'MessageFlow.Message Out.MQOutput'.
The message flow node 'MessageFlow.Message Out.MQOutput' was unable to write an MQSeries message as it failed to put it to the queue 'SUB_Q' belonging to queue manager 'QM1'. The completion and reason code from the MQPUT operation are as displayed.
Message 8
( QM1.default ) Unable to write message to queue 'ORDER_SUB_Q' on queue manager 'QM1'. MQPUT failed: MQCC=2; MQRC=2003; message flow node 'MessageFlow.Message Out.MQOutput'.
The message flow node 'MessageFlow.Message Out.MQOutput' was unable to write an MQSeries message as it failed to put it to the queue 'ORDER_SUB_Q' belonging to queue manager 'QM1'. The completion and reason code from the MQPUT operation are as displayed.
Message 9
( QM1.default ) Unable to write message to queue 'SYSTEM.DEAD.LETTER.QUEUE' on queue manager 'QM1'. MQPUT failed: MQCC=2; MQRC=2003; message flow node 'MessageFlow.Message Out.MQOutput'.
The message flow node 'MessageFlow.Message Out.MQOutput' was unable to write an MQSeries message as it failed to put it to the queue 'SYSTEM.DEAD.LETTER.QUEUE' belonging to queue manager 'QM1'. The completion and reason code from the MQPUT operation are as displayed.
It appears the messages are saying, "Something went wrong" and that's about it. I can watch the subscription queues start to get messages in them so I believe most of this is working but can't pinpoint where the problem is occuring. What/where should I check to resolve this? |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Thu Nov 06, 2003 7:49 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
It seems you have two problems.
1) Your WMQI system can not write to the SYSTEM.DEAD.LETTER.QUEUE, for reason 2003.
2) Something is going wrong in the 'Extract Data from database' node.
Reason code 2003 is 'MQRC_BACKED_OUT'. This isn't very helpful in telling me why the message could not be put onto SYSTEM.DEAD.LETTER.QUEUE. All it really tells me is that the message was backed out of the transaction that was putting it there.
In order to find out what went wrong with the 'Extract Data from database' node, you probably want to enable debug or user tracing. This is done using the mqsichangetrace command. The command itself is documented in the Administration guide, but there are examples in the Websphere MQ Integrator [Broker] Problem Determination Guide. This is available on the IBM documentation website.
Or you could look at your database and see if it 's logging any errors that will tell you what's going on.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| TonyD |
| Posted: Thu Nov 06, 2003 4:37 pm Post subject: |
|
|
Knight
Joined: 15 May 2001
Posts: 540
Location: New Zealand
|
| The 2003 rc that you are getting for your application queue and also the DLQ suggests that your log may be full. Any possibility that your flow is looping? |
|
| Back to top |
|
|
| GaryGWood |
| Posted: Fri Nov 07, 2003 8:58 am Post subject: |
|
|
Apprentice
Joined: 14 Oct 2003
Posts: 48
Location: Austin, TX
|
Thanks for the suggestions!
Jeff - I am running traces today and will check with the DB admins and see if they have anything going.
Tony - the flow is pretty straightforward although some record sets returned may be a few thousand records (certainly less than 10,000). Would that be enough to fill the log? Otherwise there is no looping or any place I can see that would allow things to "pile up". |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri Nov 07, 2003 10:49 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Tony may have had a better call than I did as to what the problem is.
I missed that the 2003 was happening on the Output node as well.
The logs he's talking about are your MQSeries log files, which are specified when you crtmqm and specify the size and number of primary and secondary log files.
It could be that you are running out of space to hold the entire transaction that you're performing in the logs. You can increase the number of primary and secondary logs without recreating the queue manager, but you can't increase the size of the individual log files without dropping and recreating the queue manager.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| GaryGWood |
| Posted: Mon Nov 10, 2003 11:45 am Post subject: |
|
|
Apprentice
Joined: 14 Oct 2003
Posts: 48
Location: Austin, TX
|
Thanks Jeff, appreciate your input. I recreated the QM with larger and more log files and am still getting the error. So far the DB admins have told me they see nothing unusual but I think they mey be kind of blowing me off so I'm giong to go get tough this afternoon 
We are opening a support incident with IBM also and see if we can get this resolved. I'll post the outcome when that happens. If anything else occurs to anybody, rest assured that I am interested and appreciative.
Thanks!
Gary |
|
| Back to top |
|
|
| TonyD |
| Posted: Mon Nov 10, 2003 1:40 pm Post subject: |
|
|
Knight
Joined: 15 May 2001
Posts: 540
Location: New Zealand
|
| Commit failure for some other reason than log space would therefore appear to be the case, as you have suggested to your DBAs?! ...as per Jeff's comment re the 'Extract Data' node. Are there any other likely messages in the MQ error logs? ...or the database log, e.g. db2diag (if DB2) or equivalent. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Mon Nov 10, 2003 3:07 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
The other thing I wonder is if the message size is too big for both the output queue and the DLQ, or perhaps for the Queue Manager. The 2003 is likely masking the real reason code.
I'm guessing that Gary didn't see anything useful in the logs from mqsichangetrace?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| GaryGWood |
| Posted: Wed Nov 12, 2003 7:32 am Post subject: |
|
|
Apprentice
Joined: 14 Oct 2003
Posts: 48
Location: Austin, TX
|
I think I have it resolved - still testing but it looks right!
Several things were at work here:
This is on a prototype box so some defaults were in place, namely:
log file size was at 256
log primary files were at 3
maximum uncommitted messages were at 1000
circular logging was used
Also, the SQL (delivered by the DB admin guys and modified to ESQL by me) was supposed to return less than 1000 messages but for some reason is returning 30,000. Thankfully, DB developers will resolve that one.
With the above settings, the large result set and the messages being put on the queue as persistant, I was simply running out of log file space with the circular log.
I changed to linear logging, upped the sizes on everything and modified the ESQL (don't tell the DB developers  ) and now it all works as expected.
Jeff, Tony - thanks for the input, your ideas are what lead me down the right path to resolve this! |
|
| Back to top |
|
|
|
|
