| Author |
Message
|
| Bnikpour |
 Posted: Fri Jun 07, 2013 9:56 am Post subject: MQ queue unable to GET after messages delivered |
 |
|
Novice
Joined: 19 Apr 2011
Posts: 20
Location: Phoenix, AZ
|
Issue:
We are running Websphere MQ Version 7.0.1.5. When messages are put to the Request queue, one of the messages prevents the consuming application from being able to consume it and any subsequent messages. As a result messages pile up in the request queue. We are unable to save off any of the messages because our tools are also unable to perform a GET on the queue and all the messages have to be cleared from the queue in order to restore it to normal operation resulting in a
subsquent loss of data.
All other queues on the QM behave normally
Symptoms:
Error code 2009 is shown when trying to GET from the queue using any tools (M071,MQ Explorer)
All handles on the queue from the consuming application die
WPS is able to put new messages onto the queue and they keep piling up in the request queue. Messages are in XML format.
None of the messages that cause this error go to the Error queue or the hold queue defined for the request queue . We are sure that the Error queue works as designed as other illformated messages do occasionally end up in it.
What we have tried:
We originally thought the issue maybe related to message size due to APAR
IZ63971: CLIENT APPLICATION RECEIVES 2009 RETURN CODE WHEN TRYING TO GET
LARGE MESSAGE FROM QUEUE: FDC WITH PROBE ID RM496003 OR RM496002
We tired putting a large 42kb message through our performance environment and it was picked up successfully by the consuming application. This is not the issue.
Any ideas from the experts on the forums?
_________________
Addicted to E-mail, not by choice.
MQ & WAS Admin |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Fri Jun 07, 2013 10:20 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
What is your poison message handling ?
Are you retrieving the message with properties or RFH2?
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Bnikpour |
| Posted: Fri Jun 07, 2013 10:40 am Post subject: |
|
|
Novice
Joined: 19 Apr 2011
Posts: 20
Location: Phoenix, AZ
|
Message is retrieved with RFH2
Consuming application is running on WAS using JMS
Poison messages should be rerouted to a Error queue on the same queue manager, and the normal function of both the consuming application and the QM is to route those messages back into a error queue. We also tested this and it works as most poison messages do in fact end up in this error queue.
Totally perplexed as to why in this one specific case that is not happening. 
_________________
Addicted to E-mail, not by choice.
MQ & WAS Admin |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jun 07, 2013 12:13 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| if amqsbcg doesn't show the message, then it's not committed or the q storage file is corrupt somehow... |
|
| Back to top |
|
|
| Bnikpour |
| Posted: Fri Aug 23, 2013 9:11 am Post subject: |
|
|
Novice
Joined: 19 Apr 2011
Posts: 20
Location: Phoenix, AZ
|
Hi everyone,
Just wanted to update this thread on the root cause for the issue in case anyone in the future has it again.
Root Cause
The root cause was that the messages were being manually edited and resubmitted to the queue using a 8 year old version of RFHutil-C. This version incorrectly strips some of the header information from messages containing special characters which leaves the header size too small for a RFH2 header.
The poison message is ,however, able to successfully enter the queue and a GET is performed by the consuming application. However, the poison message was then unable to be committed by the consuming application which left the queue unable to GET.
Once we asked the developers to update their RFHutil-C to the newest version the problem went away.
Moral of the story:
Make sure your application developers are using the newest version of SupportPacs
 
_________________
Addicted to E-mail, not by choice.
MQ & WAS Admin |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Aug 23, 2013 9:30 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Thanks for letting us know the reason for the problem and the fix.
You would not be surprised how reluctant many sites are to upgrade or to even try newer versions of tools.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Gerd-in-ZA |
| Posted: Thu Aug 29, 2013 3:07 am Post subject: |
|
|
Novice
Joined: 13 Sep 2006
Posts: 14
Location: Johannesburg, South Africa
|
Another popular cause for this kind of behavior would be a log overrun - smallish circular logs (e.g. the size offered as default by MQ Explorer) - one or more applications write enough data to the log so that the next bit to be written would overwrite the first bit that is still needed. MQ will then back out transactions to free up log space. If I remember correctly there are about three or so reason codes that can come up in such a situation and 2009 should be one of those.
Please note that - as luck would have it - sometimes you can't identify a single offending application and sometimes you can. Occasionally it helps to remind a programmer that he wanted to include MQCMIT calls in his program ...
In any event, this is also a perfect opportunity to remind people that the default log sizes are WAY TOO SMALL for normal production use. And you can up the number of log files during a QM restart, but to change the log file size requires a complete redefinition of the QM - better give it a good mouthful right up front ...
_________________
-- Gerd -- |
|
| Back to top |
|
|
|
|
