| Author |
Message
|
| Sam Uppu |
 Posted: Thu Jun 11, 2009 10:14 am Post subject: MQ error 2101: OBJECT_DAMAGED |
 |
|
Yatiri
Joined: 11 Nov 2008
Posts: 610
|
Hi Guys,
we are MQ version 7.0.0.1 on Solaris 10.
On 6/10 the application(java) identified an MQ error that started on 6/1. The application was getting a 2101 (MQRC_OBJECT_DAMAGED). I saw a 2009 (MQRC_CONNECTION_BROKEN) in the QMgr logs, and when I did a display of queues, the queue did not show up. The solution was to bounce the queue manager. We are using circular logging.
I have searched many of the links but not really found a solution. I am wondering why an object goes Corrupt?. What are the reasons?.
Any insight would be helpful.
Thanks. |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Thu Jun 11, 2009 12:28 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
The only way to recover a damaged object is to do media recovery. This of course means you need to run with linear logging.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Jun 11, 2009 1:40 pm Post subject: Re: MQ error 2101: OBJECT_DAMAGED |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| Sam Uppu wrote: |
| I am wondering why an object goes Corrupt?. What are the reasons? |
Someone / something deleted the q file.
Someone / something modified the q file.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Jun 11, 2009 1:53 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9470
Location: US: west coast, almost. Otherwise, enroute.
|
I'd suspect someone, not something. At least that's been my experience.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Sam Uppu |
| Posted: Thu Jun 11, 2009 2:00 pm Post subject: |
|
|
Yatiri
Joined: 11 Nov 2008
Posts: 610
|
|
| Back to top |
|
|
| Sam Uppu |
| Posted: Thu Jun 11, 2009 2:03 pm Post subject: |
|
|
Yatiri
Joined: 11 Nov 2008
Posts: 610
|
As far as I know nobody changed the MQ objects on this server for this Qmgr. Nobody changed the queues or queue file under /var/mqm/qmgrs/<qmgrname>/queues/<queuename>/q
Your experiences will help me a lot on this.
Thanks. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Jun 11, 2009 4:09 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
That's a link to Handling messages greater than 4MB long.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Jun 11, 2009 4:15 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| Sam Uppu wrote: |
As far as I know nobody changed the MQ objects on this server for this Qmgr. Nobody changed the queues or queue file under /var/mqm/qmgrs/<qmgrname>/queues/<queuename>/q
|
How can you be 100% sure?
A few months ago there were "SAN problems" here. Some other MSCS clusters would wig out, but thankfully my MQ ones stayed up. This went on for a few days, impacting these other application clusters. During that timeframe, we got 2 damaged queues, the first time in years. Apparently there was some corruption of the q files. Never knew exactly what happened. But if I got a damaged q, and a corrupted q file can cause that, and the underlying storage that holds the q file was having problems....
| PeterPotkay wrote: |
Someone / something deleted the q file.
Someone / something modified the q file. |
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| Sam Uppu |
| Posted: Thu Jun 11, 2009 7:31 pm Post subject: |
|
|
Yatiri
Joined: 11 Nov 2008
Posts: 610
|
Peter,
I dont have the answer when you ask whether I am 100% sure. I may be wrong and something got changed. I may agree with you.
But the app team may ask me that are you 100% sure that something got changed?. If so, how can you prove it?. Then I dont have any evidence to support my statement. Thatswhat I am asking/ looking for. Is there any solid way of saying in which cases the MQ object may go corrupt/ damage and can we find the root cause for this damage and how to avoid this in the future not to repeat the same failure/ damage?.
I am looking for some kind of information which I can pass it to the app team saying this is what has happened and which caused the object to damage. IBM documentation is talking about the solution, how to recover the damaged object but not telling about in which circumstances the object will be corrupted and how to find out the root cause(what caused the issue)?.
Thanks. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Jun 11, 2009 7:52 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
May be I should have said reliably recover a damaged object. Seems to me you were just so lucky that the logs still held the object from point of damage on. Once the circular log has been overwritten the only possibility for you is to delete and recreate the damaged object...
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Sam Uppu |
| Posted: Wed Jun 24, 2009 6:48 am Post subject: |
|
|
Yatiri
Joined: 11 Nov 2008
Posts: 610
|
The application is inquiring the current depth of this queue. Do you think that is causing the object go damage?.
In the ffstsummary, I see the below output:
AMQ8763.0.FDC 2009/06/22 12:49:08.958124-5 amqzlaa0_nd 8763 241 XC130004 xehExceptionHandler STOP OK
AMQ8763.0.FDC 2009/06/22 12:49:09.444613-5 amqzlaa0_nd 8763 241 AQ051000 aqsStartQOp STOP_ALL OK
In the QMgr logs, I see the below error repeatedly:
06/22/09 12:53:54 - Process(13425.2 User(mqm) Program(amqrmppa)
AMQ9544: Messages not put to destination queue.
EXPLANATION:
During the processing of channel 'TO.QM2' one or more messages could not
be put to the destination queue and attempts were made to put them to a
dead-letter queue. The location of the queue is 2, where 1 is the local
dead-letter queue and 2 is the remote dead-letter queue.
ACTION:
Examine the contents of the dead-letter queue. Each message is contained in a
structure that describes why the message was put to the queue, and to where it
was originally addressed. Also look at previous error messages to see if the
attempt to put messages to a dead-letter queue failed. The program identifier
(PID) of the processing program was '13425'.
Nobody touched this server and nothing changed from MQ side.
This is the third time in a row since 3 weeks we encouter this error(2101).
Can you guys throw some light on this?.
Thanks. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jun 24, 2009 7:15 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Sam Uppu wrote: |
| The application is inquiring the current depth of this queue. Do you think that is causing the object go damage?. |
Even if it isn't, it's not a good thing for an application to be doing.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Jun 24, 2009 7:36 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9470
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| The application is inquiring the current depth of this queue. |
Why is it inquiring the depth of the queue?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Sam Uppu |
| Posted: Wed Jun 24, 2009 10:32 am Post subject: |
|
|
Yatiri
Joined: 11 Nov 2008
Posts: 610
|
| bruce2359 wrote: |
| Quote: |
| The application is inquiring the current depth of this queue. |
Why is it inquiring the depth of the queue? |
The application which is pushing the msgs onto this queue is a batch application and it will send the messages when the batch application is started. The pulling application is not that fast enough to process all the messages which are coming onto this queue within a given time. So the sending app is inquiring the current depth of this queue not to overflow the msgs and land onto DLQ. When the queue is full, the sending app will stop sending the msgs onto this queue.
Thanks. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jun 24, 2009 10:39 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| Sam Uppu wrote: |
| bruce2359 wrote: |
| Quote: |
| The application is inquiring the current depth of this queue. |
Why is it inquiring the depth of the queue? |
The application which is pushing the msgs onto this queue is a batch application and it will send the messages when the batch application is started. The pulling application is not that fast enough to process all the messages which are coming onto this queue within a given time. So the sending app is inquiring the current depth of this queue not to overflow the msgs and land onto DLQ. When the queue is full, the sending app will stop sending the msgs onto this queue.
Thanks. |
Bad design. The messages will not go to the DLQ. The app will get a q full error code on the MQPUT that is over the limit, and then the app can stop. Inquiring on the depth constantly is a waste of coding effort and CPU.
Anyway, no, IBM did not design the MQINQ call such that is shreds a queue and makes the q useless.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
|
|
