| Author |
Message
|
| manojsu |
 Posted: Wed Jul 25, 2007 1:56 am Post subject: MQ Object got corrupted |
 |
|
Centurion
Joined: 19 Jul 2006
Posts: 147
Location: Bangalore
|
Hi All,
Yesterday I faced a weird problem, my WMQ V6 on AIX got currupted on which on broker and the ConfigMgr were running on in the production environment.
It was repeatedly thrashing FDC files and also within the AMQ Error logs it was repeatedly saying that it is getting shut down due to some temporary object AMQ.***** and then the WMQ got shut down.
in the logs it was told to delete the object, and then restart but i had no clue where this object existed.
We were able to start the queue Manager, but after some time it throws a "AMQ.****" wid in the logs and crashes.
Since it was production we did a save qmgr and recreated the queue manager using the existing objects and were able to recover.
But till now we are bto able to analyse as to why the Queue Manager got corrupted, and in that case is there any better way of recover other than the recreate option
Many Thanks
Regards
S.Manoj |
|
| Back to top |
|
 |
| Vitor |
| Posted: Wed Jul 25, 2007 2:08 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
What are the various levels of software involved? Are you patched to date?
If the corrupt object is a temp queue (AMQ*) then delete / redefine is the only realstic means of recovery (because whatever's using the queue will need a note of the name!).
But why they're getting corrupted needs investigation. Shouldn't happen (and doesn't on our set up)
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jul 25, 2007 2:30 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Vitor wrote: |
What are the various levels of software involved? Are you patched to date?
If the corrupt object is a temp queue (AMQ*) then delete / redefine is the only realstic means of recovery (because whatever's using the queue will need a note of the name!).
But why they're getting corrupted needs investigation. Shouldn't happen (and doesn't on our set up) |
If what is getting corrupted is a temp queue you should make sure that the underlying model queue is not corrupted. The best protection against this kind of error is linear logging. From time to time we get an OS alert about logspace filling up. The cause is a corrupted MQ object (level 6.0.2.0).
All we need to do is bounce the qmgr, make sure rdcmqimg works fine (no more corrupted objects), run the linear log archiving manually (MS0L) or wait for its next scheduled run, and we are back in business... No operations interruption in a clustered environment...
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| manojsu |
| Posted: Wed Jul 25, 2007 2:56 am Post subject: |
|
|
Centurion
Joined: 19 Jul 2006
Posts: 147
Location: Bangalore
|
The version is :
$ dspmqver
Name: WebSphere MQ
Version: 6.0.0.0
CMVC level: p000-L050519
BuildType: IKAP - (Production)
Errors:
24/07/07 11:57:09 - Process(602190.20) User(esbprd) Program(amqzlaa0_nd)
AMQ6184: An internal WebSphere MQ error has occurred on queue manager PBKRNG01.
EXPLANATION:
An error has been detected, and the WebSphere MQ error recording routine has been called. The failing process is process 602190.
ACTION:
Use the standard facilities supplied with your system to record the problem
identifier, and to save the generated output files. Contact your IBM support center. Do not discard these files until the problem has been resolved.
I have no clue as how to find the CSD installed, please do let me know.
Also i was not able to locate where the temporory queue existed in the file system and also was not able to see the same too. And hence had to do a recreate of the same.
And yeah i did try bouncing the queue manager a couple of yime before i could recreate the queue manager (as i dint wanna loose the data)
I have been looking at the log files for couple of hours, but till now have no clue as what caused the problem.
My guesses are : 1> The Model Queue getting corrupt.
2> Bad Block in the disk space.
Would be helpful if can shed some light on how can i go on my investigation on the same and get a concrete answer. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jul 25, 2007 3:02 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| manojsu wrote: |
The version is :
$ dspmqver
Name: WebSphere MQ
Version: 6.0.0.0
|
So you have no maintenance applied (if you did the numbers would not all be zero!). I'd recommend you consider applying at least Fix Pack 1 if not fixing up to date
| manojsu wrote: |
I have been looking at the log files for couple of hours, but till now have no clue as what caused the problem.
My guesses are : 1> The Model Queue getting corrupt.
2> Bad Block in the disk space.
Would be helpful if can shed some light on how can i go on my investigation on the same and get a concrete answer. |
I found the comments of fjb_saper very interesting; I'd not considered a problem with the model queue but it does sound likely doesn't it? I'd go with his suggestions as a first move.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| manojsu |
| Posted: Wed Jul 25, 2007 4:46 am Post subject: |
|
|
Centurion
Joined: 19 Jul 2006
Posts: 147
Location: Bangalore
|
yeah this is also one thingthat occurred into my ind, but as i told earlier I was not able to locate the temporary queue and the base model queue that it was referring to, and as per the words of saper, i have never used of rdcmqimg, and have no clue of how to use it.
And i have tried bouncing the queue manager, but it was of now help |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jul 25, 2007 4:56 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| manojsu wrote: |
| yeah this is also one thingthat occurred into my ind, but as i told earlier I was not able to locate the temporary queue and the base model queue that it was referring to, and as per the words of saper, i have never used of rdcmqimg, and have no clue of how to use it. |
The temporary queue probably went as part of the error, if not when the queue manager was bounced. I'd hazard a guess that if it's a broker queue it's being created from the system object; you may be able to resolve the problem (temporarially at least) by getting the queue manager to recreate them
The rcdmqimg command is in the System Administration guide, and only applies if you're using linear logging.
Do you plan to apply maintenance as part of a longer term fix strategy?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| manojsu |
| Posted: Wed Jul 25, 2007 6:06 am Post subject: |
|
|
Centurion
Joined: 19 Jul 2006
Posts: 147
Location: Bangalore
|
But it is not a Broker Queue, as during that point of time my broker flows, broker and configmgr are stopped.
Yeah, I have proposed for upgarding to the latest fix pack/csd. 
but its a take my manager has to do.. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jul 25, 2007 2:35 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Keep in mind that bouncing a qmgr with linear logs has the added advantage (at least in 6.0) that it automatically tries to recover (as in rcvrmqobj) any damaged object from Media....
Running rcdmqimg just shows me that it is now running fine as the program abended when trying to run before the qmgr shutdown == hence my assumption of having fixed the dammaged object...
Oh, and by the way while the qmgr was down I also cleared the memory (amqiclen)...
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| manojsu |
| Posted: Thu Jul 26, 2007 11:46 pm Post subject: |
|
|
Centurion
Joined: 19 Jul 2006
Posts: 147
Location: Bangalore
|
Yeah, I guess i should have tried cleaning the memory, and will remember this the next time, but here since i have never used this, will this clear any of my transactional cache or somethin as i guess this will clear up the memory used by WMQ.
Many thanks for your suggestions |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Jul 27, 2007 12:55 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| manojsu wrote: |
Yeah, I guess i should have tried cleaning the memory, and will remember this the next time, but here since i have never used this, will this clear any of my transactional cache or somethin as i guess this will clear up the memory used by WMQ.
Many thanks for your suggestions |
What is the memory used by MQ when the qmgr is down? If you have multiple qmgrs only clear the memory for the qmgr involved (-xvhm)...
The memory is being held while the qmgr is down to provide for a faster start up and avoid releasing the memory (so it cannot be used by a different process and lead to scarce resources for MQ when you bounce the qmgr??)
I hope this clears up some of your questions...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
