| Author |
Message
|
| PGBuff |
 Posted: Mon Mar 17, 2014 6:15 am Post subject: What if Media Log is itself Damaged Or Untraceable ? |
 |
|
Novice
Joined: 08 Nov 2013
Posts: 11
|
Hi,
What if the Media Log itself has got a chances of being damaged Or untraceable ?
If I get that straight & more clear, my queue manager is up and running (cause after stop and start, qmgr was able to find the recovery log) BUT some channels/queues/processes are missing and those are not getting recovered because media log is not traceable by qmgr.
* What if media log is damaged, how can I get those objects back ?
* What if media log exists but Qmgr still not able to make anything out of it, then how can I get those objects back ?
I know if I take the media log from past (may be a day or two days ago) , there are always chances to get the persistent messages from them and cause the duplicate message feed to the connecting application. (Correct me if I'm wrong).
* What if I'm not even able to get the objs recovered from older media logs, then how can I get those objects back ?
I know there are too many what-if.. but these are cases going on !
Kindly suggest !!
 |
|
| Back to top |
|
 |
| Vitor |
| Posted: Mon Mar 17, 2014 6:24 am Post subject: Re: What if Media Log is itself Damaged Or Untraceable ? |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| PGBuff wrote: |
| If I get that straight & more clear, my queue manager is up and running (cause after stop and start, qmgr was able to find the recovery log) BUT some channels/queues/processes are missing and those are not getting recovered because media log is not traceable by qmgr. |
There's no difference between a media log and a recovery log. What exactly can the queue manager not trace?
| PGBuff wrote: |
| * What if media log is damaged, how can I get those objects back ? |
If the media log is damaged, the object should be recovered from the object file.
| PGBuff wrote: |
| * What if media log exists but Qmgr still not able to make anything out of it, then how can I get those objects back ? |
Raise a PMR - you're describing a product fault.
| PGBuff wrote: |
| I know if I take the media log from past (may be a day or two days ago) , there are always chances to get the persistent messages from them and cause the duplicate message feed to the connecting application. (Correct me if I'm wrong). |
You're wrong. The queue manager should resolve the persistent messages from the log and the queue object. Unless you're not manageing the linear logs properly.
| PGBuff wrote: |
| * What if I'm not even able to get the objs recovered from older media logs, then how can I get those objects back ? |
Raise a PMR
| PGBuff wrote: |
I know there are too many what-if.. but these are cases going on !
Kindly suggest !! |
I would look seriously and critically at your WMQ linear log management and disk management strategies. Media recovery is not as problematic and buggy as you're describing when done according to the product documentation.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| exerk |
| Posted: Mon Mar 17, 2014 6:28 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
Just to be sure - your queue manager does have linear logging?
If your media log is not 'traceable', that suggests that either you do not run the record image (rcdmqimg command) and that you manage redundant logs based on the restart log only, and have archived redundant logs. If you have not forced images into the logs, and archived all the logs you consider redundant but containing image information, you will need to restore them to the queue manager log directory and restart your queue manager again.
More to the point, what changes were made that caused you to restart your queue manager?
EDIT: 'quick-draw Vitor' beat me to the bloody keyboard - again!
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| PGBuff |
| Posted: Mon Mar 17, 2014 7:11 am Post subject: Re: What if Media Log is itself Damaged Or Untraceable ? |
|
|
Novice
Joined: 08 Nov 2013
Posts: 11
|
| Vitor wrote: |
| PGBuff wrote: |
| If I get that straight & more clear, my queue manager is up and running (cause after stop and start, qmgr was able to find the recovery log) BUT some channels/queues/processes are missing and those are not getting recovered because media log is not traceable by qmgr. |
There's no difference between a media log and a recovery log. What exactly can the queue manager not trace? |
I agree.. media and recovery are used synonymously.
I suspect Qmgr is not able to find the Media Log File itself though its there in the active logs AND qmstatus clearly display that Media Log file name.
| Vitor wrote: |
| PGBuff wrote: |
| * What if media log is damaged, how can I get those objects back ? |
If the media log is damaged, the object should be recovered from the object file. |
How ?  I mean is that same as recreate mq object command ?
| Vitor wrote: |
| PGBuff wrote: |
| * What if media log exists but Qmgr still not able to make anything out of it, then how can I get those objects back ? |
Raise a PMR - you're describing a product fault. |
Yea.. certainly.. I would do that.
| Vitor wrote: |
| PGBuff wrote: |
| I know if I take the media log from past (may be a day or two days ago) , there are always chances to get the persistent messages from them and cause the duplicate message feed to the connecting application. (Correct me if I'm wrong). |
You're wrong. The queue manager should resolve the persistent messages from the log and the queue object. Unless you're not manageing the linear logs properly. |
okey... how we decide that linear logs are not managed properly.. is that administration related to those mq log files that are getting used at the moment by qmgr AND which're not required by qmgr for restart & media recovery ? Or anything more ? 
| Vitor wrote: |
| PGBuff wrote: |
| * What if I'm not even able to get the objs recovered from older media logs, then how can I get those objects back ? |
Raise a PMR |
| Vitor wrote: |
| PGBuff wrote: |
I know there are too many what-if.. but these are cases going on !
Kindly suggest !! |
I would look seriously and critically at your WMQ linear log management and disk management strategies. Media recovery is not as problematic and buggy as you're describing when done according to the product documentation. |
(On the lighter note..).. Yea .. certainly I would love to handover the support of those qmgr servers to you. lol
but what criteria would you suggest for disk management in the highly active and critical system (ofcourse linear logs are there for qmgr).. any ratio between the primary and secondary logs ? |
|
| Back to top |
|
|
| PGBuff |
| Posted: Mon Mar 17, 2014 7:21 am Post subject: |
|
|
Novice
Joined: 08 Nov 2013
Posts: 11
|
| exerk wrote: |
| Just to be sure - your queue manager does have linear logging? |
yea.. it is there. aah.. I missed that in OP.  
| exerk wrote: |
| If your media log is not 'traceable', that suggests that either you do not run the record image (rcdmqimg command) and that you manage redundant logs based on the restart log only, and have archived redundant logs. If you have not forced images into the logs, and archived all the logs you consider redundant but containing image information, you will need to restore them to the queue manager log directory and restart your queue manager again. |
Yea.. I thought so.. Qmgr restart would be required here incase of old media image is taken. It is clear not all the logs were archived and it was not redundant.
| exerk wrote: |
| More to the point, what changes were made that caused you to restart your queue manager? |
Nothing on MQ end.. but there might be some external factors on the server beyond the control of MQ. Not determined yet. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Mar 17, 2014 7:41 am Post subject: Re: What if Media Log is itself Damaged Or Untraceable ? |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| PGBuff wrote: |
| I suspect Qmgr is not able to find the Media Log File itself though its there in the active logs AND qmstatus clearly display that Media Log file name. |
What entries in the queue manager log lead you to that conclusion? Better information, better advice.
| PGBuff wrote: |
| I mean is that same as recreate mq object command ? |
| PGBuff wrote: |
| okey... how we decide that linear logs are not managed properly.. is that administration related to those mq log files that are getting used at the moment by qmgr AND which're not required by qmgr for restart & media recovery ? Or anything more ? |
Nope - that's it. You need to ensure that recovery (media) images are taken correctly, required logs are not archived off and effectively what the InfoCenter says needs to be done is done. There's a support pac which I'm too lazy to look up with scripts to help with this.
| PGBuff wrote: |
| (On the lighter note..).. Yea .. certainly I would love to handover the support of those qmgr servers to you. lol |
You probably couldn't afford me, but if you'd like to suggest a number we can discuss it.....
| PGBuff wrote: |
| but what criteria would you suggest for disk management in the highly active and critical system (ofcourse linear logs are there for qmgr).. any ratio between the primary and secondary logs ? |
Be clear - I'm talking about disk & log management here, not sizing. How many primary / secondary logs you need is a factor of transaction size, volume, timing and a number of other things which are not connected to recovery. What you're describing has nothing to do with the size & numbre of your logs.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| exerk |
| Posted: Mon Mar 17, 2014 7:45 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| PGBuff wrote: |
| Yea.. I thought so.. Qmgr restart would be required here incase of old media image is taken. It is clear not all the logs were archived and it was not redundant. |
I don't understand what you mean by that statement...
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| PGBuff |
| Posted: Mon Mar 17, 2014 9:33 am Post subject: Re: What if Media Log is itself Damaged Or Untraceable ? |
|
|
Novice
Joined: 08 Nov 2013
Posts: 11
|
| Vitor wrote: |
| PGBuff wrote: |
| I suspect Qmgr is not able to find the Media Log File itself though its there in the active logs AND qmstatus clearly display that Media Log file name. |
What entries in the queue manager log lead you to that conclusion? Better information, better advice. |
entries in the queue manager *error* log leads to a clue i.e. to which media log file should be used by qmgr to recover from media loss/damage, the same file is in the 'qmstatus all'. but when recovery is attempted , it says it can not identify 'that' medial log file. |
|
| Back to top |
|
|
| JosephGramig |
| Posted: Mon Mar 17, 2014 9:57 am Post subject: |
|
|
Grand Master
Joined: 09 Feb 2006
Posts: 1244
Location: Gold Coast of Florida, USA
|
Sounds like your logs are damaged. Open a PMR and hope they can help you.
| Code: |
| rcdmqimg -m <QmgrName> -t all "*" |
Should save and move stuff to the current active logs. It will give you two files (might be the same one). All files older than the oldest of the two are no longer needed as they will never be used again. |
|
| Back to top |
|
|
| shashivarungupta |
| Posted: Mon Mar 17, 2014 10:29 am Post subject: |
|
|
Grand Master
Joined: 24 Feb 2009
Posts: 1343
Location: Floating in space on a round rock.
|
| JosephGramig wrote: |
Sounds like your logs are damaged. Open a PMR and hope they can help you.
| Code: |
| rcdmqimg -m <QmgrName> -t all "*" |
Should save and move stuff to the current active logs. It will give you two files (might be the same one). All files older than the oldest of the two are no longer needed as they will never be used again. |
yea..PMR might help !!
Above command would save the image for ALL (including temporary objs , if any) the mq objs of that queue manager, at this moment when qmgr is running and some of the MQ objs are found damaged.
But I think in the above OP.. it was captured somewhere only some of the object are damaged and there are chance that queues might have some persistent data.. how to recover that in this case ?
In such cases.. when the log for media recovery is damaged itself.. what to do next to fix the issue at that moment (temporary/permanently) ?
Open a sev-1 PMR would again take some time, incase of production/business impacting issues., cause PMR team might need some logs/evidences and of course time to analyze and say something...
_________________
*Life will beat you down, you need to decide to fight back or leave it. |
|
| Back to top |
|
|
| shashivarungupta |
| Posted: Mon Mar 17, 2014 11:14 am Post subject: |
|
|
Grand Master
Joined: 24 Feb 2009
Posts: 1343
Location: Floating in space on a round rock.
|
| exerk wrote: |
| More to the point, what changes were made that caused you to restart your queue manager? |
In some or other scenarios, which I've seen (and sometimes created locally) ... I found that shifting the active logs (which are required by mq qmgr for media and restart recovery) creates the issue like damaged mq qmgr log files.
And shifting of active logs might not be intentional i.e. could have been done by Server operation team during their maintenance exercise .. by mounting/unmounting of mq file system (when qmgr was running).
YES, this exercise could have been done in coordination with the MQ team to end the queue manager properly before they do anything on mq file system where qmgr logs were mounted.
Some of the FDC would be obvious in such cases, which might be useful for PMR Team.
In one of the cases, FDC with 'HL200143' Probe Id would get generated where 'Component' was mqlRandomRead, 'Major Errorcode' was hrcE_MQLP_BADLSN. These could be more understood by PMR team.
_________________
*Life will beat you down, you need to decide to fight back or leave it. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Mar 17, 2014 11:59 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| shashivarungupta wrote: |
In some or other scenarios, which I've seen (and sometimes created locally) ... I found that shifting the active logs (which are required by mq qmgr for media and restart recovery) creates the issue like damaged mq qmgr log files.
And shifting of active logs might not be intentional i.e. could have been done by Server operation team during their maintenance exercise .. by mounting/unmounting of mq file system (when qmgr was running).
YES, this exercise could have been done in coordination with the MQ team to end the queue manager properly before they do anything on mq file system where qmgr logs were mounted. |
See my comments above re: disk manangement.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Andyh |
| Posted: Mon Mar 17, 2014 12:42 pm Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 239
|
There appears to be a fair amount of confusion in this thread, which makes it somewhat difficult to comment sensibly without repeating the entire thread.
If you're suffering from damaged objects, and these damaged objects are not self inflicted (e.g by manipulating queue manager files outside of queue manager control), then you should raise this with MQ support (regardless of whether you are able to recover those objects with the rcrmqobj command).
If all log extents newer then the MEDIALOG are still available (and intact), then it should be possible to recover a damaged object from those queue manager recovery logs (assuming linear logging) to the current point in time. This is conceptually different from replaying messages from the recovery logs, which is only supported for messages which have not subsequently been consumed (i.e. messages still in the queue at the current time).
The information necessary to recover consumed messages is present in the recovery logs, but IBM does not provide a tool to replay those messages.IBM does provide the dmpmqlog command which can be used to inspect the content of the recovery logs.
If the relevant media recovery logs are not available (in exactly the state as when the queue manager FINISHED writing to them) the damaged object would have to be deleted and recreated. If you want MQ support to be able to do a full investigation into the cause of both the damage, and/or any inability to recovery a damaged object then you should save a copy of the queue file and the recovery logs BEFORE recovering the object. A damaged object cannot ever be recovered from the object file (in the event that the required log files are not available for media recovery).
At queue manager restart, log extents from RECLOG or later may be referenced by the queue manager. Log extents older than RECLOG, but equal to or newer than MEDIALOG will only ever be used (by the queue manager) for media recovery. If a log equal to or newer than RECLOG is corrupt, or not available then queue manager restart would be expected to fail.
Taking a media image of all objects will move the MEDIALOG forwards, but will not directly effect RECLOG and so will not necessarily reduce the number of active log files.
Possible reasons for an HL200143 FDC would include:
1. Invalid archive/restore procedures, for example copying a recovery log extent to archive media before the queue manager had finished writing to that log extent. The CURRLOG queue manager status attribute identifies the log extent currently being written to by the queue manager, and all earlier log extents can then safely be copied as required. The RESET QMGR TYPE(ADVANCELOG) command can be used to request the queue manager to move to a new log extent to allow the earlier 'current' extent to be archived.
2. Using LogWriteIntegrity=SingleWrite in an inappropriate environment
Regards
Andy. |
|
| Back to top |
|
|
| shashivarungupta |
| Posted: Wed Mar 19, 2014 4:26 am Post subject: |
|
|
Grand Master
Joined: 24 Feb 2009
Posts: 1343
Location: Floating in space on a round rock.
|
thanks for the information.
hrcE_MQLO_UNEXPECTED_OS_ERROR , in one of the latest scenario the FDC was found with aforesaid error , when MQ was trying to read a log record.
_________________
*Life will beat you down, you need to decide to fight back or leave it. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Mar 19, 2014 5:34 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| shashivarungupta wrote: |
thanks for the information.
hrcE_MQLO_UNEXPECTED_OS_ERROR , in one of the latest scenario the FDC was found with aforesaid error , when MQ was trying to read a log record. |
Seems silly but as nobody asked yet... Did you check the permissions on the log that cannot be read by the qmgr and what are they? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
