| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| MQ Logging and Archive Issues |
« View previous topic :: View next topic » |
| Author |
Message
|
| vivica12 |
 Posted: Wed Oct 24, 2007 11:14 am Post subject: MQ Logging and Archive Issues |
 |
|
Acolyte
Joined: 13 Jul 2007
Posts: 58
|
As I read more and more about MQ Logging, I get more confused.
I read on a forum here the following
At 5.3 this was a no brainer, becasue the MQ limit was only about 4.5 GB for all the logs together if you made them as big as possible and as many as possible. At 6.0, they can be a lot bigger, so let your disk space be the guide.
I have not seen that anywhere else. Can someone explain this? Is this a total log space MQ limit in 5.3? Will doing a rcdmqimg be an issue if /var/mqm/log is 10GB in size?
Second question. I am on MQ v5.3, I have Primary logs set to 3, secondary set to 2 (yes it's defaults, but I didn't build it), using linear logging. When I look in my log directory there are 100's of logs, not just 5. I realize this is because my log file size is small, but doesn't this mean that P/S don't really matter?
Is circular logging better? Can you still recover persistant messages in circular logging?
Logging and archiving in MQ have caused us so many issues, definately more work then it should be.[/i]
_________________
Vivica - signing off |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Wed Oct 24, 2007 11:58 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
It's only with circular logging that you are guaranteed a maximum number of log files - because it's only with circular logging that MQ will re-use a log file.
Primary/Secondary are all about "active log files".
With linear logging, you have always had to have a process in place to manage the removal/archiving of inactive log files.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Oct 24, 2007 1:43 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| jefflowrey wrote: |
| With linear logging, you have always had to have a process in place to manage the removal/archiving of inactive log files. |
You might want to try the MO73 Support Pac as a starting point for this. If you're running rcdmqing regularly (as your other post indicates) and you have well behaved applications then it's unlikely you actually need the 100s of log files you mentiion.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Oct 24, 2007 7:53 pm Post subject: Re: MQ Logging and Archive Issues |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| vivica12 wrote: |
As I read more and more about MQ Logging, I get more confused.
I read on a forum here the following
At 5.3 this was a no brainer, becasue the MQ limit was only about 4.5 GB for all the logs together if you made them as big as possible and as many as possible. At 6.0, they can be a lot bigger, so let your disk space be the guide.
I have not seen that anywhere else. Can someone explain this? Is this a total log space MQ limit in 5.3? |
That quote looks familiar.....anywho, in MQ 5.3 if you made your circualr logs as big as possible that was the limit. About 4.5 GB.
| vivica12 wrote: |
Is circular logging better? Can you still recover persistant messages in circular logging?
|
Define the scenario that you have to recover from.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| JuulV |
| Posted: Thu Oct 25, 2007 12:04 am Post subject: |
|
|
Apprentice
Joined: 05 Sep 2007
Posts: 28
Location: Belgium
|
Sounds like you would be helped by reading the relevant chapters in the System Admin Guide, notably:
crtmqm command in the WMQ Control Commands chapter, and
the chapters on Recovery and Restart
But in short:
The total size of the active log files is limited and documented (I'll let you do the maths, but WMQ 5.3 allows at most 63 log files of 16,384 blocks of 4K each, and WMQ 6.0 increases this to 255 (Win) or 511 (Unix/linux) files of 65,535 blocks).
When you're using circular logging, you'll never get more log files than defined by the "logPrimaryFiles" and "logSecondaryFiles" attributes of the queue manager. Normally, "logPrimaryFiles" log files are permanently allocated (unless this value was increased after creating the queue manager, in which case the additional Primaries only get allocated when they're first needed). If WMQ needs more logfiles to hold active log data (as needed for restart recovery), up to "logSecondaryFiles" additional log files may be created by the QMgr; these files will be deleted again when they're no longer needed. Although not really 100% correct, you could say that with linear logging, all files that exist are active log files.
When you're using linear logging, every time a log file fills up, a new log file is allocated (until you run out of disk space, or regularly clean up the log directory, E.g. using support pack MO73). Only those log files which are needed for recovery (because they contain data from active transactions, or are otherwise needed for restart), are active log files.
In either case however, you can't have more active log files than the sum of the "logPrimaryFiles" and "logSecondaryFiles" attributes of the queue manager. If this happens, all calls that require records to be written to the log are failed with MQRC_RESOURCE_PROBLEM, until log space becomes available again (because transaction commit or roll back) (and thus yes, even when using linear logging, "logSecondaryFiles" does have a meaning).
Whether to use circular or linear logging, depends on your (application) needs:
Circular logging only ensures RESTART recovery, allowing all transactions that were active at the time of a failure to be rolled back (up to the last quiesce point). MEDIA IMAGES (which you talk about) are not created when using linar logging!!!
Linear logging allows both RESTART (roll back) and MEDIA (roll forward) recovery. MEDIA IMAGES are used here to limit the amount of log data that has to be processed for media recovery.
In conclusion, linear logging offers better recoverabilty, but requires some additional house keeping of inactive log files (there's no such thing as a free lunch, my friend).
_________________
Juul Vanparijs
Senior Developer
Cressida Technology Ltd |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Oct 25, 2007 2:41 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| JuulV wrote: |
| MEDIA IMAGES (which you talk about) are not created when using linar logging!!! |
I'm sure you meant to say:
MEDIA IMAGES (which you talk about) are not created when using circular logging!!! 
In fact when you run rcdmqimg it will tell you whether you can create a media image or not as this is the program that does create the media image... and yes it is recommended you run it at a regular interval when using linear logging
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| vivica12 |
| Posted: Thu Oct 25, 2007 5:12 am Post subject: |
|
|
Acolyte
Joined: 13 Jul 2007
Posts: 58
|
I truly have read the manual, I've read additional docs, I've read many posts etc. But what I read isn't equating to what I see happening.
Here are the specifics right this minute.
My log definitions are:
LogPrimaryFiles=3
LogSecondaryFiles=2
LogFilePages=1024
Meaning 4MB log file sizes (yes i realize that probably isn't large enough but I can't recreate now)
If I check my required logs right now I get
checkmqlogs: QMGR: Needed for Media Recovery: S0349799.LOG
checkmqlogs: QMGR: Needed for Qmgr Restart: S0349876.LOG
If I look in my actual /var/mqm/log/active/qmgr/ directory I have 83 logs AFTER the S0349799.LOG, and it is my udnerstanding that I need to keep all of those logs. They are listed below for proof.
Obviously that's more then my 5 P/S log files defined. I do run a rcdmqimg (with an log removal) daily at 3am, but a lot of time it only recovers less then 5% of the space for me. I also have it set to run when file system reaches 70% full. This has bitten us badly before (PMR's open with IBM etc), and logs actually filled up.
I am jsut trying to understand how to configure logging better, and where I get so many logs from -- so that I can build my new v6 environment properly.
thanks
log listing:
-rw-rw---- 1 mqm mqm 4202496 Oct 25 03:00 S0349799.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 03:00 S0349800.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 03:00 S0349801.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 03:01 S0349802.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 03:15 S0349803.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 03:22 S0349804.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 03:26 S0349805.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 03:40 S0349806.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 03:52 S0349807.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 04:00 S0349808.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 04:11 S0349809.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 04:21 S0349810.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 04:31 S0349811.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 04:43 S0349812.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 04:53 S0349813.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 05:03 S0349814.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 05:12 S0349815.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 05:20 S0349816.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 05:29 S0349817.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 05:40 S0349818.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 05:49 S0349819.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 05:58 S0349820.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 06:04 S0349821.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 06:11 S0349822.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 06:16 S0349823.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 06:20 S0349824.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 06:23 S0349825.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 06:28 S0349826.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 06:32 S0349827.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 06:36 S0349828.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 06:40 S0349829.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 06:43 S0349830.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 06:46 S0349831.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 06:49 S0349832.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 06:52 S0349833.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 06:55 S0349834.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 06:57 S0349835.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:00 S0349836.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:03 S0349837.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:05 S0349838.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:06 S0349839.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:08 S0349840.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:09 S0349841.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:11 S0349842.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:14 S0349843.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:16 S0349844.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:18 S0349845.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:19 S0349846.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:20 S0349847.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:22 S0349848.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:23 S0349849.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:25 S0349850.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:27 S0349851.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:29 S0349852.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:30 S0349853.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:32 S0349854.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:35 S0349855.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:37 S0349856.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:39 S0349857.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:40 S0349858.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:42 S0349859.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:43 S0349860.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:44 S0349861.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:45 S0349862.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:46 S0349863.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:47 S0349864.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:48 S0349865.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:49 S0349866.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:50 S0349867.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:51 S0349868.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:52 S0349869.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:53 S0349870.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:54 S0349871.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:55 S0349872.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:56 S0349873.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:57 S0349874.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:58 S0349875.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 07:59 S0349876.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 08:00 S0349877.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 08:01 S0349878.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 08:02 S0349879.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 08:03 S0349880.LOG
-rw-rw---- 1 mqm mqm 4202496 Oct 25 08:02 S0349881.LOG
_________________
Vivica - signing off |
|
| Back to top |
|
|
| vivica12 |
| Posted: Thu Oct 25, 2007 5:27 am Post subject: |
|
|
Acolyte
Joined: 13 Jul 2007
Posts: 58
|
In either case however, you can't have more active log files than the sum of the "logPrimaryFiles" and "logSecondaryFiles" attributes of the queue manager. If this happens, all calls that require records to be written to the log are failed with MQRC_RESOURCE_PROBLEM, until log space becomes available again (because transaction commit or roll back) (and thus yes, even when using linear logging, "logSecondaryFiles" does have a meaning).
This comment, based on my settings, I should only ever have 5 active logs, and obviously that is not true, and applications are NOT receiving the MQRC_RESOURCE_PROBLEM failure, and 90% of our apps are using persistant messages, and would have to be written to a log.
I have seen this before in the manual, which is part of what prompted all my questions, it isn't happening this way in my setup, and I don't quite understand why.
_________________
Vivica - signing off |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Oct 25, 2007 5:30 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Run rcdmqimg more often.
The P/S, again, are only ACTIVE log files and not the ones that may be needed for media recovery.
The last log file you need for qmgr restart is the S0349876. The newest log file listed is S0349881.
You see how that's a lot closer to your P/S numbers... ? The sixth one may be there because the qmgr is getting ready to, or in the process of, moving to it.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| vivica12 |
| Posted: Thu Oct 25, 2007 5:49 am Post subject: |
|
|
Acolyte
Joined: 13 Jul 2007
Posts: 58
|
That makes more sense. Let's see if I understand.
I now have 119 logs.
Media Recovery log is still the same
Qmgr restart changed
checkmqlogs: QMGR: Needed for Media Recovery: S0349799.LOG
checkmqlogs: QMGR: Needed for Qmgr Restart: S0349916.LOG
I have 2 logs AFTER the QMGR Restart log. So my active logs are
S0349916.LOG (qmgr restart)
S0349917.LOG
S0349918.LOG
BUT all logs after S0349799.LOG and before S0349916.LOG, would be needed for media recovery.
Did I get that correct?
If I fill up log S0349919.LOG, S0349920.LOG, S0349921.LOG, will my QMGR Restart log then reset so that I continue to keep a max of 5 active logs?
Even in typing this message I am at log S0349921.LOG, so as you can see, they fill up fast.
Any recommendations on size of logs -- does the individual size of each log cause problems when doing a rcdmqimg?
(my qmgr restart just switched to log S0349919.LOG
_________________
Vivica - signing off |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Oct 25, 2007 5:58 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
All log files from the last media checkpoint are needed for media recovery.
This includes currently active log files...
In your other thread, I made some recommendations on resizing your logs. Or at least, some comments...
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Oct 25, 2007 6:10 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| vivica12 wrote: |
BUT all logs after S0349799.LOG and before S0349916.LOG, would be needed for media recovery.
Did I get that correct?
|
You need S0349799 for media recovery, everything from S0349916 onwards for restart. Once you run a rcdmqimg the *799 will become a more current log (which will now have the latest image in it).
And I concur with jefflowrey in his comments on your other post.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
