| Author |
Message
|
| George Carey |
 Posted: Mon Sep 17, 2012 9:16 pm Post subject: rewording and more |
 |
|
Knight
Joined: 29 Jan 2007
Posts: 500
Location: DC
|
In my previous post I should have said: "... long running trx not crossing more then one logfile ..." (instead of " ...not crossing a checkpoint .."). Anyway ....
Thinking about those 99 log primary files not yet used ... maybe they are referred to as available (not active or inactive ... like logbuffers in a pool). And maybe the log manager looks at both the available number of primary logfiles and the number of active logfiles and does a little ratio calculation and says hey my active log files to total primary now available is less then some threshhold %age ... go allocate and format some more ?? I don't know ...
Anybody ... Anybody ... Ferris where is Ferris?
_________________
"Truth is ... grasping the virtually unconditioned",
Bernard F. Lonergan S.J.
(from book titled "Insight" subtitled "A Study of Human Understanding") |
|
| Back to top |
|
 |
| George Carey |
| Posted: Mon Sep 17, 2012 9:47 pm Post subject: a post missed |
|
|
Knight
Joined: 29 Jan 2007
Posts: 500
Location: DC
|
Thought I posted this but may not have..
Maybe the 99 other logfiles are called available, kind of like an active pool for logbuffers and when a ratio of active to available hits a threshold percentage new logfiles get allocated and formated from the available disk space. Don't know?
Anyway found this about long running trxs ... which almost answers the question but still does not say when new linear logfiles are allocated and formatted (at least in relation to this concept of a primary set of log files)
| Quote: |
| "...When the log head is moved and you are using circular logging, the primary log files might become eligible for reuse and the logger, after filling the current file, reuses the first primary file available to it. If you are using linear logging, the log head is still moved down the active pool and the first file becomes inactive. A new primary file is formatted and added to the bottom of the pool in readiness for future logging activities. ..." |
from:
http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/topic/com.ibm.mq.amqzag.doc/fa14550_.htm
Now I know what the log head is but what is the bottom(the last log record?) and now the text says the bottom of the pool ... almost like he talking about a logbuffer pool?
_________________
"Truth is ... grasping the virtually unconditioned",
Bernard F. Lonergan S.J.
(from book titled "Insight" subtitled "A Study of Human Understanding") |
|
| Back to top |
|
|
| exerk |
| Posted: Tue Sep 18, 2012 12:30 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
I think (and I stress think) that many years ago I read that 'new' linear log files are created when the current file is at 80% usage, but even trawling my document archives I can't find it so it may be a false memory.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| Andyh |
| Posted: Tue Sep 18, 2012 12:57 am Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 237
|
As I stated in my previous append, for circular logging I favour a small number of large extents, for linear logging there are more complex considerations specific to each situation.
There is a considerable overhead, both in terms of performance and administration to using linear logging. From a performance perspective the amount of data written to the device hosting the log will be at least twice as high, as each linear log extent is pre-formatted before use. From an admin perspective there is a considerable burden to managing the linear log extents and the frequency with which media images are recorded.
250-1000 small msgs/sec is a very light workload in todays terms, unless you've got very severe application serialization or disk limitations (e.g write cache is important) I don't think there's likely to be much of a performance issue with the choice of log config. |
|
| Back to top |
|
|
| George Carey |
| Posted: Tue Sep 18, 2012 6:25 am Post subject: application serialization |
|
|
Knight
Joined: 29 Jan 2007
Posts: 500
Location: DC
|
Yes, application serialization is a key component. Data points must maintain and processed in their sent sequence order.
Not super high throughput put not slow either. 250msg/sec is 15,000 msg/min and 900,000msg/hr and 21.6 million msg/24hr-day and none are to be lost or out of order. Message size is roughly 1k hdr+payload.
So for circular a max setting of -lf 65535 seems would be best in all cases as these will be allocated once for all time and not dynamically(except for secondary which we plan not to use). In linear you pay a price for formating 256mb dynamically. So what might be a rule of thumb be for a logfile extent size?
In your example you suggested make the log primary size a 2-tuple of the longest trx time x data logging rate. Would that be for the whole primary allocation or should that be the size for a logfile extent file? A 15 sec trx time for a 10mb logging rate would I think say make your extent size 150mb so any current trx would complete before a another new extent is needed. So use 150mb the multiple till the end of your available log space. The only consideration might be how much of an impact to current logging does allocating and formatting 150mb extent make.
Am I looking at this correctly?
Finally when do linear primary logs get allocated and formatted? Still trying to get an answer to this.
Ah Exerk:
| Quote: |
| I think (and I stress think) that many years ago I read that 'new' linear log files are created when the current file is at 80% usage, but even trawling my document archives I can't find it so it may be a false memory. |
I didn't see your post there at first. So you are saying you think it is a ratio test being done by the Log Manager! Well we have 1 bid for 80% do I hear another? We got 80, 80 right here 80 do I hear 85 ...
_________________
"Truth is ... grasping the virtually unconditioned",
Bernard F. Lonergan S.J.
(from book titled "Insight" subtitled "A Study of Human Understanding") |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Sep 18, 2012 12:41 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
This new linear logfile when primary at 80% could be secondary files.
Primaries get assigned as soon as the oldest primary gets freed. 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| George Carey |
| Posted: Tue Sep 18, 2012 4:57 pm Post subject: Gets Freed |
|
|
Knight
Joined: 29 Jan 2007
Posts: 500
Location: DC
|
Gets Freed ... a new term of ambiguity.
Do you mean by this ... is no longer needed as a checkpoint and media image was taken moving the Head of the log past this extent and so is no longer needed for restart or recovery and has become inactive ... Freed ?...which then in circular logging means can be reused and in linear can be archiveNdeleted or just deleted.
Ok then, you ARE saying when primaries are allocated and formatted (and did before as well ... rereading your earlier post). When the oldest primary is used. Let me try to interpret that. If I have a -lp 10 and -ll,
a.) First are 10 log extents created when qmgr is created(linear) (yes,no)
b.) Are you saying when the 1st (because it would be the oldest) log extent gets 'freed' (as defined above) this triggers the log manager to go allocate and format another extent. This would keep the pool of active and/or available extents at 10!! Which is something I was guesstimating way back in the beginning of this post.
C.) Am I interpreting you properly and as said in 'My Cousin Vinny' ... Are you sure? ... that is how it works ..
_________________
"Truth is ... grasping the virtually unconditioned",
Bernard F. Lonergan S.J.
(from book titled "Insight" subtitled "A Study of Human Understanding") |
|
| Back to top |
|
|
| George Carey |
| Posted: Wed Sep 19, 2012 4:21 am Post subject: Related question |
|
|
Knight
Joined: 29 Jan 2007
Posts: 500
Location: DC
|
An important related question to all this that would factor into how to size primary log file extents for 'circular' logging.
As noted in the InfoCenter documentation illustrations for long running transactions. When a LR trx holds an extent from being freed(as defined above) because the trx has not completed with either a commit or rollback, is (and here is the question), is the whole extent(i.e. the whole logfile) unavailable to be reused until the trx completes or not?
Example: Say a log file has been assigned the max size of 65535 pages which is 256MB. If I have a logged 'put' in the last say 1MB of this extent/logfile that is part of a not yet completed transaction but a checkpoint has been done also in the last 1MB of this file lets say before the 'put'. Are the remaining 255MB available to be recycled (i.e. written to again for circular logging or must the whole extent/logfile be available, meaning the 'put' trx must complete freeing the whole logfile/extent before any of the logfile can be reused?
This would argue for smaller logfiles if one cannot reuse any part of it until all trxs even trxs after a checkpoint but still in the same logfile hold the log file as inaccessible to be reused for circular logging.
Anybody ...
_________________
"Truth is ... grasping the virtually unconditioned",
Bernard F. Lonergan S.J.
(from book titled "Insight" subtitled "A Study of Human Understanding") |
|
| Back to top |
|
|
| George Carey |
| Posted: Wed Sep 19, 2012 6:12 am Post subject: T-Rob comment |
|
|
Knight
Joined: 29 Jan 2007
Posts: 500
Location: DC
|
Here is something from a recent post by T-Rob on the List-server forum. It relates to an approach for sizing log files:
"...You can
tune the latency between the primary and secondary QMgrs by adjusting the
size of the log files. The smaller they are, the quicker they roll off.
Too small and transactions are rolled back to free log space. You need
them to be large enough to hold all simultaneous outstanding transactions
and small enough to minimize the recovery point. With a bit of
experimentation and tuning you can get a decent recovery point on a busy
QMgr. It tends to be less useful if your log files don't rotate often,
though. ..."
Smaller seems to be better but not too small. Not clear on that last sentence , however.
_________________
"Truth is ... grasping the virtually unconditioned",
Bernard F. Lonergan S.J.
(from book titled "Insight" subtitled "A Study of Human Understanding") |
|
| Back to top |
|
|
| nathanw |
| Posted: Wed Sep 19, 2012 6:19 am Post subject: |
|
|
Knight
Joined: 14 Jul 2004
Posts: 550
|
So basically how long is a piece of string
| Quote: |
With a bit of
experimentation and tuning you can get a decent recovery point on a busy
QMgr. |
May I suggest that this is the best piece of advice. Calculations can only take you so far. Work out the values the experiment to fine tune.
_________________
Who is General Failure and why is he reading my hard drive?
Artificial Intelligence stands no chance against Natural Stupidity.
Only the User Trace Speaks The Truth  |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Sep 19, 2012 6:23 am Post subject: Re: T-Rob comment |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9405
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| "It tends to be less useful if your log files don't rotate often, though. " |
| George Carey wrote: |
| Not clear on that last sentence , however. |
For circular logs, there should to be enough primary log files to anticipate brief spikes in number of transaction UofWs, and duration of these UofW's. Yes, secondaries provide for this; but initial allocation of additional primaries is a bit less labor-intensive than allocation of secondaries in-flight.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Andyh |
| Posted: Wed Sep 19, 2012 11:29 am Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 237
|
The 80% value that's being quoted in this thread relates to the point at which the queue manager will start taking action to recover log space. When 80% of the total defined (primaries + secondaries) space is in use the queue manager will start considering asynchronous rollback of long running transactions in order to recover log space.
I indicated in my earlier append that with linear logging, new extents are normally scheduled to be formatted (asynchronously) as each current extent fills. It's actually a little more complicated than this, in that the queue manager tries to keep the amount of active log space to less that 80% of the defined total of primaries and secondaries. The queue manager actually schedules the allocation of a new extent when an old extent is no longer required for restart recovery (that is no active transaction is referenced in that log extent).
When I suggested that the size of the log should be based upon the duration of the longest running transaction, and the MAXIMUM data rate on the log, I intended the calculation to be for the size of the set of primary log extents. Note that I didn't intend that 2 should actually be the factor, more that 2 would be the minimum factor I'd consider.
One other thing you might like to think about is what effect a long running transaction could have on restart recovery times. Imagine that you've got a transaction that runs for 100 seconds, if the queue manager should crash while this transaction is active then ALL of the log records written by that transaction would have to be read during restart recovery. Very often when a long running transaction is discovered, the sensible action is to reduce the duration of that transaction rather than to increase the size of the recovery log.
Even with linear logging a transaction will not be allowed to span more than the space defined to primaries and secondaries combined. For example, in the example where there are 100 extents in the log directory, if the queue manager were defined with 8 primaries and 2 secondaries then at most 10 of these log files could be 'active'. The other 90 might still be required for media recovery, but should not be required for restart recovery. |
|
| Back to top |
|
|
|
|
