| Author |
Message
|
| George Carey |
 Posted: Sat Sep 15, 2012 7:47 pm Post subject: performance and logfiles |
 |
|
Knight
Joined: 29 Jan 2007
Posts: 500
Location: DC
|
Which is better to have:
1.)Small to medium number of logfiles of a larger size or
2.)Medium to Large number of logfiles of a smaller size
What is the trade off?
Cannot figure this out from Peter Toghill's performance recommendations.
It says for logfilepages ... max is 65536 (which gives 4096 x 65536 pages = 256MB per file?) where a page is 4096 bytes or 4K.
If I have 50GB available for Log space how should it be allocated 1.) or 2.)
_________________
"Truth is ... grasping the virtually unconditioned",
Bernard F. Lonergan S.J.
(from book titled "Insight" subtitled "A Study of Human Understanding") |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Sat Sep 15, 2012 9:26 pm Post subject: Re: performance and logfiles |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
| George Carey wrote: |
Which is better to have:
1.)Small to medium number of logfiles of a larger size or
2.)Medium to Large number of logfiles of a smaller size
What is the trade off?
Cannot figure this out from Peter Toghill's performance recommendations.
It says for logfilepages ... max is 65536 (which gives 4096 x 65536 pages = 256MB per file?) where a page is 4096 bytes or 4K.
If I have 50GB available for Log space how should it be allocated 1.) or 2.) |
Depends on what your need and throughput is and whether you are on windows or Unix/Linux...
On Windows you might want to reserve the disk space so it can't be stolen... using a max of primary files.
On Unix/Linux you can use the minimum of primary and make up the max with secondary files. This allows you to measure the usage by the number of logfiles used or ready for archiving (linear logging). (Remember file size is allocated by the system and will not get stolen for other purpose...without admin intervention!)
In any case remember to check on how many times a day you wish to recycle log files and plan for the size and number accordingly.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| George Carey |
| Posted: Sun Sep 16, 2012 8:39 am Post subject: log file cycling |
|
|
Knight
Joined: 29 Jan 2007
Posts: 500
Location: DC
|
Talking about linux boxes with small messages 50-100bytes data payload, relatively high throughput currently 250 msgs/sec going to 500 to 1000 messages/sec over next few years.
What about log file size small or large?
I would think large for linear would pay a price as when a logfile goes inactive and a new one needs to be allocated and formatted to become active to keep the active pool constant(which from reading appears how it works) the time to format a large logfile would be severely impactful to the running environment.
Also if SAN or other storage with mirroring are being used for data and logs there is no benefit to linear log as recovery can be handled by other means then MQ (see peter toghill developerwork piece of 2007). In circular logs, formatting for primaries is done on QMGR creation. Only secondary if ever needed get formatted dynamically.
What do you mean by recycling the log files? In linear a new logfile is allocated when old goes inactive(i.e. space used and no longer needed for restart recovery) it just needs to be purged or archived for media recovery.
In circular, inactive logfiles just get reused. What does recycle the logs mean?
_________________
"Truth is ... grasping the virtually unconditioned",
Bernard F. Lonergan S.J.
(from book titled "Insight" subtitled "A Study of Human Understanding") |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Sep 16, 2012 11:20 am Post subject: Re: log file cycling |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
| George Carey wrote: |
Talking about linux boxes with small messages 50-100bytes data payload, relatively high throughput currently 250 msgs/sec going to 500 to 1000 messages/sec over next few years.
What about log file size small or large?
I would think large for linear would pay a price as when a logfile goes inactive and a new one needs to be allocated and formatted to become active to keep the active pool constant(which from reading appears how it works) the time to format a large logfile would be severely impactful to the running environment.
Also if SAN or other storage with mirroring are being used for data and logs there is no benefit to linear log as recovery can be handled by other means then MQ (see peter toghill developerwork piece of 2007). In circular logs, formatting for primaries is done on QMGR creation. Only secondary if ever needed get formatted dynamically.
What do you mean by recycling the log files? In linear a new logfile is allocated when old goes inactive(i.e. space used and no longer needed for restart recovery) it just needs to be purged or archived for media recovery.
In circular, inactive logfiles just get reused. What does recycle the logs mean? |
You seem to be under the misconception that you need to create the log file before using it. Remember minimum primary log files is 2!
This means when you use file 1 you create file 2 so that you always have files available... whether #2 is secondary or primary does not matter much...
Agreed we do not see the throughputs you are specifying there. My best advice, test and see what combination of primaries and secondaries works best for you.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sun Sep 16, 2012 10:08 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
There's certainly no substitute for running your own tests.
It is slower to span a transaction across more than one log file, and slower to allocate a new log file.
So it is nominally better to have fewer, larger log files. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Sep 17, 2012 4:31 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9403
Location: US: west coast, almost. Otherwise, enroute.
|
And you end up with fewer (linear) log files to manage (archive, delete) manually.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| George Carey |
| Posted: Mon Sep 17, 2012 6:42 am Post subject: log value |
|
|
Knight
Joined: 29 Jan 2007
Posts: 500
Location: DC
|
So how big.
The value in doco says 65535 is max logfilepages at 4k which gives 256MB for each log file. Then the max 511 log files gives a lot of log space.
I keep seeing a number 16384 ... is that an older max logfilepages or what is that?
_________________
"Truth is ... grasping the virtually unconditioned",
Bernard F. Lonergan S.J.
(from book titled "Insight" subtitled "A Study of Human Understanding") |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Sep 17, 2012 6:55 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| You should have enough primary log file space to handle your normal transaction load without moving to secondary log, and then enough secondary to handle the worst case you can think of. |
|
| Back to top |
|
|
| exerk |
| Posted: Mon Sep 17, 2012 7:15 am Post subject: Re: log value |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| George Carey wrote: |
...Then the max 511 log files gives a lot of log space.
... |
Dependent on platform 
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| Andyh |
| Posted: Mon Sep 17, 2012 8:01 am Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 237
|
The maximum size of a log file increased from 16384 pages to 65535 pages in MQ V6.
If you're using circular logs then a smaller number of larger log files is generally preferable. The size of each log file is fixed for all time when a queue manager is created.
Allocating sufficient primaries such that use of secondary logs is unusual is preferable.
The principal considerations when choosing the number of log files are the durations of the longest individual transactions and the peak throughput on the log. For example if you're logging at 10MB/sec and your longest transaction takes 15 seconds you would choose a log spec based upon the fact that 150MB of log space might be used while this transaction ran. It would then be necessary to apply some sort of fudge/contingency factor to this calculation. For example in the above case 300MB of primary log space might be a minimal starting point for consideration.
In the case on linear logging things are a little more complicated, for example are the inactive logs archived for disaster recovery. Similarly archiving and formatting will take place in units of the log size and hence a large log file size might be less attractive. |
|
| Back to top |
|
|
| George Carey |
| Posted: Mon Sep 17, 2012 3:59 pm Post subject: Ok now we're talking |
|
|
Knight
Joined: 29 Jan 2007
Posts: 500
Location: DC
|
Ok good feedback AndyH.
So how would you allocate your 300MB primary log space?
-lp 300 with each logfile=1mb
-lp 20 with each logfile=15mb
-lp 3 with each logfile=100mb
-lp other with each logfile=other?
What advantage or disadvantage to using more log files no matter the size?
Finally when does a logfile get allocated and formated?(talking linear here)
rgds,
GTC
_________________
"Truth is ... grasping the virtually unconditioned",
Bernard F. Lonergan S.J.
(from book titled "Insight" subtitled "A Study of Human Understanding") |
|
| Back to top |
|
|
| George Carey |
| Posted: Mon Sep 17, 2012 6:28 pm Post subject: when is log file formatted |
|
|
Knight
Joined: 29 Jan 2007
Posts: 500
Location: DC
|
Still need to understand when the linear log files are allocated and formatted.
This seems to me to be key. If I have 10 logfiles or 100logfiles I will come to the end at some point for either as logfiles are not recycled with linear, they are either active or inactive and the inactive ones need to be either archived and deleted or just deleted to get space back to allocate more logfiles.
But when are these more logfiles allocated and formatted? As soon as one is full another is allocated and formatted or when the full set of primaries comes to an end (ie in my example after 1 is used or after 10 are used or after 100 are used?) and then how many get allocated and formated 10, 100 or 1 or none of these?
Hypothetical question: Say I have 110MB for log space. I choose linear log and allocate logfilepages so each logfile is 1MB in size. So I allocate up the whole space -lp100 and -ls 10.
1.) When do these 100 log primary files get allocated and formatted
2.) What happens when I filled up the first primary log file.
Just another way of asking my previous question.
TIA
I am befuddled ... a frequent occurence.
_________________
"Truth is ... grasping the virtually unconditioned",
Bernard F. Lonergan S.J.
(from book titled "Insight" subtitled "A Study of Human Understanding") |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Sep 17, 2012 7:27 pm Post subject: Re: when is log file formatted |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9403
Location: US: west coast, almost. Otherwise, enroute.
|
| George Carey wrote: |
1.) When do these 100 log primary files get allocated and formatted
2.) What happens when I filled up the first primary log file.
|
1) Primaries are allocated and formatted at qmgr creation. Secondaries are allocated and formatted when needed.
2) When the first primary is filled, the first is closed, and the 2nd is used.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Sep 17, 2012 7:40 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
I think you're looking at linear logs the wrong way.
Let's say you have 100 files in the directory. Chances are (if everything is working right) you only have 2 active files in the lot # 99 and #100...
The others are obsolete and can be archived, after you run rcdmqimg (from memory and its going fast)...
Now if you have a long running transaction all 100 files could be active...
So the answer is really "it depends".
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| George Carey |
| Posted: Mon Sep 17, 2012 8:02 pm Post subject: linear |
|
|
Knight
Joined: 29 Jan 2007
Posts: 500
Location: DC
|
Bruce said:
| Quote: |
1) Primaries are allocated and formatted at qmgr creation. Secondaries are allocated and formatted when needed.
2) When the first primary is filled, the first is closed, and the 2nd is used. |
Linear, linear ... circular is trivial(I think). On circular primary is allocated at QMGR creation time. I noted that.
Try using my example!!
I know a long running trx can make all 100 logs active. That is not my example say checkpoint has been done and no long running transaction going across checkpoint has happened. I have come to the end of a 'linear'(bruce) logfile and need to go to the next one. What does MQ log manager do? In fact what does primary log even mean for linear logging. This brings me back to my repeated question when do primary logs get allocated and formated(linear!!). If you say at qmgr creation time, then when we use up these primaries(which we will !) when do new ones get allocated.
Also a question on definition of tems. Lets say we allocate 100 primary logs ... no activity done yet except internal MQ house keeping. So say one log file is being used. Are the 99 other log primary files said to be inactive or active or what ? That may be a definitional confusion factor.
_________________
"Truth is ... grasping the virtually unconditioned",
Bernard F. Lonergan S.J.
(from book titled "Insight" subtitled "A Study of Human Understanding") |
|
| Back to top |
|
|
|
|
