| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Calculation for sizing?? |
« View previous topic :: View next topic » |
| Author |
Message
|
| sac063 |
 Posted: Mon Aug 29, 2005 4:21 am Post subject: Calculation for sizing?? |
 |
|
Apprentice
Joined: 23 Jan 2004
Posts: 36
|
Hi..
I am back again with the painful topic of log sizing. If I need to size my linear logs for 400K persistent messages/day (with backup taken every day). I need to figure out - what exactly should be the primary log number? Here is a calculation I have made based on the guidance given in system admin guide -
1) Put persistent message (750 bytes + message length If the message is large, it is divided into segments of 15700 bytes, each with a 300-byte overhead.) - Put once by external app and once by broker - 2 * 400K* 750 = 572 MB
2) Get message - 260 bytes - 400K*260 = 100 MB
3) Syncpoint, commit or Syncpoint, rollback - No Syncpoint transactions
4) Create object - approx. MQ objects 700 * 1500 bytes = 1 MB
5) Delete object - 300 bytes - Ignored
6) Alter attributes - 1024 bytes - None
7) Record media image - 800 bytes + image - 300 queues for 5000 maxdepth (2KB msgs) - 3000MB
 Checkpoint - The system has 4*400K queue operations/day for 400K messages. If we consider 10K queue operations as the only basis for checkpoint then - 300MB.
All added gives me hardly 4GB size. How many logs should be made and of what size?
Hoping to get this right from manuals rather than trial and erorr... 
Thanks... |
|
| Back to top |
|
 |
| hopsala |
| Posted: Mon Aug 29, 2005 7:58 am Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
How did you get 4GB? 572+100+300+300+300 = ~1.5GB...
Also, if I may ask, I did not quite understand what you meant by the whole "checkpoint" process, did you not already calculate part of this in "put persistent message"? (I do know what a checkpoint is, and how it works, so no need explaining that - just how it concerns given context)
Anyway, you should always remember, that backing up once per day does not mean that that is all the log size you need - log size is better established by maximum malfunction time - that is, if it is possible that an application will go down for a whole week, you will have to have logs big enough to hold 7 days worth of activity.
Think what happens if an appl goes down for two days, in which the queue slowly fills up, and then there's a disk crash. What will happen is that the logs only have one day's activity - recreating lost messages from the log backup may be possible, but with great difficulty (a few qm restart, booting with log and different RBA's, deleting duplicates due to overlap... it's a pain).
So estimate your "malfunction depth" in days, then multiply it by three to be on the safe side  - that's your log size.
Hope this helps. |
|
| Back to top |
|
|
| EddieA |
| Posted: Mon Aug 29, 2005 9:13 am Post subject: |
|
|
Jedi
Joined: 28 Jun 2001
Posts: 2453
Location: Los Angeles
|
| hopsala wrote: |
| and then there's a disk crash |
Hmmmm. Didn't you say this 10 days ago:
| hopsala wrote: |
| Moreover, today discs do not "crash", it simply doesn't happen anymore |
Cheers,
_________________
Eddie Atherton
IBM Certified Solution Developer - WebSphere Message Broker V6.1
IBM Certified Solution Developer - WebSphere Message Broker V7.0 |
|
| Back to top |
|
|
| sac063 |
| Posted: Mon Aug 29, 2005 9:59 am Post subject: |
|
|
Apprentice
Joined: 23 Jan 2004
Posts: 36
|
Thanks for the answer.
| hopsala wrote: |
| How did you get 4GB? 572+100+300+300+300 = ~1.5GB... |
The calculation is ~4GB. You have counted a 3000MB as 300MB 
Anyways, more importantly, my question still stays unanswered. How do you split the the logs? If your total log size comes upto 4 GB then how many primary and secondary logs will you have? Does it depend solely on the message size?
Thanks... |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Aug 29, 2005 11:57 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7716
|
Make them as big as you can and as many as you can, up to the limit of of your hard drive space or MQ. Easy! Why limit yourself? At 5.3 this was a no brainer, becasue the MQ limit was only about 4.5 GB for all the logs together if you made them as big as possible and as many as possible. At 6.0, they can be a lot bigger, so let your disk space be the guide.
Primary versus secondary - good question. I was going to post something about this myself. IBM gives you the option of how many primary vs secondary, but with zero guidance, so why do we have the option even? Never understood the primary vs the secondary. Who cares? Just log what you have to log, what do I care if it is primary or secondary logs being used?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| hopsala |
| Posted: Mon Aug 29, 2005 12:07 pm Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
| hopsala wrote: |
| Moreover, today discs do not "crash", it simply doesn't happen anymore |
EddieA - I did say that, and I stand by it! however, I don't go around persuading anyone who asks about logs that it's not important, and unncessary, etc etc etc... that way no one will get anywhere, so I humbly help with the specific request, and only if asked, talk about greater matters at hand
Oh, and another thing - if someone does not have RAID or imaging backup for some arcane financial/political/technical reason, logging is definitely the way to go. So, back to the subject:
| sac063 wrote: |
| How do you split the the logs? If your total log size comes upto 4 GB then how many primary and secondary logs will you have? Does it depend solely on the message size? |
As I said before, first decide you "malfunction depth" - say you chose it to be 5 days. I would go with something like 8 logs per day - to keep it flexible and managable - that means about 40 primary logs, and accordingly maybe 10 secondary giving you over another day's respite in case of real havoc.
So 40 primary, 10 secondary, each 0.5GB - yields 25GB - very reasonable these days.
It's really quite arbitrary - you have to find a mid-way between having too many logs to handle, and logs too big to manage. |
|
| Back to top |
|
|
| hopsala |
| Posted: Mon Aug 29, 2005 12:12 pm Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
| PeterPotkay wrote: |
| what do I care if it is primary or secondary logs being used? |
But you do care!
Primary (in linear logging) logs is the constant pool of logs used, these are you're "active" logs, and cannot be deleted no matter if they're in use or not. Secondary (again, in linear logging) are only used in case of a long-running transaction, which exceeds the active pool, so that these logs are added to the active pool on need.
Things is, if a secondary log was added to active pool, I don't think it is ever decreased from it, so your active pool grows and grows... only manual handling (requires restart) can decrease active pool size.
So if you ever want to delete inactive logs, instead of them building up on you, you should keep secondary logs checked.
At least, this from my small xp and the manual. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Aug 29, 2005 1:05 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7716
|
| hopsala wrote: |
Primary (in linear logging) logs is the constant pool of logs used, these are you're "active" logs, and cannot be deleted no matter if they're in use or not. |
So understand you are talking to someone who thinks Linear is overkill, so most of my expirience is with Circular (disks don't die, neither does SAN, so if my message is on the q file, I am confident MQ got it!) Anyway, without getting into the whole Linear / Circular debate.....
Are you saying if I create a QM with 60 Primary Linear Logs, and I start the QM up, I need all 60 logs as 1 big chunk, even though probably 59 of them have not been used? That can't be I think. As MQ checkpoints off the logs that are no longer needed, you can archive them. And if I have 60P and 3S, or 3P or 60S, it makes no diff. MQ will let me know which logs are no longer needed. And if that's the case, why do I care how many are P and how many are S?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| hopsala |
| Posted: Mon Aug 29, 2005 1:35 pm Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
| PeterPotkay wrote: |
| So understand you are talking to someone who thinks Linear is overkill, so most of my expirience is with Circular (disks don't die, neither does SAN, so if my message is on the q file, I am confident MQ got it!) |
I couldn't agree more, linear logs are baaaad No really, see what I just told EddieA - today disks don't crash, and volume is practically unlimited, why bother with linears?
| PeterPotkay wrote: |
| Are you saying if I create a QM with 60 Primary Linear Logs, and I start the QM up, I need all 60 logs as 1 big chunk, even though probably 59 of them have not been used? That can't be I think. As MQ checkpoints off the logs that are no longer needed, you can archive them. |
| the guy who wrote System Admin Guide wrote: |
If a new checkpoint is recorded in the second, or later, primary log file, the first file
becomes inactive and a new primary file is formatted and added to the end of the
primary pool, restoring the number of primary files available for logging. In this
way the primary log file pool can be seen to be a current set of files in an
ever-extending list of log files. |
(In truth, I haven't really tried this, and don't have the time right now - but the manual spoketh, and unless I misunderstood, it means that a qm has at least as many P logs active + possibly a few Ss)
Also, it's not true that after checkpoint inactive logs "are no longer needed" - they are not necessary for normal restart, but if you misplace ( ) the queue files, you do need them, whether archived or available; I personally prefer them being on-disk, and archive only really old logs.
Let me put it this way - for someone who, for whatever reason, uses linear logging - secondary-primary considerations are worth thinking about. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Aug 29, 2005 1:37 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20697
Location: LI,NY
|
Peter you do care because of disk utilization:
Primaries are always there visible in the directory.
Secondaries are created and used and deleted as needed.
This is more a Circular logging consideration than a linear logging consideration.
If your directory holds a max of 40 Logs you can run with
20 primaries and 20 secondaries and your used space will always be 50 % or more. If you have 10 primaries and 30 secondaries your used space will vary between 25 % and 100 % but never be less than 25 %.
In linear logging I would say it is prudent to have enough space to hold logs for 26 hours.
Enjoy  |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Aug 29, 2005 7:23 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7716
|
I guess if I size my var/mqm/logs dir to hold a worst case scenario, all primaries and secondaries are used, what do I care if the total used space of that dir is 25%, 50% or 100%. If var/mqm/log *might* get to say, 4 GB, then I have to plan it might get to 4 GB any second now. So basically I have said my logs are going to use up 4 GB. I don't know what I get if half of the time or even most of the time the dir was only 1 GB. I can't give that extra 3 GB to anyone else anyway. In effect, the 4 GB is "in use" regardless.
Hopsala, I read your quotes, but still don't see why a QM with 60 Primaries would be bad. Both you and fjb_saper kinda allude to a possible scenario why I would want the # of primaries to be smaller, but that's in all cases? I guess I want to understand why in Scenario A you use this many Ps and this many Ss, and in Scenario B, you use this many, etc. If there is no rule, then I would like MQ to ask you at QM creation time "What is the max space do you want your Active logs to take?" and then just create the logs as big as possible. When it comes to logs, I think IBM gives to many options - the size of the logs, the # of logs, the types of logs - Arggghhhh! All without clear rules as to why you would want different choices.
Here is how much space I have available - Log me as best as possible and be done with it. If Circular logs had the abilty to restore damaged objects (really, why can't they?), there would be Z-E-R-O reason for all the headache of Linear archiving. As someone mentioned in another post, archiving of Linear logs should be in the base product at the very least.
The Persistent committed message is on the q file - good enough for me! Disk that holds the q file might get stomped on by Godzilla? Then write the q file data to another disk, sort of like double q files - oh wait, RAID does that already.
Now the boys at Hursley are a lot smarter than I, and have alot more time to think about this than I, so maybe I could be convinced otherwise given enough time with them, but I think hopsala got it right in that other post - logging in distributed MQ could be made better and a heck of a lot easier if it was being designed with today's technology in mind.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| hopsala |
| Posted: Tue Aug 30, 2005 6:05 am Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
| PeterPotkay wrote: |
| I think IBM gives to many options |
That's a well known IBMish disease, it pervades every IBM product i've ever known (WMQ, WAS, WBI, ICS, DB2, z/OS and associated products). Once upon a time (20 y ago) all programmers were hackers, and there were no standards, so this was relevant; now all programmers are users, and everything is by the book - thus it became redundant.
I would also love having a "Standard" qmanager installation, in which I don't have to worry about all this, but retain the old "custom" option.
| PeterPotkay wrote: |
| Hopsala, I read your quotes, but still don't see why a QM with 60 Primaries would be bad. |
Well, as you already know, we are generally in agreement here. But let's assume for a moment that someone does not have RAID or any other hardware recovery mechanism, and let's assume that he is an incumbent in a very large organization which is, like every large org I know, utterly paranoid about everything technological and has an organizational rule saying all projects must have at least one full backup.
(This is not a theoretical scenario, I know many such sites, especially banks which are usually 10 years behind technologically)
So, with that given state in mind logs become a rather important thing to manage. Also, if someone uses some real MQ throughput (not this measly 4GB/day, but i'm talking 100GB/day) disk size begins to matter.
Take a comp with 2 qmgrs sitting on the same disk, which can contain up to 1TB. Each will have 4 primary logs of 100GB each (that is 4 days, I believe fjb_saper is optimistic with his 26hours estimate) and 2 secondaries.
As you can see, it is impossible for both to extend to an active pool of 6 primary, coz there's no room. Secondaries should only be used as a last resort, and you don't have to plan for a last resort in Both qm's, since it's not likely to happen.
And another point: If log file size is too big, and one log file gets screwed (used to happen in older mq versions, doesn't really happen nowadays I think), you lose a lot of info; if it is too small, trying to trace activity in logs becomes a pain.
It all depends on how large a HD he has there, and what sort of hardware recovery. To sum things up, I agree that all this hoopla around "how to manage log files" is, 99 times out of a 100, more effort than it's worth.
Oh, and
| PeterPotkay wrote: |
| As someone mentioned in another post, archiving of Linear logs should be in the base product at the very least. |
That was me, that someone |
|
| Back to top |
|
|
| hopsala |
| Posted: Tue Aug 30, 2005 6:06 am Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
| fjb_saper wrote: |
| Secondaries are created and used and deleted as needed. |
As you can see by my quote from the manual a few posts ago, that's not what the manual says. Is this true for linear logs? If a secondary is used, and then is made redundant, is it removed from the active pool? (have you tried this?) |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Aug 30, 2005 7:23 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20697
Location: LI,NY
|
| hopsala wrote: |
| fjb_saper wrote: |
| Secondaries are created and used and deleted as needed. |
As you can see by my quote from the manual a few posts ago, that's not what the manual says. Is this true for linear logs? If a secondary is used, and then is made redundant, is it removed from the active pool? (have you tried this?) |
This was more specific for circular logging. You can see the same effects if you run rcdmqimg in linear logging.
| hopsala wrote: |
| I believe fjb_saper is optimistic with his 26hours estimate) and 2 secondaries |
Sorry for not being clear enough. We are talking about 2 primaries and 50 secondaries here. (50 to give you some room should they all fill up. (60 max)).
The main advantage is if you have to have more than 1 qmgr live in the space you have a better view on real space utilization.
We run 2 qmgrs with over 30 Gig of log space each (they are on different hosts) and archive easily from 30 to 100 64 MB files a day (linear logging)
This started with an 8 MB log space and an unknown traffic volume (huge messages flowing through).
Well with 8 MB you will not survive the night with the type of traffic we had. I stayed up one night to keep the qmgr alive while running with 12 MB and barely kept the qmgr alive and going archiving more than 200 64 MB files that night...(got more disk space the next morning).
With 30 Gig 2 primaries and 50 secondaries I check the volume used in log space (< 20 %) and know that I can sleep in peace. If I had 50 primaries and 12 secondaries by the time a volume problem would hit it would be too late to react....
Just my 2 cents...
As for the 26 hours... I noticed that every 24 hours a check point gets written. The next checkpoint being at 10000 msgs does not seem to make any difference to ms0l (java linear log archiving utility). Yes restart log and image log start may vary (check in qmgr error log) but the archiving utility still thinks its the ones from right around midnight.... We record the image twice a day..., once just before midnight.
So you need enough log space to hold the logs until you can archive them.
Hope this clarifies a bit where I come from... |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
