| Author |
Message
|
| pnimith |
 Posted: Wed Oct 05, 2005 10:46 am Post subject: logging for really large messages with MQ 5.3 |
 |
|
Novice
Joined: 10 May 2005
Posts: 23
Location: ny, usa
|
hi all
i have MQ 5.3/CSD 7 on Solaris.
we have an anticipated volume of around 9000 messages/day, each upto 100MB in size. all these need to be persistent and we are using linear logging.
the calculation of log space needed comes to 900GB plus change! (am i wrong here?)
since 5.3 has max log space limited to 4.5 GB, both log design and management will be a tough job.
could anyone throw some light on different approaches i should consider?
TIA |
|
| Back to top |
|
 |
| wschutz |
| Posted: Wed Oct 05, 2005 11:01 am Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
Quick ,get MQ V6:
| Quote: |
| There has also been a requirement to permit larger and/or more logfiles – in this release, the active log size increases from 4GB to 128GB on Unix, 64GB on Windows |
-- Whats new in MQ V6 prez
_________________
-wayne |
|
| Back to top |
|
|
| pnimith |
| Posted: Wed Oct 05, 2005 11:10 am Post subject: |
|
|
Novice
Joined: 10 May 2005
Posts: 23
Location: ny, usa
|
thanx wschutz,
that would definitely be a good solution. but, if i had to, i would have tough time convincing PM.
i really do want to find a workable solution with MQ 5.3. i hope to find it here! |
|
| Back to top |
|
|
| hopsala |
| Posted: Wed Oct 05, 2005 11:12 am Post subject: Re: logging for really large messages with MQ 5.3 |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
| pnimith wrote: |
we have an anticipated volume of around 9000 messages/day, each upto 100MB in size. all these need to be persistent and we are using linear logging.
the calculation of log space needed comes to 900GB plus change! (am i wrong here?) |
You don't have to have logs that contain a day's worth of activity, just the log activity since last checkpoint. From the manuals:
| Quote: |
During normal running, checkpoints are taken either every 30 minutes provided there are at least 100 log records,
but also driven when 10000 log records have been written (V5.3). |
Every such checkpoint is a point of consistency, meaning all your queue files, buffers and logs maintain the same state. So if you just need regular backup, last checkpoint is all you need, if you need backup in case queue files are completely lost, that's something else altogther. Another considering is if you have RAID (or some other imaging backup) or not.
The issue of "how many logs" has a fairly clear explanation in the manual (system admin guide) and has been discussed here many times, so use the search button. For example Calculation for sizing??. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Oct 05, 2005 12:53 pm Post subject: Re: logging for really large messages with MQ 5.3 |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
| hopsala wrote: |
| pnimith wrote: |
we have an anticipated volume of around 9000 messages/day, each upto 100MB in size. all these need to be persistent and we are using linear logging.
the calculation of log space needed comes to 900GB plus change! (am i wrong here?) |
You don't have to have logs that contain a day's worth of activity, just the log activity since last checkpoint. From the manuals:
| Quote: |
During normal running, checkpoints are taken either every 30 minutes provided there are at least 100 log records,
but also driven when 10000 log records have been written (V5.3). |
Every such checkpoint is a point of consistency, meaning all your queue files, buffers and logs maintain the same state. So if you just need regular backup, last checkpoint is all you need, if you need backup in case queue files are completely lost, that's something else altogther. Another considering is if you have RAID (or some other imaging backup) or not.
The issue of "how many logs" has a fairly clear explanation in the manual (system admin guide) and has been discussed here many times, so use the search button. For example Calculation for sizing??. |
Sure but with 9000 messages per day he does not get to 10000 log records per day ? Right ? So even though he would never have the space used by active logs it may still be needed for automatic log management.
Enjoy  |
|
| Back to top |
|
|
| hopsala |
| Posted: Wed Oct 05, 2005 1:00 pm Post subject: Re: logging for really large messages with MQ 5.3 |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
| fjb_saper wrote: |
| Sure but with 9000 messages per day he does not get to 10000 log records per day ? Right ? So even though he would never have the space used by active logs it may still be needed for automatic log management. |
First of all, 9000 messages usually means ~18000 transactions, the MQPUT of the channel being one (or according to HBINT) and the MQGET being the second; but yes, of course there is a good chance they won't get up to 10000 in one day. However, to re-state:
| hopsala wrote: |
During normal running, checkpoints are taken either every 30 minutes provided there are at least 100 log records,
but also driven when 10000 log records have been written (V5.3). |
every 30 minutes... |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Oct 05, 2005 1:14 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
| hopsala wrote: |
During normal running, checkpoints are taken either every 30 minutes provided there are at least 100 log records,
but also driven when 10000 log records have been written (V5.3). |
Sorry but this is definitely not what I see happening on one of my servers running on Solaris at 5.3 CSD08...
We have easyly over 50,000 messages going through in a period of 24 hours. I run rcdmqimg twice a day. And despite all this the support pack MS0L (java) wants to only archive once a day...
What is confusing the picture I fear are long living transactions because of a Transaction Manager that is not working to expected specs ... but we are working on that...(XA Problems)
Enjoy  |
|
| Back to top |
|
|
| pnimith |
| Posted: Wed Oct 05, 2005 1:24 pm Post subject: |
|
|
Novice
Joined: 10 May 2005
Posts: 23
Location: ny, usa
|
correct me if/where wrong:
if MQ automatically checkpoints every 30 minutes and if i had a cron job set up to service the logs every 30 minutes (archiving outdated logs to external media), MQ will function without any noticeable degradation in performance with available disk space of 100-150 GB.............. 
thanx |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Oct 05, 2005 1:40 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
| pnimith wrote: |
correct me if/where wrong:
if MQ automatically checkpoints every 30 minutes and if i had a cron job set up to service the logs every 30 minutes (archiving outdated logs to external media), MQ will function without any noticeable degradation in performance with available disk space of 100-150 GB..............
thanx |
Did you read my post ? I wish... I run the archiving of the logs every hour. Most of the time nothing gets archived... even though in the qmgr logs the information is not the same as the one returned by the support pack about the earliest needed log...
You can safely assume that you want to run rcdmqimg and the archiving at least once a day and that the logs will be archived. Anything beyond is up for grabs and may depend on the support pack used for archiving...
 |
|
| Back to top |
|
|
| hopsala |
| Posted: Mon Oct 10, 2005 6:51 pm Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
| fjb_saper wrote: |
| hopsala wrote: |
During normal running, checkpoints are taken either every 30 minutes provided there are at least 100 log records,
but also driven when 10000 log records have been written (V5.3). |
Sorry but this is definitely not what I see happening on one of my servers running on Solaris at 5.3 CSD08...
We have easyly over 50,000 messages going through in a period of 24 hours. I run rcdmqimg twice a day. And despite all this the support pack MS0L (java) wants to only archive once a day...
What is confusing the picture I fear are long living transactions because of a Transaction Manager that is not working to expected specs ... but we are working on that...(XA Problems) |
(Sorry for the delay, this was somehow lost in my "last posts" lists)
How do you explain the discrepancy between the documents and viewed behavior then? Does this happen in other QMs aswell?
Since I have witnessed 30min checkpoint behavior to be generally true in the past, I believe the problem with your case is either the XA manager (that may somehow change normal checkpoint behavior) or the fact you have long running transactions; doubtless a long-running transaction will cause the QM to issue checkpoints only relevant up to the beginning of the transaction, thus altering archiving behavior completely.
(Another possiblity would be the use of MS0L, maybe it's simply a bug - though I doubt it)
Has anyone else come across what fjb_saper sais? Are 30min checkpoints a myth? |
|
| Back to top |
|
|
| Nigelg |
| Posted: Tue Oct 11, 2005 1:12 am Post subject: |
|
|
Grand Master
Joined: 02 Aug 2004
Posts: 1046
|
30 min checkpoints are not a myth, but that is only the default behaviour.
A checkpoint will occur if any of the below conditions are met:
At 30 minute intervals, if a minimum number of logged operations have occurred. Note that the interval and the minimum number are configurable, but the parameters are not published (CBNP).
After 10000 logged operations. This number is also CBNP.
At qmgr shutdown.
If the log is getting full (80%, also CBNP) then a checkpoint is run to see if it frees log space.
_________________
MQSeries.net helps those who help themselves.. |
|
| Back to top |
|
|
| markt |
| Posted: Tue Oct 11, 2005 1:41 am Post subject: |
|
|
Knight
Joined: 14 May 2002
Posts: 503
|
Checkpoints don't really have anything to do with the amount of log space required. Checkpoints help with recovery time in the event of abends, and a number of other things get done at checkpoints, but they do not materially affect the required ACTIVE log space. The 4GB (or the increased size for WMQ V6) max space is the maximum ACTIVE size - ie the amount of work done since the oldest uncommitted transaction. Hence the need to avoid long-running units of work.
The size of logs used for media recovery can be much larger than the ACTIVE maximum.
(On z/OS different techniques are used, and checkpoints can shunt active transactions up in the log, freeing older space.) |
|
| Back to top |
|
|
| hopsala |
| Posted: Sat Oct 15, 2005 9:34 am Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
Apologies, but either I'm not following you, or there's a discrepancy here:
| markt wrote: |
| Checkpoints don't really have anything to do with the amount of log space required. Checkpoints help with recovery time in the event of abends, and a number of other things get done at checkpoints, but they do not materially affect the required ACTIVE log space. |
If checkpoints affect "recovery time" then they affect "needed recovery space" - when a message is written to queue, and there was no buffer flush or checkpoint, that message must be replayed from the log onto the queue files upon restart (if qm crashed). Now, as the number of such messages increases, you need more and more active space; naturally the active space you had defined remains constant, but when That space fills and still no checkpoint - you go to secondary space, and then on to problems.
Example: 4MB logs, write 2 messages 3MB each - being less than 10000 records, and less than 30 min - what then?
If you claim that a checkpoint is issued whenever this happens, maybe so, but this is undocumented; plus there are important performance reprocussions - the second message MQCMIT has to wait for the checkpoint to finish, so if this behavior continues you'll have a qm with the performance graph of a yoyo: fast-slow-fast-slow...
(p.s due to the 64KB buffer size of queues in Dist-systems, you'll have to improvise - 100 queues, and 1 20KB msg per queue or similar. In z/OS buffer size is configurable, so it's easier)
| markt wrote: |
| ACTIVE size- ie the amount of work done since the oldest uncommitted transaction. Hence the need to avoid long-running units of work. |
Here we are in agreement, but in addition to what I stated earlier - active log is the amout of work since olderst uncommited msg + since first commited msg not written to queue files (checkpoint or buffer flush).
| markt wrote: |
| The size of logs used for media recovery can be much larger than the ACTIVE maximum. |
Again I agree - this discussion encompasses restart "recovery" only (normal and in case of QM crash), disregarding media recovery in case of queue damage etc.
So, since you seem rather proficient concerning this mechanism, i'll be glad for an explanation. |
|
| Back to top |
|
|
|
|
