| Author |
Message
|
| rxm8778 |
 Posted: Tue Mar 24, 2009 7:11 am Post subject: How big can the MQ log file system can be? |
 |
|
Apprentice
Joined: 15 Sep 2005
Posts: 37
|
Info that may help understand my question:
platform: AIX
MQ version: 6
LogType: circular
Now that we have this info out the way, here is what I am pondering upon:
I was always under the impression that the more free disk space you had laying around, the more disk space you were able to throw at the filesystem of the MQ Logfile /var/mqm/log. For example, whether you allocate 2GB to your log file system or 10GB to your log filesystem, the entire storage space would be used as needed. But was my assumption really correct???
If there is a hard limit as far as the number of primary and secondary log file you can define (according to the MQ manual total primary + total secondary cannot be more than 511) and there is a limit as far as the maximum LogFilePages you can define (according to the MQ manual max LogFilePages = 65 535) THEREFORE there is a hard limit on the amount of filesystem space that will be used by the logs.
Am I correct here? or am I missing something.  |
|
| Back to top |
|
 |
| Vitor |
| Posted: Tue Mar 24, 2009 7:44 am Post subject: Re: How big can the MQ log file system can be? |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| rxm8778 wrote: |
| there is a hard limit on the amount of filesystem space that will be used by the logs. |
There's a limit (by platform) of how many log files the queue manager can use as you say.
IMHO if you've so many uncommitted transactions you hit this limit (given that the queue manager will reclaim the space on commit or shortly after) then your design needs work!
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Mar 24, 2009 7:52 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
There is no limit on how much space on the file system the log files can consume.
It is also true that there is a limit on how much space on the file system the log files can consume.
Which one is true of your particular queue manager depends entirely on how you have configured your queue manager.
Oddly enough, there's quite a lot of information on exactly this subject in the System Administration manual in the MQ Info Center.
   |
|
| Back to top |
|
|
| rxm8778 |
| Posted: Tue Mar 24, 2009 10:50 am Post subject: |
|
|
Apprentice
Joined: 15 Sep 2005
Posts: 37
|
@Vitor I agree with you that if that limit is being hit than the applications are probably designed poorly. But I am just a little curious about what that limit is:
According to the admin manual:
log file pages is 1024, gives a log file size of 4 MB
therefore if I set my log file pages to 65 535 the log file size should be 256MB right?
and if the total amount of primary + secondary log file is 511 then the maximum amount of space I would need to store all the log files at once is (in the worse case scenario where the log kept growing to that MAX number):
511 X 256MB = 130,816MB => 127GB roughly
So a 127GB is about the maximum amount of space the MQ log could ever use if everything is set to MAX? Is that true? Again I am just making sure my understanding of the max number of the log are correct.
@MQJeff I know there is alot info on log sizing in the MQ manual and I have been reading it. I am merely ensuring my interpretation of it is correct.
thanks for your help... |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Mar 24, 2009 12:55 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| rxm8778 wrote: |
| I agree with you that if that limit is being hit than the applications are probably designed poorly. |
Poorly? 100+ Gb of uncommited transactions? That's not poorly; that's bad on a biblical scale.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| exerk |
| Posted: Wed Mar 25, 2009 12:21 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| rxm8778 wrote: |
| ...So a 127GB is about the maximum amount of space the MQ log could ever use if everything is set to MAX? Is that true?... |
Yes, but only for circular logging.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| rxm8778 |
| Posted: Wed Mar 25, 2009 9:24 am Post subject: |
|
|
Apprentice
Joined: 15 Sep 2005
Posts: 37
|
In my environment, I have the following settings on one of my queue managers:
Log:
LogPrimaryFiles=260
LogSecondaryFiles=250
LogFilePages=16384
LogType=CIRCULAR
LogBufferPages=4096
LogPath=/var/mqm/log/XXXXXX/
LogWriteIntegrity=TripleWrite
According to my calculations:
LogFilePages is set to 16384
therefore each logfile have a size of 16,384 X 4KB = 65,536KB
the total space used by all the log files (if they were all full) is:
65,536KB X 511 = 33,488,896KB
which is equal to 31.9375GB roughly 32GB
The filesystem for /var/mqm/log/XXXXXX/ is set to 6GB (it should have probably been set to 32GB).
I have been consistently getting the following error on that queue manager: AMQ7469
But even with the filesystem being only 6GB, I have been monitoring it and it never goes higher than 30% full (1.8GB). So that mean to me that I am no where close to running out of space.
What could be causing those AMQ7469 errors?  |
|
| Back to top |
|
|
| JosephGramig |
| Posted: Wed Mar 25, 2009 11:03 am Post subject: |
|
|
Grand Master
Joined: 09 Feb 2006
Posts: 1244
Location: Gold Coast of Florida, USA
|
Could be a lot of things, most likely an application not committing frequently enough.
Also consider this, you probably have monitoring software that will monitor the file system for your logs (which should be separate from everything else). It will probably give you a warning at something reasonable like 70% full, so target just under that for your primary logs. When you get the file space warning, then you know it has allocated secondary logs and you should start thinking about capacity.
Are you at 6.0.2.6? If not, apply that and retest.
Also, if you use SAN for the logs, you can switch to SingleWrite. Of course, with Linear logs, it is always safe to use single write. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Mar 25, 2009 11:16 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| rxm8778 wrote: |
| What could be causing those AMQ7469 errors? |
- You're on an old version of WMQ and there's a bug? The queue manager is hitting the maximum number of uncommitted transactions & responding as if the log is full?
- There's not as much space as you think? Something else is soaking space on that file system?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| rxm8778 |
| Posted: Wed Mar 25, 2009 11:58 am Post subject: |
|
|
Apprentice
Joined: 15 Sep 2005
Posts: 37
|
Well... I think I am getting warmer:
I looked into /var/mqm/log/XXXXXXX/active
and found the following:
....
-rw-rw---- 1 mqm mqm 4202496 Mar 25 11:58 S0000380.LOG
-rw-rw---- 1 mqm mqm 4202496 Mar 25 11:59 S0000381.LOG
-rw-rw---- 1 mqm mqm 4202496 Mar 25 11:59 S0000382.LOG
-rw-rw---- 1 mqm mqm 4202496 Mar 25 11:59 S0000383.LOG
Now back to my calculations: 
4202496B is the size of each of the logs. (4104KB)
Since the manual says: The log file size is specified in units of 4 KB pages
therefore this reveals that the LogFilePages = 4104KB / 4KB = 1026. (not quite 1024 as I would have expected which is the default setting for this property when a Queue Manager is created) 
But since the queue manager is currently showing
LogFilePages=16384 it seems like someone might have changed this number without recreating the queue manager thinking it would have taken effect. And also set the file system to 6GB but in reality this amount of filesystem space will never be used entirely. 
So my log might still be getting full but my filesystem monitoring tool would not catch it by monitoring the filesystem for 100% full. Maybe monitoring it for 30% full might be more indicative of the log file filling up.
======
Now that I have this log size/file system size question cleared up, this is only solving part of the issue.
Now I need to investigate which application is the culprit.... This is like looking for a needle in a haystack since that queue manager is servicing 100's of applications a day.
Any idea how to tackled this and isolate the culprit??? 
FYI
Name: WebSphere MQ
Version: 6.0.2.3
CMVC level: p600-203-080123
BuildType: IKAP - (Production)
Last edited by rxm8778 on Wed Mar 25, 2009 3:07 pm; edited 2 times in total |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Mar 25, 2009 12:14 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| rxm8778 wrote: |
Any idea how to tackled this and isolate the culprit???
|
Try DIS CONN & look at the UOW stats.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| JosephGramig |
| Posted: Thu Mar 26, 2009 4:16 am Post subject: |
|
|
Grand Master
Joined: 09 Feb 2006
Posts: 1244
Location: Gold Coast of Florida, USA
|
You are most likely quite correct that somebody changed the LogFilePages and did not know it would not take effect.
Your next step is to save the QMGR and any data and do an amqoamd -s. Delete and recreate with the correct LogFilePages. Remember to remove this QMGR from all your clusters first.
6023 is a good level but 6026 is better and resolve a minor security exposure on UNIX systems. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Mar 26, 2009 4:30 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| JosephGramig wrote: |
| Your next step is to save the QMGR and any data and do an amqoamd -s. |
Or you could skip a step and just use --saveSecurity, --saveSecurityNoHeader, -z or -Z with saveqmgr... |
|
| Back to top |
|
|
| rxm8778 |
| Posted: Thu Mar 26, 2009 5:27 am Post subject: |
|
|
Apprentice
Joined: 15 Sep 2005
Posts: 37
|
I will definitely suggest that the Queue Manager gets re-created if they want to truly take advantage of the space they have laying around. But this is more of a long term solution that may not get implemented this soon. My true focus now is to identify the cause of AMQ7469. Meaning who caused it?
| Vitor wrote: |
Try DIS CONN & look at the UOW stats. |
runmqsc shows me a snapshot of live data. So I am not sure running this command will be much help to go back in time and troubleshoot why I was getting AMQ7469 on a particular day.
Unless I am missing something, I don't think this command will help me troubleshoot unless the issue re-occurs at the exact moment I am running this command. right?
Is that my only option?
| JosephGramig wrote: |
| 6023 is a good level but 6026 is better and resolve a minor security exposure on UNIX systems. |
What are these numbers? Is that something related to amqoamd?
Last edited by rxm8778 on Thu Mar 26, 2009 5:36 am; edited 1 time in total |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Mar 26, 2009 5:35 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| rxm8778 wrote: |
| Unless I am missing something, I don't think this command will help me troubleshoot unless the issue re-occurs at the exact moment I am running this command. right? |
No, you've got it
| rxm8778 wrote: |
| Is that my only option? |
Pretty much. Once you start getting the errors, the culprit will start getting failures from the qmgr and probably restart it's UOW & the qmgr will destroy a lot of evidence by rolling the log back.
The only other option is to canvas the application and see who got a weird abend from WMQ at the time in question.
| rxm8778 wrote: |
| JosephGramig wrote: |
| 6023 is a good level but 6026 is better and resolve a minor security exposure on UNIX systems. |
What are these numbers? Is that something related to amqoamd? |
They're abbreviations of build levels - WMQv6.0.2.3 & WMQv6.0.2.6
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
