| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| LOG size (again) thoughts |
« View previous topic :: View next topic » |
| Author |
Message
|
| sebastia |
 Posted: Mon Mar 17, 2008 4:00 am Post subject: LOG size (again) thoughts |
 |
|
Grand Master
Joined: 07 Oct 2004
Posts: 1003
|
Hello everybody.
Today, we have some doubts (again) with the LOG size.
We have lots of message producers (lets say 100).
Message rate shall be 500 messages / second (yes, per second),
and message size shall be 4 KB.
We want to be able to recover 24 hours, so the simple operation is
500 msg/seg*86400 seg/day*4KB/msg = 172,8 GB for LOG
Jeff - I still remember what you said 30 Oct 2007 :
As long as the messages are PUT and GET within individual units of work - not all 3,000 in one transaction - you can configure it to only have enough active log space for one message.
Yes, our producers do commit their messages one by one.
And the consumers are WAS processes that manage the units of work, and manage messages in bunches of 32 or 16 or 8 messages.
So, in today's case, if I did understand it properly, all the space we need is for the "uncommitted" messages, that shall be
(*) on producer side : 100 producers having one message pending to commit
(*) on consumer side : the number of WAS processes (lets say 10) multiplied by the number of grouped message they handle (lets say 32)
So, the max number of uncommitted message at any moment shall be
100 + 32 * 10 = 420
SO, WE NEED LOG SPACE FOR 420 MESSAGES ONLY,
this is 420 msg * 4KB/msg = 1680 KB
Am I right ?
[/u] |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Mon Mar 17, 2008 4:07 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Well. Yes, but.
You need to keep in mind channel traffic.
And, generally, you're better off having more log space than less.
Disk space is cheaper than downtime.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| sebastia |
| Posted: Mon Mar 17, 2008 4:16 am Post subject: |
|
|
Grand Master
Joined: 07 Oct 2004
Posts: 1003
|
Yes, that was 16 Oct 2007 -
The prevailing wisdom these days is that disk space is cheaper than downtime, so ...
But I have to give the customer a figure, a calculation, a number
Shall I use a 200 % protection factor ( for channel traffic ) ?
The channels are ... a cluster, so quite invisible to me ...
Or let me ask a reverse question :
=======================
IF WE USE A 100 GB LOG file, is there any penalty for the size of the log ?
I mean, is a 100 MB log faster to manage than a 200 GB log
( if the same amount of message are (un) commited at a given moment )
???
((( I am able to say that I am learning to ask more clever questions than a year ago ... ))) |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Mar 17, 2008 4:21 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| sebastia wrote: |
Or let me ask a reverse question :
=======================
IF WE USE A 100 GB LOG file, is there any penalty for the size of the log ?
I mean, is a 100 MB log faster to manage than a 200 GB log |
IMHO if you notice the difference in the 2 logs you need better firmware in the disc units.
It's also faster to have a 200Gb log running without intervention than trying to fix a 100Mb log that's run out of space.
The trick is to have log files & queue files on separate devices (which they should be for recovery purposes of course) that run through separate controllers. But (again IMHO) if you're so hung up on performance you're tracking disc i/o to this level you'll need to do tuning elsewhere as disc is unlikely to be a serious bottleneck.
| sebastia wrote: |
| ((( I am able to say that I am learning to ask more clever questions than a year ago ... ))) |
Probably a good definition of experience and expertise! 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| sebastia |
| Posted: Mon Mar 17, 2008 4:32 am Post subject: |
|
|
Grand Master
Joined: 07 Oct 2004
Posts: 1003
|
Thanks, Vitor, for your ideas
Yes, we shall have one disk with Data
and on a different physical disk there will be the LOG.
In this way, they can write in parallel
We did create the "final" file system on the AIX machine
( /var/mqm/qmgrs/QMN/queues/ and /var/mqm/log/ )
with a separate device for the queues and for the LOG,
and then "created" the queue manager.
The fact that MQ allocates the disk space before using it
suggests to me it uses pointers to write directly into it
so the LOG size shall be not VERY important
from the performance point of view. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Mar 17, 2008 5:04 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Mar 17, 2008 5:23 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| PeterPotkay wrote: |
| sebastia wrote: |
IF WE USE A 100 GB LOG file, is there any penalty for the size of the log ? |
The biggest a log file can be on Windows or Unix is 256MB (4096 x 65535).
You want your individual log files as big as possible.
|
I'd assumed in the earlier posts that the 100Gb of log was a total value, not a single log file.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| sebastia |
| Posted: Mon Mar 17, 2008 5:37 am Post subject: |
|
|
Grand Master
Joined: 07 Oct 2004
Posts: 1003
|
| If max is 256 MB, then 100 GB can only be reached by "-lp", of course |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
