| Author |
Message
|
| Sam Uppu |
 Posted: Mon Jun 01, 2009 8:12 am Post subject: file system allocation for MQ |
 |
|
Yatiri
Joined: 11 Nov 2008
Posts: 610
|
Hi Guys,
OS: AIX and MQ vesrion 7
I am trying to allocate MQ file systems and install on this server which is going to handle larger messages of size 100 meg.
I am planning with file system allocation as below:
/usr/mqm of size 1 gig for installables
/var/mqm of size 1 gig
/var/mqm/errors - 1gig
/var/mqm/log - 8 gig ( using circular logging and going with 20 primary log files each of 256 meg, 65000 pagesx1024)
Question:
I am under the impression that MQ will store the messages in its queue buffers upto 128 K(64 bit QMgr) and will move the messages to /var/mqm/log when bigger than 128 K. I am looking into the Queue buffer sizes of this link.
Correct me if I am wrong.
All I am looking is in a DR case if the app goes down and not pulling the msgs off of the queues, my queue manager should not go down with filling up of the disk space.
Your thoughts and suggestions are appreciated.
Thanks. |
|
| Back to top |
|
 |
| gbaddeley |
| Posted: Mon Jun 01, 2009 4:02 pm Post subject: Re: file system allocation for MQ |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| Sam Uppu wrote: |
...I am trying to allocate MQ file systems and install on this server which is going to handle larger messages of size 100 meg.
I am planning with file system allocation as below:
..
/var/mqm of size 1 gig |
How many large messages do you want to have queued at any given time? This will only allow 9 messages! Disk is cheap. Why not go for something much larger, say 100 gig?
| Quote: |
| /var/mqm/errors - 1gig |
Why? Are you expecting a huge number unexpected failures to produce huge FFST / FDC files?
| Quote: |
| /var/mqm/log - 8 gig ( using circular logging and going with 20 primary log files each of 256 meg, 65000 pagesx1024) |
At 100 meg per msg, this will allow active UOW's to contain up to ~64 uncommitted messages. That's not very many! The log file system should be at least as large as the /var/mqm file system containing the qmgr directory.
_________________
Glenn |
|
| Back to top |
|
|
| vol |
| Posted: Mon Jun 01, 2009 10:42 pm Post subject: |
|
|
Acolyte
Joined: 01 Feb 2009
Posts: 69
|
| Quote: |
| I am under the impression that MQ will store the messages in its queue buffers upto 128 K(64 bit QMgr) and will move the messages to /var/mqm/log when bigger than 128 K. |
Where did you get that weird idea?
The 128K may refer to an in-memory buffer.
All msgs are stored (unreliably) in the q files in individual queue dirs.
Persistent msgs are stored (reliably) in the log files. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jun 01, 2009 11:49 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| vol wrote: |
| All msgs are stored (unreliably) in the q files in individual queue dirs. |
I think what you meant was that some classes of message may not be stored in the q files. Those which are, are always stored reliably. There is no instance where a message is stored in a q file and cannot be retrieved because the file's unreliable.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| exerk |
| Posted: Tue Jun 02, 2009 12:07 am Post subject: Re: file system allocation for MQ |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| gbaddeley wrote: |
| Sam Uppu wrote: |
| .../var/mqm/log - 8 gig ( using circular logging and going with 20 primary log files each of 256 meg, 65000 pagesx1024) |
At 100 meg per msg, this will allow active UOW's to contain up to ~64 uncommitted messages. That's not very many! The log file system should be at least as large as the /var/mqm file system containing the qmgr directory. |
And I would be very concerned at any application putting/getting 100MB messages that allowed more than a few such messages uncommitted.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| Sam Uppu |
| Posted: Tue Jun 02, 2009 5:32 am Post subject: |
|
|
Yatiri
Joined: 11 Nov 2008
Posts: 610
|
| vol wrote: |
| Quote: |
| I am under the impression that MQ will store the messages in its queue buffers upto 128 K(64 bit QMgr) and will move the messages to /var/mqm/log when bigger than 128 K. |
Where did you get that weird idea?
The 128K may refer to an in-memory buffer.
All msgs are stored (unreliably) in the q files in individual queue dirs.
Persistent msgs are stored (reliably) in the log files. |
We are using persistent messages. All the persistent messages will be stored in the logs.
In which cases the msgs will stay in the q files in individual queue dirs?. For non persistent msgs only or/ and also for persistent?.
Thanks. |
|
| Back to top |
|
|
| Sam Uppu |
| Posted: Tue Jun 02, 2009 5:35 am Post subject: Re: file system allocation for MQ |
|
|
Yatiri
Joined: 11 Nov 2008
Posts: 610
|
| gbaddeley wrote: |
| Sam Uppu wrote: |
...I am trying to allocate MQ file systems and install on this server which is going to handle larger messages of size 100 meg.
I am planning with file system allocation as below:
..
/var/mqm of size 1 gig |
How many large messages do you want to have queued at any given time? This will only allow 9 messages! Disk is cheap. Why not go for something much larger, say 100 gig?
| Quote: |
| /var/mqm/errors - 1gig |
Why? Are you expecting a huge number unexpected failures to produce huge FFST / FDC files?
| Quote: |
| /var/mqm/log - 8 gig ( using circular logging and going with 20 primary log files each of 256 meg, 65000 pagesx1024) |
At 100 meg per msg, this will allow active UOW's to contain up to ~64 uncommitted messages. That's not very many! The log file system should be at least as large as the /var/mqm file system containing the qmgr directory. |
You are right. I should go with larger disk size. As we are dealing with persistent msgs, just /var/mqm/log needs to be increased or / and also the /var/mqm as well?.
To be on the safer side, I wanted to use 1 gig for /var/mqm/errors.
Let me know your thoughts.
Thanks. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Jun 02, 2009 3:26 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
You look like way undersized in disk space. Disk space is cheap. Having an outage because you did not allocate enough is going to cost you more...
Go with 8 GB for the queues and 16 GB for the logs...That should keep you for a while unless you do a lot of very big messages... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Tue Jun 02, 2009 4:12 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| vol wrote: |
| Quote: |
| I am under the impression that MQ will store the messages in its queue buffers upto 128 K(64 bit QMgr) and will move the messages to /var/mqm/log when bigger than 128 K. |
Where did you get that weird idea?
The 128K may refer to an in-memory buffer.
All msgs are stored (unreliably) in the q files in individual queue dirs.
Persistent msgs are stored (reliably) in the log files. |
It looks like he got that idea from the developerWorks article linked above :
| Quote: |
Queue buffer sizes
Each queue in the queue manager is assigned two buffers to hold messages, one for non-persistent messages and one for persistent messages. After messages spill out of the buffer, they move to the operating system file system. You can change the buffer sizes to limit this overspill so that data is more readily available to the queue manager.
The buffer for the non-persistent messages has a default size of 64K for the 32-bit queue managers (Windows, Linux32) and 128K for 64-bit Queue Managers (AIX, Solaris, HPUX, Linux64). The buffer for persistent messages has a default size of 128K for 32-bit Queue Managers and 256K for 64-bit Queue Managers. The maximum size supported for both queue buffers is 100MB.
The impact of changing the buffer sizes is to increase storage requirements for the queue manager. The increase in storage depends on the size of the buffers and the number of queues to which it applies... |
Persistent messages are actually stored on both the q files and the log files.
Non-persistent messages may not actually make it out onto the q files, and they are never written to the logs.
_________________
Glenn |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue Jun 02, 2009 6:53 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Regarding the idea that persistent messages are stored in circular logs:
Create a QM with only 100 MB total for primary and secondary logs. Now go ahead and MQPUT (no syncpoint) 5 GB worth of persistentr messages to a queue on that QM.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| AkankshA |
| Posted: Tue Jun 02, 2009 7:56 pm Post subject: |
|
|
Grand Master
Joined: 12 Jan 2006
Posts: 1494
Location: Singapore
|
| PeterPotkay wrote: |
Regarding the idea that persistent messages are stored in circular logs:
Create a QM with only 100 MB total for primary and secondary logs. Now go ahead and MQPUT (no syncpoint) 5 GB worth of persistentr messages to a queue on that QM. |
wont that fail and create errors... increase CPU utilization etc ??
I hv faced "not enough log space available" error in system logs (windows) for MQ queue manager having circular logs while deploying heavy (real heavy) bar files thru WMB....
When i increased the logs it worked fine...
_________________
Cheers |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue Jun 02, 2009 9:04 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
No, it won't fail.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| Sam Uppu |
| Posted: Wed Jun 03, 2009 7:10 am Post subject: |
|
|
Yatiri
Joined: 11 Nov 2008
Posts: 610
|
| PeterPotkay wrote: |
Regarding the idea that persistent messages are stored in circular logs:
Create a QM with only 100 MB total for primary and secondary logs. Now go ahead and MQPUT (no syncpoint) 5 GB worth of persistentr messages to a queue on that QM. |
Peter,
As per the discussions above from you and others, I see that both for persistent and non persistent msgs, the msg wont log to log files under /var/mqm/log (unix)...then why we are using the log files at all?.
I all the time thought, log files are used for logging persistent msgs for recovery. Does this logging technique applies only to linear logging for recovery and not at all useful for circular logging?.
Thanks. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jun 03, 2009 8:13 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Circular logs will log UOWs for persistent messages. If you have 100MB of logs, you can put 75GB of persistent messages onto the queues as long as you commit often enough to not use up 100 MB of space. And you have 75GB of space allocated for the queues.
If you put all the messages under syncpoint and don't MQCMIT, you will fail as soon as you try to put more than 100 MB of messages (slightly less due to other overhead, but you get the point).
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jun 03, 2009 8:20 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| PeterPotkay wrote: |
| Circular logs will log UOWs for persistent messages. If you have 100MB of logs, you can put 75GB of persistent messages onto the queues as long as you commit often enough to not use up 100 MB of space. And you have 75GB of space allocated for the queues. |
As long as *everyone* using transactions at the same time commits often enough to not use up 100MB of log space simultaneously. As a simple example, suppose you satisfy all of the above requirements and are successful. Then you switch from writing to a QLOCAL to writing to a QREMOTE/QCLUSTER. Now the channel starts a transaction against the messages in the XMITQ.... |
|
| Back to top |
|
|
|
|
