| Author |
Message
|
| career |
 Posted: Tue Feb 19, 2008 3:23 pm Post subject: MQ persistence/ writtng to logs |
 |
|
Apprentice
Joined: 09 Jul 2007
Posts: 36
|
Hi all,
i have this question and ans are much appreciated.
If my program has 10 messages to be commited to queue inside unit of work then
are these 10 messages written to log (assuming that messages are to be made persistent) individually or can they be written at a time?
is there any thing that MQ provides to write the 10 messages at a time instead of one at a time? |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Tue Feb 19, 2008 4:13 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
"are these 10 messages written to log (assuming that messages are to be made persistent) individually or can they be written at a time?"
As specified in the official documentation, as your application MQPUTs a persistent message, it is written first to the log, then written to the queue. If you're on z/OS, the message is written to a log buffer, which is written to the physical log when the buffer manager software decides to do so.
"is there any thing that MQ provides to write the 10 messages at a time instead of one at a time?"
MQ implements Message Groups, which are one or more physical messages that comprise a logical message. This is a programming construct; but individual message segments in the message group are still written one at a time.
What issue/problem are you trying to address?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| career |
| Posted: Tue Feb 19, 2008 4:18 pm Post subject: |
|
|
Apprentice
Joined: 09 Jul 2007
Posts: 36
|
Thanks bruce!
we are just trying to reduce the IOs to increase the performance. not sure how much we can improve |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Feb 19, 2008 4:20 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
What platform? What version of MQ? What performance "problem" do you see?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| career |
| Posted: Tue Feb 19, 2008 4:23 pm Post subject: |
|
|
Apprentice
Joined: 09 Jul 2007
Posts: 36
|
we are working on LINUX v 4 and MQ 6.0.
we have no problem. this is just a thought. just curious if there is some thing like that to reduce write operations. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Feb 19, 2008 4:28 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
|
| Back to top |
|
|
| Mr Butcher |
| Posted: Wed Feb 20, 2008 12:49 am Post subject: |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
| Quote: |
| If you're on z/OS, the message is written to a log buffer, which is written to the physical log when the buffer manager software decides to do so. |
uhm... no. 
on z/OS, the message is written to the buffer pool which is assigned to the pagset the queue resides on. the log-related data is written to the log buffer. at commit time, the log buffer is written to dasd. this "hardens" the UOW. the message may still be in the buffer pool. after a specific time (2 checkpoints and something else, it is described somewhere) the message is written from the buffer pool to the pageset itself.
so a "good" performant message never hits the pageset as it is put to and get from the buffer pool, but its related log data (if persistent and receovery-relevant) is written to the log dataset latest at commit time.
otherwise both (message and log data) is in buffer and lost when mq crashes.
_________________
Regards, Butcher |
|
| Back to top |
|
|
| career |
| Posted: Wed Feb 20, 2008 7:54 am Post subject: |
|
|
Apprentice
Joined: 09 Jul 2007
Posts: 36
|
Actually i am looking at some thing which improves MQ performance related to disk I/O.
Is there any one who worked on it? or any advices to improve this performance will be much appreciated. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Feb 20, 2008 7:59 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| career wrote: |
we are working on LINUX v 4 and MQ 6.0.
we have no problem. this is just a thought. just curious if there is some thing like that to reduce write operations. |
If you put 10 messages but put them all in one UOW and then only issue one MQCMIT that will reduce I/O versus doing 10 MQCMITs.
Here is a quote from something I posted a few years ago based on a class or seminar. I assume MQ 6.0 still works the same way. By having an outstanding MQGET with no syncpoint and using an MQPUT with no syncpoint you can eliminate writes. Whether you should eliminate syncpoint for persistent messages is another question.
| Quote: |
There is a Space Table kept to manage the space in the queue buffer and the queue file.
NP messages are stored in a shared memory buffer by preferance. If the buffer overflows, they are written to the file system buffer but there is never a synchronous disk write for NP messages.
When a PUT happens, and the message is persistent, it will be allocated space in the queue file. If the message is non persistent, it will be allocated space in the queue buffer, or in the queue file if the buffer is full.
LOGS - If the message is persistent, there will normally be 1 log record written. If the message is not persistent, there is no write to the log record, even if the message spilled over to the queue file.
There is an exception to the rule of persistent messages being written to logs. If an app does an MQPUT outside of syncpoint to a queue where there is an MQGET waiting outside of syncpoint, the persistent message is passed thru in memory with no writes to the log.
If the space was allocated in the queue buffer, the message data is copied to the queue buffer. If the space was allocated to the queue file, the data will be written to the queue file via the file system buffer. If a log record is needed to record the update, it will be written before the message data is written to the queue file. If the message is put under syncpoint, neither write will by synchronous. A synchronous write to the log will be required when the transaction is committed or rolled back. |
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| career |
| Posted: Wed Feb 20, 2008 11:48 am Post subject: |
|
|
Apprentice
Joined: 09 Jul 2007
Posts: 36
|
peter thanks
i am just trying to refresh my understanding. correct me if i am wrong
| Quote: |
If you put 10 messages but put them all in one UOW and then only issue one MQCMIT that will reduce I/O versus doing 10 MQCMITs.
|
very Clear to me which reduces I/O
| Quote: |
| When a PUT happens, and the message is persistent, it will be allocated space in the queue file. If the message is non persistent, it will be allocated space in the queue buffer, or in the queue file if the buffer is full. |
so there are two writes for the persistent messages one to logs and other to queue files.
| Quote: |
| There is an exception to the rule of persistent messages being written to logs. If an app does an MQPUT outside of syncpoint to a queue where there is an MQGET waiting outside of syncpoint, the persistent message is passed thru in memory with no writes to the log. |
as messages are not written to logs and are stored in memory it is up to us how much we trust our h/w
| Quote: |
| If the space was allocated in the queue buffer, the message data is copied to the queue buffer. If the space was allocated to the queue file, the data will be written to the queue file via the file system buffer. If a log record is needed to record the update, it will be written before the message data is written to the queue file. If the message is put under syncpoint, neither write will by synchronous. A synchronous write to the log will be required when the transaction is committed or rolled back. |
if the message is to be persistent and recovery is needed then we cannot limit I/O's other than limiting MQCMITs under UOW
thanks a lot. please correct me if i am wrong |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Feb 20, 2008 3:57 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Did you check out that link? Especially the part on concurrent application processing and UOW?
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
