| Author |
Message
|
| root |
 Posted: Thu Aug 18, 2005 1:05 pm Post subject: Does MQ keep two copies of a persistent message ?? |
 |
|
Novice
Joined: 23 Sep 2004
Posts: 13
|
In the system admin guide it says that "the size of the log depends, typically, on the number and size of the persistent messages the queue manager needs to handle"
So it would seem that persistenet messages are kept in the queue file and in the logs.
Would this be true? Are there two copies of persistent messages?
_________________
Who's the Master? |
|
| Back to top |
|
 |
| kevinf2349 |
| Posted: Thu Aug 18, 2005 2:12 pm Post subject: |
|
|
Grand Master
Joined: 28 Feb 2003
Posts: 1311
Location: USA
|
| Quote: |
So it would seem that persistenet messages are kept in the queue file and in the logs.
|
Yes that is correct. |
|
| Back to top |
|
|
| hopsala |
| Posted: Thu Aug 18, 2005 4:05 pm Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
The real answer is much more complex than a simple yes/no. There used to be a redbook named something like "performance tuning in z/OS"- but it is relevant for all OSs. You should look it up.
Anyway, a short summary would go something like this:
There are three components at play here: memory buffers, log files and queue files. (there are also log buffers, but it's not that important)
Each time you commit an mq action (either explicitly or implicitly) an entry is written to the log. If it is a persistent message the entire message is written to the log.
while a message is uncommited, it resides in the memory buffer. If it remains there a long time because it is either uncommited, or no one MQGET it, or the buffer is 75% full, the message is written to the queue file.
Every once in a while (in unix/win 10000 transactions) there's a checkpoint, which flushes all memory buffer content to queue files.
So, memory buffers maintain uncommitted and recently written messages. Queue files are used to store long-running uncommited and ungotten messages. Log files contain everything commited.
(btw, you can see here the meaning of the proverb "a good queue is an empty queue" - if msgs are put and gotten at a fast pace, they are only written to memory buffers and log files. If they stay in the queue too long, they fill up the memory buffers, written to log files and to queue files - one extra IO operation)
It is worth mentioning that each queue maintains an entry stating when was the last queue file update; On startup the QM checks each queue, and if it is not updated messages are taken from the log and put to the queue file. This is dobe automatically on each startup.
Mind that there are no uncommited messages on startup, they are either backed-out or commited when you issue endmqm.
All this wonderful mechanism and more, to supply us with the marvelous performance MQ so eloquently bears to this day  |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Aug 18, 2005 4:52 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Circular Logs do not contain committed Persistent messages. Try it yourself. Create a QM with the default circular logs, which will be about 5 MB worth. Then go ahead and put 50 MB of persistent messages into the queue. Go look at the q file. It will be over 50MB large, but your logs are still only 5 MB.
Linear logs do store persistent messages, both committed and not yet committed, and uncomitted nonpersistent messages.
Circular logs only store *uncommited* persistent or non persistent messages.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mqrules |
| Posted: Thu Aug 18, 2005 6:06 pm Post subject: |
|
|
Centurion
Joined: 01 Jun 2005
Posts: 100
Location: US
|
| Quote: |
| Linear logs do store persistent messages, both committed and not yet committed, and uncomitted nonpersistent messages. |
Why should the uncommitted non persistent messages be written to the LOG?
| Quote: |
Circular logs only store *uncommited* persistent or non persistent messages.
|
Actually, I find both statments confusing. My understanding has been:
If the message that is put is persistent, it will be allocated space in the queue file (in case it has to be written to the queue file for one of the 3 resons the previous responder stated). If the message is committed, it will be written to the log.
If the message is non persistent though, it will be allocated space in the queue buffer (or in the queue file if the buffer is full). But it is never written to the log (Linear or circular).
Regardless of the message persistence, the message will be deleted from the queue file when it is retrieved. But the persistent message will stay in the log (linear or circular).
Am I missing something here? |
|
| Back to top |
|
|
| markt |
| Posted: Thu Aug 18, 2005 7:52 pm Post subject: |
|
|
Knight
Joined: 14 May 2002
Posts: 508
|
>> Circular Logs do not contain committed Persistent messages.
Wrong. They do. But they might be overwritten quickly.
You need to think of the logs as RECOVERY logs, not audit trails. So information about anything that needs to be rebuilt after an abend has to be in the log. Once the data is in its "proper" place on a qfile on disk, then the log is not needed.
A persistent message is in the log BEFORE the qfile is updated, so whether or not the transaction has committed the logs need to contain the data in case the qmgr ends before the qfile has been forced to disk. But once the qfile HAS been synced to disk then the information in the log is redundant and can be overwritten.
There are also times when it is safe not to write persistent messages to the logs at all (so you can't guarantee that the data is in there even in the case of linear logs). |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Aug 19, 2005 4:48 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| markt wrote: |
>> Circular Logs do not contain committed Persistent messages.
Wrong. They do. But they might be overwritten quickly.
|
Yes, Mark's quote is much more accurate. Mine was a bit misleading. For circular logs, persistent committed messages can be overwritten, so it is possible to have gigs and gigs of persistent messages on the queues, and only have 5 MB of logged messages (the latest 5 MB), if your circular logs happened to be only 5 MB.
But if the logs were linear, than every one of those persistent messages has dedicated space in the logs, meaning your linear logs would be taking up gigs and gigs.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| root |
| Posted: Fri Aug 19, 2005 6:38 am Post subject: |
|
|
Novice
Joined: 23 Sep 2004
Posts: 13
|
Thanks you all for your insightful answers.
| markt wrote: |
| >> Once the data is in its "proper" place on a qfile on disk, then the log is not needed. |
That's what I would think would happen, but my problem below would dictate other wise.
| PeterPotkay wrote: |
For circular logs, persistent committed messages can be overwritten, so it is possible to have gigs and gigs of persistent messages on the queues, and only have 5 MB of logged messages (the latest 5 MB), if your circular logs happened to be only 5 MB.
But if the logs were linear, than every one of those persistent messages has dedicated space in the logs, meaning your linear logs would be taking up gigs and gigs. |
Peter I believe your 100% correct on this. The question is why would MQ need to have the persistent messages on the queue AND in the logs if everything is synched up. Well now that I've confused everyone. I believe that MQ does NEED both copies of the persistent messages to RECOVER from a queue manager RESTART (normal initiated by admin) and FAILURE.
Here's the scenario we've been testing for the last two days.
Here's the environment
[mqm@nymwfarmlx1-m mqm]$ uname -a
Linux nymwfarmlx1-m 2.4.9-e.38enterprise #1 SMP Tue Feb 10 23:59:30 EST 2004 i68
[mqm@nymwfarmlx1-m mqm]$ mqver
Name: WebSphere MQ
Version: 530.9 CSD09
CMVC level: p530-09-L041213
BuildType: IKAP - (Production)
LogDefaults:
LogPrimaryFiles=3
LogSecondaryFiles=2
LogFilePages=1024
LogType=CIRCULAR
LogBufferPages=17
As you can see we don't have a large amount of circular log space.
4k*1024 = 4 MB * 5 (primary+secondary) = 20 MBs.
Not sure if the math is right here, but know i'm pretty close.
Here's the test we ran, it's very simple place 150,000.
Messages are sent down from our mainframe so I believe it's the MCA that is actually doing the mq put.
We display the queue and show that all 150,000 messages are there.
No application is reading or writing to the queues.
IPPROCS(0)
OPPROCS(0)
I browsed the queue to ensure that
ALL messages are persistent
ALL messages have expiry set to -1
We then just simply restart the queue manager
endmqm MWLXACT
strmqm MWLXACT
We then display the queue. To our Shock we lost over 50,000 messages.
I have a PMR open with IBM and they suggested that we read all the messages from the queue and do a count. We did that and we did not recover the lost 50,000 messages.
I have run this test about 4 times now always losing messages.
This has really freaked me out.
My belief is that I made a terrible error by defining such a small CIRCULAR log. I believe that MQ NEEDS to have ALL of the persistent messages stored on its log in order to recover from a restart or failure.
I believe the CIRCULAR log is being overwritten and I am losing the ability to recover (from a restart) those messages which get overwritten.
Therefore my log better be big enough to hold all of the persistent messages that may sit on my queue at any given time.
I'm going to try this same test on my solaris machines. I'll also going to try increasing the circular log to see if that can fix the issue.
I'll keep everyone posted on what happens. I just want to know if anyone knows if this is the normal behaviour of MQ. I honestly think it is, I keep going back to the system admin maunal. The part where it talks about calculating your log size. It makes a point for you to count all the persistent messages that the queumanager will handle.
_________________
Who's the Master? |
|
| Back to top |
|
|
| root |
| Posted: Fri Aug 19, 2005 7:24 am Post subject: |
|
|
Novice
Joined: 23 Sep 2004
Posts: 13
|
| hopsala wrote: |
Queue files are used to store long-running uncommited and ungotten messages. Log files contain everything commited.
|
Does this mean that the queue files do NOT contain a copy of the commited persistent messages. If this is true then things are starting to make sense.
_________________
Who's the Master? |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri Aug 19, 2005 7:45 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| root wrote: |
| Does this mean that the queue files do NOT contain a copy of the commited persistent messages. |
No.
The q files hold messages that haven't been retrieved yet. If the persistent message is committed, then it will be in the queue file until it is gotten or expired.
What is the total size of the 150,000 messages, in bytes? How big is each message?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Aug 19, 2005 9:15 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| root wrote: |
We then just simply restart the queue manager
endmqm MWLXACT
strmqm MWLXACT
We then display the queue. To our Shock we lost over 50,000 messages.
I have a PMR open with IBM and they suggested that we read all the messages from the queue and do a count. We did that and we did not recover the lost 50,000 messages.
I have run this test about 4 times now always losing messages.
This has really freaked me out.
My belief is that I made a terrible error by defining such a small CIRCULAR log. I believe that MQ NEEDS to have ALL of the persistent messages stored on its log in order to recover from a restart or failure.
I believe the CIRCULAR log is being overwritten and I am losing the ability to recover (from a restart) those messages which get overwritten.
Therefore my log better be big enough to hold all of the persistent messages that may sit on my queue at any given time.
|
You got a definite problem. That is not the way MQ works.
I just did the same test. 150,000 persistent non expiring messages each 1K in length to a QM with Circular logs. MQ 5.3.0.10 on Win XP SP1. Circular logs of only 5 MB. The queue file is 225 MB. So I have way more message data than logs.
Stop and restart the QM.
All 150,000 messages are still there, as expected. Q file still 225 MB.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| tkane |
| Posted: Fri Aug 19, 2005 10:45 am Post subject: |
|
|
Voyager
Joined: 23 Dec 2002
Posts: 82
Location: Kansas City
|
A couple of thoughts.
When the queue manager is shutdown cleanly then the messages will all be on the qfiles and the circular logs will no longer be needed. In fact in systems where I have had smaller log file sizes than I wanted I've built another queue manager with the larger logs, shut it down and moved those new log files from a differently named queue manager into the proper directory for my queue manager and restarted with the bigger logs. There are a few other tricks to this and it's probably not supported and also documented elsewhere on the forums so I'll leave it at that.
Have you double checked that your messages are in fact persistent?
run amqsbcg and is the value printed 0 or 1?
I have known "MQ Experts" who confuse defpsist on a queue object with the Persistence of a message as indicated in it's MQMD.
I'd run the same test as you have at csd10 on my linux sandbox but its logs are sized at 64mb x 63 log files as most of my new MQs are. Disk space is so cheap these days and we've been burned with messages rolled back to release log space errors so that we are configuring MQ V5.3 queue managers with the biggest logs we can.
Good Luck
Tom |
|
| Back to top |
|
|
| hopsala |
| Posted: Fri Aug 19, 2005 11:55 am Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
This post is regarding the queue file size issue, not the missing msgs issue, which I must say I have never encountered; but to me it seems that indeed this is normal mq behavior, and that the logs must be large enough to contain all concurrent persistent messages. Would be much obliged to hear results of your tests.
I just made several experiments on Win MQ5.3, but posted it here for comfort -
http://www.mqseries.net/phpBB2/viewtopic.php?p=104809#104809
My experiments shows that we were all both right and wrong concerning queue files...
I must say I was taken aback by this behavior, and truely do not understand why (oh why) should mq write all persistent messages to queue file, as my experiments showed; it seems such a waste of IO. Any ideas??
Also, this refutes what System Administration Guide says on page 198:
| Quote: |
Persistent updates to message queues happen in two stages. First, the records
representing the update are written to the log, then the queue file is updated. The
log files can thus become more up-to-date than the queue files. |
|
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri Aug 19, 2005 12:20 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
The log is not deemed sufficient to keep persistant messages, persistantly.
For one, if they were, then circular logging COULD NOT work. A persistant message has to remain on the queue until it is gotten.
The minute there were ungotten, committed, persistant messages that were larger in size than the total amount of circular log space, then the logs would have to expand to handle new work. Thus, the logs would not be circular any more.
The issue of q file size being shrunk has been discussed several times here - it would be useful to search for the previous discussions.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| hopsala |
| Posted: Fri Aug 19, 2005 12:58 pm Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
| jefflowrey wrote: |
| For one, if they were, then circular logging COULD NOT work. A persistant message has to remain on the queue until it is gotten. |
Funny you should say that, I could not agree more, it SHOULDN'T work. circular logs, in my opinion, should only be used for test systems.
My approach comes from MQ on z/OS systems. In such systems if a message is not gotten, and a circular log goes the whole circle and comes back to the same point the msg was put, MQ gets stuck.
The solution is linear logs with offloading - every time a log fills, you archive it to tape or disk, when offloaded logs become obselete you deleted them (much like linear logs in Win/Unix).
Due to this, in MF persistent msgs, commited or uncommited are only written to queue files if the buffer fills or they are not gotten for some time. This gives MUCH better performance, with a very slight extra complexity of managing offloaded logs.
We all know that MQ was written in two completely seperate teams in IBM - one wrote MQ for MF, the other for distributed systems - and this is the result. If you ask me, the MF people did a much better job. |
|
| Back to top |
|
|
|
|
