| Author |
Message
|
| hornbeam123 |
 Posted: Thu Dec 31, 2015 8:09 am Post subject: Logging anomaly |
 |
|
Centurion
Joined: 01 Nov 2003
Posts: 101
|
I would like to understand more about the CYCLIG log usage of a target system on windows. I understand that persistent messages are not written to CYCLIC logs. The purpose of this post is to get a full understanding of what is happening in the MQ internals not to chase a particular s/w problem.
Immediately below is the setup and at the bottom of this post are my questions.
=================================================================================
Fairly standard scenario and set up
Message flow in two broker queue managers bk1 and bk2 at solaris servers 1 and 2 receiving persistent messages from a queue manager on Linux (round-robin). Broker flows routes output depending on content either to a Windows queue manager or another message flow.
At the broker
MQInputNode
Advanced-->Transaction Mode=Yes
MQOutputNode
Advanced-->Transaction Mode=Automatic
Advanced-->Persistence Mode= YES
We take a message through the input node, modify it and send to output node. In fact we have two output nodes within a flow order. The message journey in question comes out of the First phase of the FlowOrder control.
The message destination (First phase) is a clustered queue at a remote queue manager (MQ7.5 Windows).
At that remote queue manager we have cyclic logs 10 and 5 extents primary and secondary respectively.
The messages are consumed by a c# application under the guise of the .net classes and inserted into a database. I doubt whether they are got under a MQ syncpoint (needs confirming).
This message consumer is the only MQ application running on this windows server and in fact it is the only application queue defined.
We had a recent situation whereby the consumer process running at the windows server was down because of database issues and messages were quickly queuing. I noticed on browsing the primary extents that all of them contained data that represented the spread of mesages currently on the queue (approx 600,000 messages at the time).
=================================================================================
The following questions arise:
Under what circumstances, if at all, would we go to secondary extents at the remote windows server?
The broker flow I assume is putting the message in a transaction with its associated input message and then committing. Does this affect the MQPUT at the target Windows queue manager?
Apart from the maximum queue depth what governs how many messages can go to this queue? Is it just the space on the disk drive that hosts the queue manager queues folder bearing in mind this is the only one MQ application process running on this server?
At what point is the cyclic log record made redundant?
Why does MQ seemingly use ALL the primary extents and not just use one extent?
================================================================================= |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Thu Dec 31, 2015 9:17 am Post subject: Re: Logging anomaly |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| hornbeam123 wrote: |
Why does MQ seemingly use ALL the primary extents and not just use one extent? |
Working as documented and designed.
You/your sysadmin specifies the number and size of primary & secondary logs when the qmgr is created. As documented, primaries are formatted and allocated at qmgr creation. Secondaries are formatted and allocated when primaries are exhausted - usually during peak workloads.
This might be useful for you http://www-01.ibm.com/support/docview.wss?uid=swg27013886&aid=1
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| hornbeam123 |
| Posted: Thu Dec 31, 2015 10:22 am Post subject: Logging anomaly |
|
|
Centurion
Joined: 01 Nov 2003
Posts: 101
|
Thanks for your reply Bruce but it didn't really answer my questions.
I never said it wasn't working as designed.
That's the very point I understand the fundamentals but what I need to understand is what is happening under the covers to explain what I saw.
One relevant piece of information that I was reminded of from that overview presentation was the following:
"Note: The addition of the secondary logs into the active set depends on the location of the current write point in the active log set (called the head)" .
Now somebody out there expand on that and perhaps answer all my questions I would be a happy bunny. I can wait to next year.
cheers |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Dec 31, 2015 11:00 am Post subject: Re: Logging anomaly |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| hornbeam123 wrote: |
The following questions arise:
Under what circumstances, if at all, would we go to secondary extents at the remote windows server? |
On any qmgr (not just the remote), when the primaries are exhausted, secondaries will be inserted in the rotation when and where they are needed.
| hornbeam123 wrote: |
| The broker flow I assume is putting the message in a transaction with its associated input message and then committing. Does this affect the MQPUT at the target Windows queue manager? |
Yes. MQ work performed in SYNCPOINT (transactions) is logged - including persistent messages MQPUT, MQGET, transactions MQCMIT, MQBACK, object attribute changes, are written to the log.
| hornbeam123 wrote: |
| Apart from the maximum queue depth what governs how many messages can go to this queue? |
MAXDEPTH is one possibility. Max message length queue attribute would prevent a message from being MQPUT to a queue if its length >, thus affecting how many messages go to the queue. Read on.
| hornbeam123 wrote: |
| Is it just the space on the disk drive that hosts the queue manager queues folder bearing in mind this is the only one MQ application process running on this server? |
File-system space is another possibility. There is a section of the sysadmin manual titled something like "calculating log file size," or "how to calculate log file size."
| hornbeam123 wrote: |
| At what point is the cyclic log record made redundant? |
A log segment (file) will be overwritten when there are no longer active units of work on that segment.
| hornbeam123 wrote: |
| Why does MQ seemingly use ALL the primary extents and not just use one extent? |
One extent? Active log segments (extents) contain units of work (transactions) in-flight. If a long-running UofW exceeds one extent, the next extent will be used for other logging. If the last Primary extent also becomes full (of UofW's), secondaries will be inserted. If secondaries become full, the qmgr will not be able to perform any logging; and each app attempting UofW activity will receive a ReasonCode indicating log files are full.
Again, primaries are formatted and allocated at qmgr creation, and recycled (overwritten) when they become inactive.
From a search for 'crtmqm lp' (lp = log primary):
| Quote: |
On a Windows system, the minimum number of primary log files you can have is 2 and the maximum is 254. On UNIX and Linux systems, the minimum number of primary log files you can have is 2 and the maximum is 510. The default is 3.
On a Windows system, the total number of primary and secondary log files must not exceed 255 and must not be less than 3. On UNIX and Linux systems the total number of primary and secondary log files must not exceed 511 and must not be less than 3. |
What exactly is your concern?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| tczielke |
| Posted: Thu Dec 31, 2015 11:21 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
|
| Back to top |
|
|
| smdavies99 |
| Posted: Thu Dec 31, 2015 12:06 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Many solutions using MQ can operate perfectly well using Circular Logging.
In my experience the exceptions are the ones with Linear Logging. Others may well have different requirements that need linear logs.
That said, I configure the QMGRs with a log ot large log files.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Dec 31, 2015 4:29 pm Post subject: Re: Logging anomaly |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| hornbeam123 wrote: |
We had a recent situation whereby the consumer process running at the windows server was down because of database issues and messages were quickly queuing. I noticed on browsing the primary extents that all of them contained data that represented the spread of mesages currently on the queue (approx 600,000 messages at the time). |
Is this the behavior that you find confusing? What confuses you?
What anomaly are you referring to in the title of your original post?
What are the message lengths?
What are the other log attributes - other than number of primaries and secondaries?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Jan 01, 2016 8:23 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| smdavies99 wrote: |
Many solutions using MQ can operate perfectly well using Circular Logging.
In my experience the exceptions are the ones with Linear Logging. Others may well have different requirements that need linear logs.
That said, I configure the QMGRs with a log ot large log files. |
Its worth noting on the IBM MQ Appliance, the only option is circular logging. http://www-01.ibm.com/support/knowledgecenter/SS5K6E_1.0.0/com.ibm.mqa.doc/planning/pl00004_.htm?lang=en
Interesting that IBM decided not to offer Linear as an option on this platform. Does it speak more to the fact the appliance is a tightly controlled environment with storage of a design that eliminates any need for Linear, and/or was it just an opportune time for IBM to reconsider the whole Linear/Circular debate on a platform that did not need to deal with 2 decades of backwards compatibility ?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| hornbeam123 |
| Posted: Tue Jan 05, 2016 6:52 am Post subject: Logging anomaly |
|
|
Centurion
Joined: 01 Nov 2003
Posts: 101
|
Thanks for your further input everybody. This is a difficult one to get across. It is not just academic curiosity.
The anomaly I guess is why does MQ go through these primary log extents in this way and given the hugh number of messages and the quick arrival time of them we never opened any secondary.
The behavior of the logs is confusing.
The message sizes are pretty small 1 to 3k (excl mqmd) and logfile pages set to 10,240.
As I said before we have this unreliable application that every so often fails to run for several days so this causes the queue to get very large. Given the fact that persistent messages are not stored on the Q we know we have sufficient disk space for the 'queues' folder to grow.
What I can't estimate is will we ever use those secondary extents and how many should I allocate.
We also know that before the message is commited it is held on the log but after the enqueue the record in log is freed.
We established before to quote the tech conference paper "that the addition of the secondary logs nto the active set depends on the location of the current write point in the active log set (called the head)". I assume this uses the active lfh file but can we view this head in a meaningful way?
It seems to me whatever that queue depth is and it was pretty high last time we had an application failing, the secondary extents will never be used because the messages once commited the record on the primary log is freed. I think the queue manager logging mechanism was able to keep up with the arrival rate to the queue and never needed to go to secondary and that would have probably been the case if we went for a longer period with the consuming application down.
I suspect we are not going to be able to fully answer this, and to estimate when MQ would need to go to secondary (if at all) in this particular case is purely guesswork. Therefore I will increase the secondary extents just in case. Unlikely we will see the volumes and high arrival rates for another year anyway. Hopefully by that time the consuming application will be reliable. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jan 05, 2016 7:19 am Post subject: Re: Logging anomaly |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| hornbeam123 wrote: |
| Given the fact that persistent messages are not stored on the Q we know we have sufficient disk space for the 'queues' folder to grow. |
When did that become a fact? I always thought that persistent messages were stored on the queue and in the logs to ensure a copy was available in the event of a storage failure (key reason for the best practice of queue and logs being on 2 different physical mounts).
| hornbeam123 wrote: |
| We also know that before the message is commited it is held on the log but after the enqueue the record in log is freed. |
Again, how do you know that?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Andyh |
| Posted: Tue Jan 05, 2016 7:54 am Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 239
|
With linear logging then a copy of each persisted message will always exist in the logs, but does not need to be within the active log (the log files needed for crash recovery and limited by primaries + secondaries).
With circular logging, at least one copy of each persisted message will exist in either the log, or on the queue files, or in both.
With both linear and circular, the MQ queue manager is a classic "write ahead logging resource manager" where persisted data is first written to the recovery log, and then if that data is long lived may also be flushed to a backing store (the queue files).
The size of the active log should primarily be determined by the duration of the longest expected transaction, and the maximum persistent throughput of the queue manager. It is relatively independent of the total volume of persistent messages concurrently hosted by the queue manager.
I would typically suggest that the number of primaries should be large enough that secondary log space is seldom required (typically at least 80% primary log space).
Any customer with a very large active log is allowing for a very long running transaction, and thus the potential for a long crash recovery time.
For example, if you provide a 64GB active log then you're accepting that one transaction might span 64GB of log space. If the queue manager should crash while such a transaction is active then it's possible that the entire 64GB of log would need to be read to recover the queue manager to a consistent state, which could take quite some time. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jan 05, 2016 9:15 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Now that's a nice, clear description. Thank you.
 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue Jan 05, 2016 1:40 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| Andyh wrote: |
The size of the active log should primarily be determined by the duration of the longest expected transaction, and the maximum persistent throughput of the queue manager. It is relatively independent of the total volume of persistent messages concurrently hosted by the queue manager.
.
.
.
Any customer with a very large active log is allowing for a very long running transaction, and thus the potential for a long crash recovery time.
For example, if you provide a 64GB active log then you're accepting that one transaction might span 64GB of log space. If the queue manager should crash while such a transaction is active then it's possible that the entire 64GB of log would need to be read to recover the queue manager to a consistent state, which could take quite some time. |
Hmmm. Food for thought. Just because you have 1 terrabyte of storage available doesn't mean you should make the QM logs files as big as possible, and have as many of them as possible.
On the topic of secondary log files, I have never seen a secondary log file in all the years I have worked with MQ. Not that I sit and stare at my QM log directories all day long. I never really understood secondaries. If you have enough disk space for the secondaries, why not just make that many more primaries and skip the secondaries all together? Having said that, I always use 10% as my guide for creating secondaries - if I create the QM with 30 Primaries, I give it 3 secondaries and move on with my life.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
|
|
