| Author |
Message
|
| shashivarungupta |
 Posted: Mon Sep 16, 2013 11:53 pm Post subject: Large number of Primary and Secondary Active Log File ? |
 |
|
Grand Master
Joined: 24 Feb 2009
Posts: 1343
Location: Floating in space on a round rock.
|
Hi All,
As a recent changes in the application code and messages from the application connecting to MQ Qmgr, application faced an issue of MQRC_BACKED_OUT cause of all the MQ Active Primary+Secondary(7+4) log files were occupied by the Long Running Transactions.
As a primary solution the LP+LS values are increased to (40+20) and that fixed the issue.
Lately that issue resurfaced and we've found that more than five years ago the queue manager was created with 4 MB of Log Page Size and that's not enough for application Long Running Transactions at this moment.
As a Quick fix from MQ side, can the LP+LS(linear logs) values increased to 120+60 with 4 MB of messages to support their long running transactions, the concern is if that would affect the MQ Performance ?
(Quick fix is to resume the services for application which is critical and which can not modify their code to fix this issue at the moment unless they roll back their application code to older version.)
Would there be any performance Issue from MQ End in that case when there are around 180 active logs in the system , trying to serve the application long messages ?
Appreciate your inputs. thanks !!
_________________
*Life will beat you down, you need to decide to fight back or leave it. |
|
| Back to top |
|
 |
| shashivarungupta |
| Posted: Tue Sep 17, 2013 12:11 am Post subject: |
|
|
Grand Master
Joined: 24 Feb 2009
Posts: 1343
Location: Floating in space on a round rock.
|
Cause MQ is on v7.01.6 and on Unix platform... as per MQ InfoCenter.. 'The total number of primary and secondary log files cannot exceed 511 on UNIX systems'
So, I hope that would not lead to MQ Internal Issues and high CPU consumption.
_________________
*Life will beat you down, you need to decide to fight back or leave it. |
|
| Back to top |
|
|
| exerk |
| Posted: Tue Sep 17, 2013 12:12 am Post subject: Re: Large number of Primary and Secondary Active Log File ? |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| shashivarungupta wrote: |
| As a Quick fix from MQ side, can the LP+LS(linear logs) values increased to 120+60 with 4 MB of messages to support their long running transactions, the concern is if that would affect the MQ Performance ? |
What do you mean by LP+LS(linear logs)? If you've had to increase the number of logs to prevent roll-back, logic dictates you must be using circular logging in your queue manager; if you were using linear logging it's unlikely you'd see the roll-back issue but might notice that the restart log requirement might need what seems to be an excessive number of logs available.
To answer the other part of your question, performance will always be affected by long running transactions being rolled back, so ensure you have sufficient logging space to prevent it happening - then ask the applications people some very pointed questions as to how they can justify spanning a UoW for the length of time that it takes to fill the available logs. Bear in mind that if you've had to increase the number of twice their perceived issue (MQ is the problem) will be fixed, but not at the root cause of the problem (their application).
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| shashivarungupta |
| Posted: Tue Sep 17, 2013 1:08 am Post subject: Re: Large number of Primary and Secondary Active Log File ? |
|
|
Grand Master
Joined: 24 Feb 2009
Posts: 1343
Location: Floating in space on a round rock.
|
| exerk wrote: |
| shashivarungupta wrote: |
| As a Quick fix from MQ side, can the LP+LS(linear logs) values increased to 120+60 with 4 MB of messages to support their long running transactions, the concern is if that would affect the MQ Performance ? |
What do you mean by LP+LS(linear logs)? If you've had to increase the number of logs to prevent roll-back, logic dictates you must be using circular logging in your queue manager; if you were using linear logging it's unlikely you'd see the roll-back issue but might notice that the restart log requirement might need what seems to be an excessive number of logs available. |
From the roll back perspective, what if that all the 11 (7+4) logs are occupied for a single transaction in the linear log ? I believe a roll back of transaction would happen and as in this case application would receive MQ 2003 for that UoW.
Similarly incase of Circular log where one transaction would occupy a ring of logs and roll back would happen ! Isn't it !!
| exerk wrote: |
| To answer the other part of your question, performance will always be affected by long running transactions being rolled back, so ensure you have sufficient logging space to prevent it happening - then ask the applications people some very pointed questions as to how they can justify spanning a UoW for the length of time that it takes to fill the available logs. Bear in mind that if you've had to increase the number of twice their perceived issue (MQ is the problem) will be fixed, but not at the root cause of the problem (their application). |
That's the main question, from MQ Perspective what would be the Internal load on MQ Qmgr in order to manage those 180 or more than that active log files (primary+secondary) of Linear logs ?
Would there be more MQ internal processes aligned for that and would that take more of CPU on the server and would that anyhow affect MQ processing as a whole ?
_________________
*Life will beat you down, you need to decide to fight back or leave it. |
|
| Back to top |
|
|
| exerk |
| Posted: Tue Sep 17, 2013 1:22 am Post subject: Re: Large number of Primary and Secondary Active Log File ? |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| shashivarungupta wrote: |
| From the roll back perspective, what if that all the 11 (7+4) logs are occupied for a single transaction in the linear log ? I believe a roll back of transaction would happen and as in this case application would receive MQ 2003 for that UoW. |
What happens when a linear log is approaching full, for any reason? What is your understanding of what linear means?
| shashivarungupta wrote: |
| Similarly incase of Circular log where one transaction would occupy a ring of logs and roll back would happen ! Isn't it !! |
You've already experienced what would happen so your question is somewhat redundant.
| shashivarungupta wrote: |
| That's the main question, from MQ Perspective what would be the Internal load on MQ Qmgr in order to manage those 180 or more than that active log files (primary+secondary) of Linear logs ? |
You seem to be fixating, again, on linear logging when you're clearly not using it, why? Also consider, if you were using linear logging would all of the logs be active?
| shashivarungupta wrote: |
| Would there be more MQ internal processes aligned for that and would that take more of CPU on the server and would that anyhow affect MQ processing as a whole ? |
If you'd base-lined your server before increasing the log numbers you'd be able to answer your own question in regard to CPU load etc. - without base-line figures how can you capacity manage and plan for changes such as this, or declare that enough is enough for any given queue manager? The processing rate of a queue manager is going to be affected by a number of factors, and the queue manager knows how many logs it has to manage (circular), but if it has to roll back a long running transaction that will have an impact.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| shashivarungupta |
| Posted: Tue Sep 17, 2013 1:59 am Post subject: Re: Large number of Primary and Secondary Active Log File ? |
|
|
Grand Master
Joined: 24 Feb 2009
Posts: 1343
Location: Floating in space on a round rock.
|
| exerk wrote: |
| shashivarungupta wrote: |
| From the roll back perspective, what if that all the 11 (7+4) logs are occupied for a single transaction in the linear log ? I believe a roll back of transaction would happen and as in this case application would receive MQ 2003 for that UoW. |
What happens when a linear log is approaching full, for any reason? What is your understanding of what linear means? |
Linear is sequential, if primary logs are getting full the secondary logs would get created and becomes the part of active logs in case of long running transactions.
If even the secondary logs are full or getting full, there would be a resource crunch !
_________________
*Life will beat you down, you need to decide to fight back or leave it. |
|
| Back to top |
|
|
| shashivarungupta |
| Posted: Tue Sep 17, 2013 2:04 am Post subject: Re: Large number of Primary and Secondary Active Log File ? |
|
|
Grand Master
Joined: 24 Feb 2009
Posts: 1343
Location: Floating in space on a round rock.
|
| exerk wrote: |
| shashivarungupta wrote: |
| That's the main question, from MQ Perspective what would be the Internal load on MQ Qmgr in order to manage those 180 or more than that active log files (primary+secondary) of Linear logs ? |
You seem to be fixating, again, on linear logging when you're clearly not using it, why? Also consider, if you were using linear logging would all of the logs be active? |
nope, it is not necessary that all the logs would be active. Active logs entries would be required for restart of Qmgr.
_________________
*Life will beat you down, you need to decide to fight back or leave it. |
|
| Back to top |
|
|
| exerk |
| Posted: Tue Sep 17, 2013 2:09 am Post subject: Re: Large number of Primary and Secondary Active Log File ? |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| shashivarungupta wrote: |
| exerk wrote: |
| shashivarungupta wrote: |
| From the roll back perspective, what if that all the 11 (7+4) logs are occupied for a single transaction in the linear log ? I believe a roll back of transaction would happen and as in this case application would receive MQ 2003 for that UoW. |
What happens when a linear log is approaching full, for any reason? What is your understanding of what linear means? |
Linear is sequential, if primary logs are getting full the secondary logs would get created and becomes the part of active logs in case of long running transactions.
If even the secondary logs are full or getting full, there would be a resource crunch ! |
And you'd get a different MQRC. So, again, why do you have this fixation on linear logging?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| shashivarungupta |
| Posted: Tue Sep 17, 2013 2:55 am Post subject: Re: Large number of Primary and Secondary Active Log File ? |
|
|
Grand Master
Joined: 24 Feb 2009
Posts: 1343
Location: Floating in space on a round rock.
|
| exerk wrote: |
| shashivarungupta wrote: |
| exerk wrote: |
| shashivarungupta wrote: |
| From the roll back perspective, what if that all the 11 (7+4) logs are occupied for a single transaction in the linear log ? I believe a roll back of transaction would happen and as in this case application would receive MQ 2003 for that UoW. |
What happens when a linear log is approaching full, for any reason? What is your understanding of what linear means? |
Linear is sequential, if primary logs are getting full the secondary logs would get created and becomes the part of active logs in case of long running transactions.
If even the secondary logs are full or getting full, there would be a resource crunch ! |
And you'd get a different MQRC. So, again, why do you have this fixation on linear logging? |
(Edit : But MQ did not receive MQRC for resource crunch. Application team those who has been sending the Long Messages, they got MQRC 2003.)
cause MQ file system has got enough space though all the MQ Log Files were Occupied by the long running application transaction and when I looked more, 'why' back out happened .. it can be understood a bit about that.. which appl. has faced. Here.
qm.ini file has got....
Log:
LogPrimaryFiles=200
LogSecondaryFiles=100
LogFilePages=16384
LogType=LINEAR
LogBufferPages=0
LogPath=/qmgr1/mgr/mqm/log/QMGR1/
LogWriteIntegrity=TripleWrite
_________________
*Life will beat you down, you need to decide to fight back or leave it.
Last edited by shashivarungupta on Tue Sep 17, 2013 3:19 am; edited 1 time in total |
|
| Back to top |
|
|
| exerk |
| Posted: Tue Sep 17, 2013 3:18 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
Right. So, finally, we get the information that you the queue manager is linear logged - would have been so much easier if you'd stated that in the original post rather than confusing the issue.
There are three possibilities:
- Your logs are full and you get MQRC_BACKED_OUT
- Your logs are full and you get MQRC_RESOURCE_PROBLEM
- Your logs aren't full but there is a problem with the rate at which data is being written to the logs exceeding the ability of the asynchronous logging task to keep up
When it comes to logging, it's easier to manage a small number of large files than a large number of small files, but you're pretty much locked in now.
You've pretty much all you can as a tactical fix but you've got to be very careful that it doesn't become the strategic fix - kick back on the application to fix the long-running UoW issue.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| shashivarungupta |
| Posted: Tue Sep 17, 2013 4:00 am Post subject: |
|
|
Grand Master
Joined: 24 Feb 2009
Posts: 1343
Location: Floating in space on a round rock.
|
| exerk wrote: |
| Right. So, finally, we get the information that you the queue manager is linear logged - would have been so much easier if you'd stated that in the original post rather than confusing the issue. |
It was captured in the OP...
| Quote: |
| 'As a Quick fix from MQ side, can the LP+LS(linear logs) values.....' |
| exerk wrote: |
There are three possibilities:
- Your logs are full and you get MQRC_BACKED_OUT
- Your logs are full and you get MQRC_RESOURCE_PROBLEM
- Your logs aren't full but there is a problem with the rate at which data is being written to the logs exceeding the ability of the asynchronous logging task to keep up
When it comes to logging, it's easier to manage a small number of large files than a large number of small files, but you're pretty much locked in now.
You've pretty much all you can as a tactical fix but you've got to be very careful that it doesn't become the strategic fix - kick back on the application to fix the long-running UoW issue. |
Application teams are most of the times are stupid, they deploy their new versions of code without thinking and intimating an infrastructure team , and they come & cry for their messages.
_________________
*Life will beat you down, you need to decide to fight back or leave it. |
|
| Back to top |
|
|
| exerk |
| Posted: Tue Sep 17, 2013 4:14 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| shashivarungupta wrote: |
| exerk wrote: |
| Right. So, finally, we get the information that you the queue manager is linear logged - would have been so much easier if you'd stated that in the original post rather than confusing the issue. |
It was captured in the OP... |
It was inferred but it wasn't explicit. And my first question was "...What do you mean by LP+LS(linear logs)?...", which as usual you failed to answer; you'd help yourself get a lot more targeted advice if you answered questions when asked.
And the symptom of MQRC_BACKED_OUT was (to me) more indicative of circular logging as I'd have expected MQRC_RESOURCE_PROBLEM from a linear-logged queue manager.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| Andyh |
| Posted: Tue Sep 17, 2013 4:31 am Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 239
|
MQRC_BACKED_OUT is equally applicable to both linear and circular logging.
In the case of linear logging the queue manager tries to limit the possible replay to NumPrimaries + NumSecondaries and so will start to take action to backout a long running unit of work when approx 80% of the NumPrimaries and NumSecondaries is spanned by a single UOW.
In general I would advise you to investigate the long running UOW, rather than to increase the available log space, to ensure that the long running transaction is by design, rather than by accident.
Once you've established that a long running UOW really is required, there should be no significant runtime performance degradation to the queue manager through exploiting a large set of logs. You do however have to remember that by allowing the queue manager a large active log you risk needing to replay that entire log at queue manager crash recovery, and hence there is a performance implication to the potential restart times. |
|
| Back to top |
|
|
| shashivarungupta |
| Posted: Tue Sep 17, 2013 4:40 am Post subject: |
|
|
Grand Master
Joined: 24 Feb 2009
Posts: 1343
Location: Floating in space on a round rock.
|
| exerk wrote: |
| shashivarungupta wrote: |
| exerk wrote: |
| Right. So, finally, we get the information that you the queue manager is linear logged - would have been so much easier if you'd stated that in the original post rather than confusing the issue. |
It was captured in the OP... |
It was inferred but it wasn't explicit. And my first question was "...What do you mean by LP+LS(linear logs)?...", which as usual you failed to answer; you'd help yourself get a lot more targeted advice if you answered questions when asked. |
The answer for that was inferred 
_________________
*Life will beat you down, you need to decide to fight back or leave it. |
|
| Back to top |
|
|
| bkiran2020 |
| Posted: Thu Sep 19, 2013 11:12 am Post subject: |
|
|
Master
Joined: 20 Jan 2011
Posts: 243
Location: US
|
| Andyh wrote: |
MQRC_BACKED_OUT is equally applicable to both linear and circular logging.
In the case of linear logging the queue manager tries to limit the possible replay to NumPrimaries + NumSecondaries and so will start to take action to backout a long running unit of work when approx 80% of the NumPrimaries and NumSecondaries is spanned by a single UOW.
In general I would advise you to investigate the long running UOW, rather than to increase the available log space, to ensure that the long running transaction is by design, rather than by accident.
Once you've established that a long running UOW really is required, there should be no significant runtime performance degradation to the queue manager through exploiting a large set of logs. You do however have to remember that by allowing the queue manager a large active log you risk needing to replay that entire log at queue manager crash recovery, and hence there is a performance implication to the potential restart times. |
As we know that In linear logging the logs are written in countious fashion.
Is this statement make any sence
If we define the qmgr with 5 LP and 3 LS this means that a long running transcation can write 5 logs and even it can use 3 secordary logs.
Log:
LogPrimaryFiles=5
LogSecondaryFiles=3
LogFilePages=16384
LogType=LINEAR
LogBufferPages=0
LogPath=/qmgr1/mgr/mqm/log/QMGR1/
LogWriteIntegrity=TripleWrite
 |
|
| Back to top |
|
|
|
|
