| Author |
Message
|
| Scubie |
 Posted: Mon Oct 05, 2009 4:48 am Post subject: AMQ7469 on Solaris 10 with V6.0.2.2 |
 |
|
Apprentice
Joined: 09 Oct 2001
Posts: 26
Location: Pretoria, South Africa
|
Hi All
We are occassionaly getting the following on a production system:
10/05/09 13:46:57 - Process(1707.4) User(rtgs) Program(amqzmuc0)
AMQ7469: Transactions rolled back to release log space.
EXPLANATION:
The log space for the queue manager is becoming full. One or more long-running
transactions have been rolled back to release log space so that the queue
manager can continue to process requests.
ACTION:
Try to ensure that the duration of your transactions is not excessive. Consider
increasing the size of the log to allow transactions to last longer before the
log starts to become full.
I have already adjusted the logs to the following:
LogPrimaryFiles=27
LogSecondaryFiles=18
LogFilePages=16384
LogType=CIRCULAR
which is about 9 times the default but it is still happening.
Is there any way of identifying what transaction is "long-running" and any ideas what I can do about this please?
Thanks
Carlo
_________________
________________________________
Carlo Henrico
Live fast, die young, enjoy a good looking corpse! |
|
| Back to top |
|
 |
| shashivarungupta |
| Posted: Mon Oct 05, 2009 7:59 am Post subject: |
|
|
Grand Master
Joined: 24 Feb 2009
Posts: 1343
Location: Floating in space on a round rock.
|
Following two that describes the problem that you have and about which IBM suggests something:
http://www-01.ibm.com/support/docview.wss?uid=swg21204894
http://www-01.ibm.com/support/docview.wss?uid=swg1IY92929
As you have already played with the Logging resolution(make sure you have bounced the queue managers then.), next..to Identify the "Long Running Transactions".... hmm..similar kind of problem I had faced once but it was not with Log Spaces but with the Long Running Transactions and for that pls check the Mflow (if its there), check the DB calls (if its there), check the DB tables running out of space (if its there), check the Processes (at back-end MF, if its there), check the Logs for the 2 Phase Commit transactions (if its there).
I think, a little more detail is required about that, then just the error logs. (if you could provide)
_________________
*Life will beat you down, you need to decide to fight back or leave it. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Oct 05, 2009 10:21 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
I have already adjusted the logs to the following:
LogPrimaryFiles=27
LogSecondaryFiles=18
LogFilePages=16384
LogType=CIRCULAR
which is about 9 times the default but it is still happening. |
Defaults from the factory are not usually sufficient for production. Nine times the default may not be adequate either.
Did you calculate these new values based on number of messages and size of messages? Or are these guess-timates? Or trial-and-error? The WMQ System Admin manual has a section on calculating the size of logs.
LogType=CIRCULAR
Assuming that you do not have a never-ending Unit of Work, your calculations are not providing sufficient log space.
Circular vs. Linear is a frequent subject of discussion here. You have encountered one reason not to use circular logs. Of course, the benefit of circular logs is that they do not grow over time. But, there is the risk of warp-around - as you have discovered.
Are there new applications? Ask the developers.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Scubie |
| Posted: Mon Oct 05, 2009 10:33 am Post subject: |
|
|
Apprentice
Joined: 09 Oct 2001
Posts: 26
Location: Pretoria, South Africa
|
Hi
Bruce:
| Quote: |
| Did you calculate these new values based on number of messages and size of messages? Or are these guess-timates? Or trial-and-error? |
Basically guess-timates based on a number of production sites running the same system using factory default maximum doubled. I have been having this problem for some time here and have doubled and trebled to this level and it is still happening. However, it is sporadic.
I have read a bit on the discussion on LogType and interestingly nobody wants to say definately which is better - automated housekeeping versus manual... 
The only major "new" thing is a first-time implementation on Solaris - all previous versions of the app have been on HP-UX.
Shashivarungupta:
See my point above on logs...
There are a number of processes all using MQ including a C++ app, java (via JBOSS), VB front-ends etc. Is there perhaps some way of at least pinpointing the long-running process? From there I could start narrowing it down.
Thanks so far
Carlo
_________________
________________________________
Carlo Henrico
Live fast, die young, enjoy a good looking corpse! |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Oct 05, 2009 10:52 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
From the MQSC manual:
DISPLAY CONN
Use the MQSC command DISPLAY CONN to display connection information
about the applications connected to the queue manager. This is a useful command because it enables you to identify applications with long-running units of work.
You might consider a script that does a DIS CONN once per minute, looking for indoubt handles. Following a failure, you can work backwards to see which UofWs were active for a long period of time.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Oct 05, 2009 11:06 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Scubie wrote: |
I have read a bit on the discussion on LogType and interestingly nobody wants to say definately which is better - automated housekeeping versus manual...
|
That's because neither is better - one or the other will be better in a given set of circumstances.
If you've read the forum discussions then you should have some grasp of the issues that help decide which kind of logging is best in your circumstances.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| shashivarungupta |
| Posted: Mon Oct 05, 2009 11:31 am Post subject: |
|
|
Grand Master
Joined: 24 Feb 2009
Posts: 1343
Location: Floating in space on a round rock.
|
| bruce2359 wrote: |
From the MQSC manual:
DISPLAY CONN
Use the MQSC command DISPLAY CONN to display connection information
about the applications connected to the queue manager. This is a useful command because it enables you to identify applications with long-running units of work.
You might consider a script that does a DIS CONN once per minute, looking for indoubt handles. Following a failure, you can work backwards to see which UofWs were active for a long period of time. |
 Agree !!
UOWSTDA The date that the transaction associated with the current connection was started.
UOWSTTI The time that the transaction associated with the current connection was started.
USERID The user identifier associated with the connection. This parameter is not returned when APPLTYPE has the value SYSTEM but the user.
Even you can get the PID from there.
_________________
*Life will beat you down, you need to decide to fight back or leave it. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Oct 05, 2009 12:03 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
A very brief summary of circular vs. linear:
In a situation where an errant application creates a never-ending UofW, which do you want to run out of? Logs or disk space? Out-of-logs kills off mq applications; out-of-disk space kills off... the o/s?
In a situation where there is no errant application creating a never-ending UofW, but message load (message length and number of messages) grows without advance warning to the sysadmin, same question as above.
How do you manage linear logs? Refer to the WMQ Monitoring manual.
A logger event message is created when a log segment is no longer required for restart. You can write your own automation app (or buy one) to watch for these event messages; and then delete (or move) old logs.
The same manual has AMQSLOG0.C, a Sample C program to monitor the logger event queue and outputs a message to stdout when a logger event occurs
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Scubie |
| Posted: Mon Oct 05, 2009 12:13 pm Post subject: |
|
|
Apprentice
Joined: 09 Oct 2001
Posts: 26
Location: Pretoria, South Africa
|
Thanks guys, I see MQMONNTP has a graphical display for DISPLAY CONN!
I will try to identify the process first of all!
Cheers
Carlo
_________________
________________________________
Carlo Henrico
Live fast, die young, enjoy a good looking corpse! |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Oct 05, 2009 12:30 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| I will try to identify the process first of all! |
Which process? The one causing the logs to fill up? Or the process that is the innocent victim, and merely encounters the error?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Scubie |
| Posted: Mon Oct 05, 2009 12:59 pm Post subject: |
|
|
Apprentice
Joined: 09 Oct 2001
Posts: 26
Location: Pretoria, South Africa
|
Good point - hopefully the "longest running" one. It is a transactional system, and transactions should take milliseconds...
Hopefully its just a point to begin...
_________________
________________________________
Carlo Henrico
Live fast, die young, enjoy a good looking corpse! |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Oct 06, 2009 1:35 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| "The beatings will continue until the developer who admits they forgot to code a commit steps forward". |
|
| Back to top |
|
|
| shashivarungupta |
| Posted: Thu Oct 08, 2009 9:47 am Post subject: Re: AMQ7469 on Solaris 10 with V6.0.2.2 |
|
|
Grand Master
Joined: 24 Feb 2009
Posts: 1343
Location: Floating in space on a round rock.
|
| Scubie wrote: |
I have already adjusted the logs to the following:
LogPrimaryFiles=27
LogSecondaryFiles=18
LogFilePages=16384
LogType=CIRCULAR
which is about 9 times the default but it is still happening. |
I know...you might be hitting your server(s) to find out the process which is a victim.. but if you are out of space then you can create a new file system for the 'working data' of mq (if at all its feasible and possible in your work env.)
Guidelines are given under "Creating separate file systems for working data" in the PDF 'amqaac07_Websphere MQ for AIX Quick Beginnings.pdf'.
Thanks
_________________
*Life will beat you down, you need to decide to fight back or leave it. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Oct 08, 2009 10:03 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| you can create a new file system |
...and when you run out of space on the new and bigger filesystem, you can create a newer and much bigger one.
Again, which is better? Running out of log space? Or running out of filesystem space?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| shashivarungupta |
| Posted: Thu Oct 08, 2009 10:12 am Post subject: |
|
|
Grand Master
Joined: 24 Feb 2009
Posts: 1343
Location: Floating in space on a round rock.
|
| bruce2359 wrote: |
| Quote: |
| you can create a new file system |
...and when you run out of space on the new and bigger filesystem, you can create a newer and much bigger one.
Again, which is better? Running out of log space? Or running out of filesystem space? |
I would say.. the first one.  Anyways...
That's just an option till Scubie is busy while identifying the Culprit Process(s).
As i said, (if at all its feasible and possible in your work env.)
_________________
*Life will beat you down, you need to decide to fight back or leave it. |
|
| Back to top |
|
|
|
|
