| Author |
Message
|
| yuvarajc |
 Posted: Fri Nov 27, 2009 7:41 am Post subject: Log full error owing to long running transactions |
 |
|
Newbie
Joined: 27 Nov 2009
Posts: 2
|
Hi
We are facing problem with QM logs constantly stating
AMQ7469: Transactions rolled back to release log space.
EXPLANATION:
The log space for the queue manager is becoming full. One or more long-running
transactions have been rolled back to release log space so that the queue
manager can continue to process requests.
ACTION:
Try to ensure that the duration of your transactions is not excessive. Consider
increasing the size of the log to allow transactions to last longer before the
log starts to become full.
Our Queue manager configuration is provided below as
DISPLAY QMGR
10 : DISPLAY QMGR
AMQ8408: Display Queue Manager details.
QMNAME(WPS.QM1) ACCTCONO(DISABLED)
ACCTINT(1800) ACCTMQI(OFF)
ACCTQ(OFF) ACTIVREC(MSG)
ALTDATE(2009-11-24) ALTTIME(00.16.5
AUTHOREV(DISABLED) CCSID(437)
CHAD(DISABLED) CHADEV(DISABLED)
CHADEXIT( ) CHLEV(DISABLED)
CLWLDATA( ) CLWLEXIT( )
CLWLLEN(100) CLWLMRUC(999999999)
CLWLUSEQ(LOCAL) CMDLEVEL(600)
COMMANDQ(SYSTEM.ADMIN.COMMAND.QUEUE) CRDATE(2009-11-24)
CRTIME(00.08.09) DEADQ(WPS.DLTQ)
DEFXMITQ( ) DESCR( )
DISTL(YES) INHIBTEV(DISABLED)
IPADDRV(IPV4) LOCALEV(DISABLED)
LOGGEREV(DISABLED) MAXHANDS(512)
MAXMSGL(4194304) MAXPRTY(9)
MAXUMSGS(10000) MONACLS(QMGR)
MONCHL(LOW) MONQ(LOW)
PERFMEV(DISABLED) PLATFORM(WINDOWSNT)
QMID(WPS.QM1_2009-11-24_00.08.09) REMOTEEV(DISABLED)
REPOS( ) REPOSNL( )
ROUTEREC(MSG) SCHINIT(QMGR)
SCMDSERV(QMGR) SSLCRLNL( )
SSLCRYP( ) SSLEV(DISABLED)
SSLFIPS(NO)
SSLKEYR(D:\MQ\Data\qmgrs\WPS!QM1\ssl\key)
SSLRKEYC(0) STATACLS(QMGR)
STATCHL(OFF) STATINT(1800)
STATMQI(OFF) STATQ(OFF)
STRSTPEV(ENABLED) SYNCPT
TRIGINT(999999999)
Also our log details are
Log Type - Circular
Primary Log files - 30
Secondary Log files - 10
Log file size (4KB) - 1024
Also, Channels - Max Active channels is set to 1500.
During peak time, we could see around 810+ connections. When I view display connections on QM, I could see most of the sender channels having the UOWSTDA for yesterday and UOWSTTI also yesterday.
DO they have ny significance towards Long-running transactions. How will I identify long-running transactions?
Appreciate your support.
Thanks
Yuva |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Fri Nov 27, 2009 8:04 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Long running transactions are transactions that take quite some time between begin and commit/rollback.
Push the problem of identification back to the applications team.
You may also increase the number of primary / secondary logs (providing you do have the space...)
If you need to increase the log size you will have to delete and rebuild the qmgr.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Nov 27, 2009 9:39 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9472
Location: US: west coast, almost. Otherwise, enroute.
|
Two possibilities here: you have insufficient logs to handle the workload; OR you sufficient logs, but have a "long running transaction." WMQ can't tell the difference.
Your current log allocations seem very small to me. These may have worked well in test, but are clearly not sufficient for production.
How to calculate log size is well-documented in the WMQ System Admin manual. This is where your app developers need to be involved. Only they know what a transaction comprises, and whether is is "long running" or not.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| mvic |
| Posted: Fri Nov 27, 2009 4:06 pm Post subject: Re: Log full error owing to long running transactions |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| yuvarajc wrote: |
During peak time, we could see around 810+ connections. When I view display connections on QM, I could see most of the sender channels having the UOWSTDA for yesterday and UOWSTTI also yesterday.
DO they have ny significance towards Long-running transactions. How will I identify long-running transactions? |
Number of connections need not be a concern. It's what the apps are doing that is causing this.
Log space is consumed by persistent data being put to the log. Also the log preserves the ability to commit or roll back its oldest transaction, which means it must remember all the log records in that transaction.
If there is a long-running transaction and persistent messaging continuing (related or unrelated to the transaction), your log becomes fuller and fuller. And eventually it will reach a point where MQ says "enough", and rolls back that old transaction. It does this so that it can release the log space for further persistent messaging operations.
Because the things done in the transaction (MQGETs, MQPUTs) are all rolled back, the application should hopefully simply be able to restart the transaction and try it again. But maybe this is not possible to be done automatically in all cases because it was user input that began the transaction and the user gives up. It depends on the scenario but in some scenarios it can just be tried again.
The log-full condition can also happen if your transaction is not particularly long, but apps are putting large persistent messages. In this case the log again fills because of MQ's needing to remember log records from that transaction.
Increasing the log size can be done by increasing primary and secondary numbers up to the defined limits. But you cannot increase the size of each log file without also deleting/creating the queue manager. |
|
| Back to top |
|
|
| happyj |
| Posted: Mon Nov 30, 2009 5:50 am Post subject: |
|
|
Voyager
Joined: 07 Feb 2005
Posts: 87
|
sorry to hijack this thread but I have a related question.
Is there any way to get an idea of how much log space has been in use
or is in use at any time, other than a AMQ7469 error to indicate that all the log space has been used. I have some high throughput queue managers with all persistent messaging and have tended to put in lots of log space to avoid this kind of error but would like to know if capacity limits were being approached so that preemptive action could be taken. |
|
| Back to top |
|
|
| JosephGramig |
| Posted: Mon Nov 30, 2009 5:59 am Post subject: |
|
|
Grand Master
Joined: 09 Feb 2006
Posts: 1244
Location: Gold Coast of Florida, USA
|
Of course...
If you have a monitoring system in place that will give you alerts, then set it to give you a warning at 70% and critical at 85% file space full.
Then increase your primary logs to fill 70% of the file space for logs (surely you knew to separate /var/mqm/log from everything else). |
|
| Back to top |
|
|
| exerk |
| Posted: Mon Nov 30, 2009 6:32 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| JosephGramig wrote: |
Of course...
If you have a monitoring system in place that will give you alerts, then set it to give you a warning at 70% and critical at 85% file space full.
Then increase your primary logs to fill 70% of the file space for logs (surely you knew to separate /var/mqm/log from everything else). |
I'm not sure if the poster is referring to circular logs...
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| JosephGramig |
| Posted: Tue Dec 01, 2009 5:04 am Post subject: |
|
|
Grand Master
Joined: 09 Feb 2006
Posts: 1244
Location: Gold Coast of Florida, USA
|
exerk,
They said so in the first post... |
|
| Back to top |
|
|
| exerk |
| Posted: Tue Dec 01, 2009 5:12 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| JosephGramig wrote: |
Of course...
If you have a monitoring system in place that will give you alerts, then set it to give you a warning at 70% and critical at 85% file space full.
Then increase your primary logs to fill 70% of the file space for logs (surely you knew to separate /var/mqm/log from everything else). |
The above implies that linear logging is being monitored, as any space for circular logs should really already be allocated just in case secondaries need to be cut - hence my comment, which was not in reference to the first poster, but to the hijack 
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| happyj |
| Posted: Tue Dec 01, 2009 4:37 pm Post subject: |
|
|
Voyager
Joined: 07 Feb 2005
Posts: 87
|
Hi there
I run at my current shop a mix of linear and circular qmgrs and on a variety of platforms - I wouldn't choose linear these days with the kind of disks in use but it is hard to change these things when they are up and running and yes I do monitor for disk space with appropriate alarms
The question was really one of MQ log capacity and if there is any way to see how much log capacity was being used over a period of time rather
than no errors means that there is sufficient space and a transaction rollback means that there isn't (or a process is looping etc!).
I tend to allocate lots of log files ( and disk ) to high transaction and
particularly variable transaction level servers as downtime is much much
more expensive than storage but in these cost cutting times it would be
good to at least get some stats. |
|
| Back to top |
|
|
| happyj |
| Posted: Tue Dec 01, 2009 4:48 pm Post subject: |
|
|
Voyager
Joined: 07 Feb 2005
Posts: 87
|
Joseph
that's a great and very inventive suggestion - unfortunately the servers
I had in mind use linear logs and that is not likely to change. |
|
| Back to top |
|
|
| JosephGramig |
| Posted: Wed Dec 02, 2009 7:26 am Post subject: |
|
|
Grand Master
Joined: 09 Feb 2006
Posts: 1244
Location: Gold Coast of Florida, USA
|
For linear, you should be using the support pac (forgot the number but it has been posted here recently) to clean up the logs that are no longer needed.
The monitor some times also has the ability to execute commands in certain conditions. So, you might have that run the scripts under those conditions.
I would make sure to schedule those scripts to run at a time to not interfere with regular work (although I've never know that to be disruptive). |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Dec 02, 2009 9:04 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9472
Location: US: west coast, almost. Otherwise, enroute.
|
For linear logs, when a log segment is no longer required for restart, an admin message is sent to the SYSTEM.ADMIN.LOGGER.EVENT. The admin message will identify the name of the logfile segment. You can have automation delete it.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| mvic |
| Posted: Wed Dec 02, 2009 9:10 am Post subject: |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| bruce2359 wrote: |
| For linear logs, when a log segment is no longer required for restart, an admin message is sent to the SYSTEM.ADMIN.LOGGER.EVENT. The admin message will identify the name of the logfile segment. You can have automation delete it. |
 Only delete log files that are no longer needed for MEDIA recovery.
Another way to gather this information is to look in the output from DISPLAY QMSTATUS. This can be a bit easier to code than a program to read a PCF message. Just my opinion.. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Dec 02, 2009 11:07 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9472
Location: US: west coast, almost. Otherwise, enroute.
|
True, there's media recovery to consider. Other posts here have discussed the relative value of using linear log media recovery.
My view, briefly: 1) logs are best used for restart following a failure; 2) but logs are fragile; and 3) given that fragility, really, really important objects should be backed up somewhere other than logs.
(awaiting the avalanch of other opinioins)
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
