ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexGeneral IBM MQ SupportLong running transactions

Post new topicReply to topic
Long running transactions View previous topic :: View next topic
Author Message
MQMB&WAS
PostPosted: Tue Nov 12, 2019 12:37 pm Post subject: Long running transactions Reply with quote

Voyager

Joined: 12 Jun 2016
Posts: 93

Hello guys,

Could someone tell how to check for long running transactions when the mq active logs consume the entire filesystem?

Once found the culprit transaction, what can a mq admin do about it?

Couldn't find the info online, appreciate your inputs.

Thanks
Back to top
View user's profile Send private message
Vitor
PostPosted: Tue Nov 12, 2019 1:32 pm Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25858
Location: Texas, USA

One quick way is to look for a local queue with a high depth & UNCOM(YES), then ask the application that puts to that queue if there's anything they should be telling you.

(You can of course identify this queue from your documentation, the naming convention of the queue or one of the other controls you have in place......)
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Tue Nov 12, 2019 1:33 pm Post subject: Re: Long running transactions Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8534
Location: US: west coast, almost. Otherwise, enroute.

MQMB&WAS wrote:
Hello guys,

Could someone tell how to check for long running transactions when the mq active logs consume the entire filesystem?


Control command dspmqtrn

MQMB&WAS wrote:
Once found the culprit transaction, what can a mq admin do about it?

Cancel the offending application.

MQMB&WAS wrote:
Couldn't find the info online, appreciate your inputs.

Thanks

really? I googled 'mq long running transactions' and found it.
_________________
There are two types of people in this world:
1) Those that can extrapolate from incomplete data
Back to top
View user's profile Send private message
hughson
PostPosted: Wed Nov 13, 2019 2:18 pm Post subject: Reply with quote

Grand Master

Joined: 09 May 2013
Posts: 1283
Location: Bay of Plenty, New Zealand

You may find this blog post a useful read too.

IBM MQ Little Gem #19: DISPLAY CONN dates & times

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
rekarm01
PostPosted: Wed Nov 13, 2019 6:48 pm Post subject: Re: Long running transactions Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 1374

MQMB&WAS wrote:
Could someone tell how to check for long running transactions when the mq active logs consume the entire filesystem?

If the queue manager forces a commit/backout to resolve a long-running transaction, then shouldn't it be fairly easy to identify the application responsible, from the MQ error logs?

Vitor wrote:
One quick way is to look for a local queue with a high depth & UNCOM(YES), then ask the application that puts to that queue if there's anything they should be telling you.

Looking for a queue with high depth doesn't always work, when a queue manager has low volume queues with occasionally long running transactions, but also has very high volume queues with short running transactions. And it doesn't have to be that long; one long running transaction could fill up the log space within a matter of minutes, without filling up any queues.
Back to top
View user's profile Send private message
hughson
PostPosted: Wed Nov 13, 2019 8:42 pm Post subject: Re: Long running transactions Reply with quote

Grand Master

Joined: 09 May 2013
Posts: 1283
Location: Bay of Plenty, New Zealand

MQMB&WAS wrote:
Once found the culprit transaction, what can a mq admin do about it?

One other option if you can't identify the culprit to talk to them in person, might be the STOP CONN command, but that will also depend on whether 1PC or 2PC transaction, i.e. whether they are still connected.

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
gbaddeley
PostPosted: Thu Nov 14, 2019 3:04 pm Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2040
Location: Melbourne, Australia

Run "display qstatus('*') type(queue) where(uncom ne no) all".
For each queue that has uncommitted messages, run
"display qstatus('QNAME') type(handle) all" and correlate against
"display conn(*) type(conn) where(qmurid ne 0.0) all"
Look for oldest time when log was last written (UOWLOGDA & UOWLOGTI).
This will be your long running UOW.
_________________
Glenn
Back to top
View user's profile Send private message
bruce2359
PostPosted: Fri Nov 15, 2019 5:20 am Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8534
Location: US: west coast, almost. Otherwise, enroute.

What errors did you see in the active qmgr error log?

I'd have expected an error written to the active qmgr error log, such as AMQ6709: The log for the queue manager is full.

I'd have expected an error returned to the app, such as an MQCC_FAILED with 2102 (0836) (RC2102): MQRC_RESOURCE_PROBLEM. Developers are responsible for catching a wide variety of ReasonCodes.

If you are lucky, there may be only one badly behaving app filling your logs. Been there, seen this.

Any new or recently modified apps? It's also possible that a single app with multiple concurrent users is at play here.
_________________
There are two types of people in this world:
1) Those that can extrapolate from incomplete data
Back to top
View user's profile Send private message
Andyh
PostPosted: Mon Nov 18, 2019 9:21 am Post subject: Reply with quote

Disciple

Joined: 29 Jul 2010
Posts: 174

The queue manager should have dealt with the long running transaction (either backed out, or rolled forwards (if indoubt)) BEFORE the logs actually became full. If the logs did really fill to capacity (new work being refused due to log resouce constraints) as a result of a long running transaction then you should raise a service ticket with IBM.

A more common reason for a shortage of log space is that the queue manager has been created with linear logging, and the linear logs created are not being properly managed.

Management of linear log extents was a customer responsibility until 9.0.2, at which point the QMgr introduced options to allow automatic log management.
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Mon Nov 18, 2019 3:21 pm Post subject: Re: Long running transactions Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2040
Location: Melbourne, Australia

MQMB&WAS wrote:
Hello guys,
Could someone tell how to check for long running transactions when the mq active logs consume the entire filesystem?
Once found the culprit transaction, what can a mq admin do about it?
Couldn't find the info online, appreciate your inputs.
Thanks

It can be too late if the file system is full, as the qmgr may no longer be responsive for you to do anything useful.

The only alternative may be to end the qmgr, increase the number of logs, increase the file system size, and start the qmgr. Start up may take a long time, as the qmgr may need to reconcile all the logs.

Assuming you have a separate file system for the active logs, you should have alerting that detects say 80% or 90% full. This gives time to investigate and resolve the issue.
You should also have alerting on the qmgr error logs for forced roll back (AMQ7469) or roll forward (AMQ7485) or existence (AMQ7486) of long running transactions.

HTH
_________________
Glenn
Back to top
View user's profile Send private message
HubertKleinmanns
PostPosted: Fri Nov 22, 2019 12:11 am Post subject: Re: Long running transactions Reply with quote

Shaman

Joined: 24 Feb 2004
Posts: 724
Location: Germany

gbaddeley wrote:
MQMB&WAS wrote:
Hello guys,
Could someone tell how to check for long running transactions when the mq active logs consume the entire filesystem?
Once found the culprit transaction, what can a mq admin do about it?
Couldn't find the info online, appreciate your inputs.
Thanks

It can be too late if the file system is full, as the qmgr may no longer be responsive for you to do anything useful.

The only alternative may be to end the qmgr, increase the number of logs, increase the file system size, and start the qmgr. Start up may take a long time, as the qmgr may need to reconcile all the logs.

Assuming you have a separate file system for the active logs, you should have alerting that detects say 80% or 90% full. This gives time to investigate and resolve the issue.
You should also have alerting on the qmgr error logs for forced roll back (AMQ7469) or roll forward (AMQ7485) or existence (AMQ7486) of long running transactions.

HTH


You could have a look at the error logs (AMQERR01.LOG in the QMgrs data volume), have a look, which logs are still needed:

- AMQ7467 Logs needed for Restart.

- AMQ7468 Logs needed for Media Recovery.

You could at least remove the logs with a number lower than shown at AMQ7468 without any loss of functionality.

You could also remove the logs with a number lower than shown at AMQ7467, but you wouldn't be able ro perform media recovery (repairing damaged queues).
_________________
Regards
Hubert
Back to top
View user's profile Send private message Visit poster's website
bruce2359
PostPosted: Fri Nov 22, 2019 4:12 am Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8534
Location: US: west coast, almost. Otherwise, enroute.

Odd .. the OP hasnt offered details or responded to questions weve posed. Im wondering if this is a college or tech school research assignment.

When did this symptom begin? What apps were new or recently updated just before the symptom appeared? What is the configuration of the qmgr experiencing the symptom? Was/is this post a theoretical issue?

I hope we get a good grade from the professor.
_________________
There are two types of people in this world:
1) Those that can extrapolate from incomplete data
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexGeneral IBM MQ SupportLong running transactions
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.