ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexChallenge ForumChallenge Question - 06 / 2008

This forum is locked: you cannot post, reply to, or edit topics.This topic is locked: you cannot edit posts or make replies. Goto page 1, 2, 3  Next
Challenge Question - 06 / 2008 View previous topic :: View next topic
Author Message
Challenger
PostPosted: Wed Jun 04, 2008 2:56 am Post subject: Challenge Question - 06 / 2008 Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

June's Challenge deals with a real production issue faced by one of our regularly participating members whose name shall remain a secret until the winning response has been selected.

Presenting problem:

On a Unix platform: you are using linear logging and the file system where the log files reside are full as an MQ internal process is constantly writing MQ_RESOURCE_PROBLEM errors to AMQERR01.LOG .

But, when you run the "standard" Perl script (from the support pac MS62) to clean up the logs it does nothing even though there are log files that should be archived and/or deleted.

Why, and How do you fix it?

Good Luck
Back to top
View user's profile Send private message
bbburson
PostPosted: Wed Jun 04, 2008 7:10 am Post subject: Reply with quote

Partisan

Joined: 06 Jan 2004
Posts: 378
Location: Nowhere near a queue manager

Why? Because the resource problem error messages are being posted so fast into AMQERR0n.LOG that there are none of the usual AMQ7467 and AMQ7468 messages available to let the MS62 script know which log files are safe to remove.

How to deal with it? Use runmqsc commands "DIS QMSTATUS RECLOG" and "DIS QMSTATUS MEDIALOG" to find the oldest logs needed by the queue manager. Then archive/delete files that are no longer required.

Of course if you're not at version 6 this won't work and you're out of luck.
Back to top
View user's profile Send private message
Challenger
PostPosted: Wed Jun 04, 2008 9:28 am Post subject: Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

This actually happened long ago and far way on a 5.3 queue manager - but it was still possible to get ms62 to work - any ideas how? (I know it's not supported, but there are still 5.3 sites out there)
Back to top
View user's profile Send private message
Nigelg
PostPosted: Wed Jun 04, 2008 1:43 pm Post subject: Reply with quote

Grand Master

Joined: 02 Aug 2004
Posts: 1046

tail -f AMQERR01.LOG > e1
wait...
check for AMQ7467/8 in e1
repeat until the msgs are in e1
then use e1 as input to MS62.

Perl is rather twee.
Real UNIX programmers use awk.
_________________
MQSeries.net helps those who help themselves..
Back to top
View user's profile Send private message
bbburson
PostPosted: Wed Jun 04, 2008 2:02 pm Post subject: Reply with quote

Partisan

Joined: 06 Jan 2004
Posts: 378
Location: Nowhere near a queue manager

MS62 already looks in all three AMQERR0n.LOG files. If the messages aren't found there, doing a tail -f on the one will not likely gain you anything. Tail does not follow the file renames that happen when the files get too big. After a short time you'll be stuck tailing a file that is no longer being written to.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Wed Jun 04, 2008 3:42 pm Post subject: Reply with quote

Grand Poobah

Joined: 18 Nov 2003
Posts: 19770
Location: LI,NY

You need to isolate the qmgr and stop all its channels and users. Running rcdmqimg -l will give you the latest needed logs as well as create a new restart point. Archive files no longer needed.

Restart channels and applications...

Enjoy
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
Nigelg
PostPosted: Wed Jun 04, 2008 9:28 pm Post subject: Reply with quote

Grand Master

Joined: 02 Aug 2004
Posts: 1046

bburson:

I thought the idea was to come up with a solution of your own, not criticise other people's.

While we are doing that, you are wrong. The msg is witten to the file, but is removed from the file before MS62 can find it. tail -f wil dump the contents of the file as they are written.

Additionally, I also suggested that this procedure could be repeated as necessary until the msgs are found, not just run once. Of course tail does not follow file renames; once there is no more output the tail can be abandoned and restarted.

Keep your negative (and wrong) comments to yourself. If you have anything positive to say, or any answer to the challenge, post it, otherwise shut up.
_________________
MQSeries.net helps those who help themselves..
Back to top
View user's profile Send private message
Challenger
PostPosted: Thu Jun 05, 2008 12:57 am Post subject: Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

OK, well let's keep it friendly

There is a problem on tail -f with an AMQERR01.LOG, as MQ renames the file to 02 when it fills up - but "tail"s file handle doesn't follow it.

Another issue is that these AMQ74xx messages are written to the log when 1000 transactions (GETs or PUTs) are done - but if the log file directory is full then you won't be doing anymore PUTs and if MQ is well-trashed then it's unlikely you'll be doing much GETting either
Back to top
View user's profile Send private message
Nigelg
PostPosted: Thu Jun 05, 2008 1:44 am Post subject: Reply with quote

Grand Master

Joined: 02 Aug 2004
Posts: 1046

Hey, let's all rubbish my 10-second thought.
Anybody else want to put their five eggs in?

Let me put it in words of one syllable.
If it does not find the log msgs the first time, have another go until it does. There is no need to leave it running when there is no more input.

It is 10000, not 1000, and it is not puts and gets, but rather coyly named 'log operations'.

It is a complete waste of time trying to determine what logs to archive if the log directory is full anyway. The question is nonsensical. It is like asking how to change a light bulb in a cabin on the Titanic.
_________________
MQSeries.net helps those who help themselves..
Back to top
View user's profile Send private message
Challenger
PostPosted: Thu Jun 05, 2008 3:12 am Post subject: Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

OK, OK, well, "log operations" mainly means GETs and PUTS of persistent messages and yes it's 10000 not 1000 - but the issue is not to archive the old log files (who cares - throw them away) but to be able to get a snapshot of all active messages in the smallest number of log files, to save space - so you HAVE to know what is the oldest file needed for media recovery, and delete anything older - otherwise you risk losing your backup if you do need to replay the messages. Without knowing this and blindly deleting logfiles means there is a risk of a) unrecoverable queues or worse b) unstartable queue manager
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Thu Jun 05, 2008 3:00 pm Post subject: Reply with quote

Grand Poobah

Joined: 18 Nov 2003
Posts: 19770
Location: LI,NY

Again quiesce the qmgr while running:
a) stop all channels (with force if need be)
b) stop all apps connecting in bindings
c) expand enough space to run rcdmqimg -l (this will give you the new values for the restart and media logs (write them down))
d) archive unused logs to a different file volume (first manually then using the support pack) freeing up even more space
e) restart the channels
f) restart apps with bindings connection
g) schedule rcdmqimg to run on a schedule or at least once a day.
h) schedule the support pack to archive the logs after rcdmqimg has run.

And yes there is a good reason to force the business to a short outage: If we don't take the outage, the qmgr is going to crash...

Hope this helps.
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
Challenger
PostPosted: Fri Jun 06, 2008 1:57 am Post subject: Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

Sound advice. The queue manager is pretty much unusable so an outage should not be an issue - it is a crash situation.

For extra points could anybody suggest what to do if a log file WAS accidently deleted, with no backup?
Back to top
View user's profile Send private message
Gaya3
PostPosted: Fri Jun 06, 2008 2:38 am Post subject: Reply with quote

Jedi

Joined: 12 Sep 2006
Posts: 2493
Location: Boston, US

cold start procedure is the best way...if this type issue faces

Regards
Gayathri
_________________
Regards
Gayathri
-----------------------------------------------
Do Something Before you Die
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Fri Jun 06, 2008 3:07 am Post subject: Reply with quote

Grand Poobah

Joined: 18 Nov 2003
Posts: 19770
Location: LI,NY

Challenger wrote:
Sound advice. The queue manager is pretty much unusable so an outage should not be an issue - it is a crash situation.

For extra points could anybody suggest what to do if a log file WAS accidently deleted, with no backup?

If you haven't stopped the qmgr and you are missing log files the first rule would be don't stop the qmgr.
If you are running with linear logging, just try and run rcdmqimg,,, you might get lucky and it might create enough of the new files to allow you to stop the qmgr.
Otherwise just keep it (the qmgr) running until the needed logfiles have been recreated...

If you already stopped the qmgr... the cold start method seems to be the only means as Gaya3 stated. Make sure you run rcdmqimg right after restart if the qmgr is in linear logging. This should recreate the logs including the persistent messages as it creates the checkpoint.

Enjoy

These options are by no means finite. They are just the most obvious ones.

Please add your own experiences of crashes and how you solved it here.

_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
Challenger
PostPosted: Sat Jun 07, 2008 3:06 am Post subject: Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

Quote:
These options are by no means finite. They are just the most obvious ones.

Please add your own experiences of crashes and how you solved it here.


Yes please, it would be extremely interesting to see what problems other people have experienced, and how they (hopefully) fixed the situation. As I said, this challenge is based on something that actually happened, so the more the merrier.
Back to top
View user's profile Send private message
Display posts from previous:
This forum is locked: you cannot post, reply to, or edit topics.This topic is locked: you cannot edit posts or make replies. Goto page 1, 2, 3  Next Page 1 of 3

MQSeries.net Forum IndexChallenge ForumChallenge Question - 06 / 2008
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.