ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » Challenge Forum » Challenge Question - 06 / 2008

This forum is locked: you cannot post, reply to, or edit topics.  This topic is locked: you cannot edit posts or make replies. Goto page Previous  1, 2, 3  Next
 Challenge Question - 06 / 2008 « View previous topic :: View next topic » 
Author Message
samgunddi
PostPosted: Mon Jun 09, 2008 4:05 am    Post subject: Reply with quote

Acolyte

Joined: 17 May 2004
Posts: 54

we faced a similar issue, a qmgr crashed with lack of disk space. There was no way we could bring up the qmgrs "QUICKLY" with out loosing message. The volume group hosting the log disk had no additional space. The rdcmqimg even with -t qmgr option was displaying that the qmgr needed an oldest log file available for media recovery.

We came up with an interesting but simple solution (not in prd yet):

- Create a dummy file (big enough) in the log disk, to have working room.
- In case of a similar qmgr crash, delete the dummy file. Which brings up the qmgr.
- Stop all the channels, stop all the apps putting messages to the qmgr.
- remove and backup the messages from queues.
- the rcdmqimg will release the old logs.

The idea is to throttle the message with out loosing or damaging mq objects. We are at version 602 , there is a recorded APAR by IBM they recommend to upgrade the maintenance level http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg1IY96630

I believe there has to be a better solution to limit the number of linear logs created, so the qmgr will not crash. My be the qmgr will not accept any message from apps. IBM recommends using linear logs over circular logs. Hope IBM with come with some thing to address this scenario.

Any suggestion are most welcome.
_________________
IBM Certified System Administrator
- WebSphere Message Queue V5.3
- WebSphere Message Broker v6.0
Back to top
View user's profile Send private message
Nigelg
PostPosted: Mon Jun 09, 2008 4:50 am    Post subject: Reply with quote

Grand Master

Joined: 02 Aug 2004
Posts: 1046

Buy a bigger disk.

What has this got to do with IBM?
Calculate for yourself the size the log is going to grow to, and allocate sufficient disk space accordingly.
_________________
MQSeries.net helps those who help themselves..
Back to top
View user's profile Send private message
Challenger
PostPosted: Mon Jun 09, 2008 5:29 am    Post subject: Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

That's a good idea, sort of like a "reserve tank" - how did you size the dummy file - 1 or "n" times the size of one logfile?
Back to top
View user's profile Send private message
Challenger
PostPosted: Mon Jun 09, 2008 5:42 am    Post subject: Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

Quote:
What has this got to do with IBM?


IBM do recommend linear logging (which I personally disagree with, as a better solution is to use RAID to preserve disk integrity) but provide next to no tools for managing them. MS62 is written by a third-party company/consultant - it's only at version 6 that a command existed to interrogate active log filenames!
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Mon Jun 09, 2008 10:04 am    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

Challenger wrote:
That's a good idea, sort of like a "reserve tank" - how did you size the dummy file - 1 or "n" times the size of one logfile?

We use a 1GB dummy file, nice round number. You should also have your disk space monitored so you get emails when the dir is getting full and a page when the drive is getting really close to filling up. If your only alerting is happening after you're QM is down you need better monitoring in place, big time.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
Nigelg
PostPosted: Mon Jun 09, 2008 11:28 am    Post subject: Reply with quote

Grand Master

Joined: 02 Aug 2004
Posts: 1046

Quote:
IBM do recommend linear logging


Where?
_________________
MQSeries.net helps those who help themselves..
Back to top
View user's profile Send private message
Challenger
PostPosted: Mon Jun 09, 2008 11:56 pm    Post subject: Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

Quote:
Quote:
IBM do recommend linear logging


Where?

MQ Sys Admin guide - chapter 14 - Recovery and restart - Linear Logging:

Quote:
Use linear logging if you want both restart recovery and media or forward recovery (recreating lost or damaged data by replaying the contents of the log).


AFAIK these same words have been spoken since the mid-nineties - when RAID disks weren't in common use. Use the base OS for media recovery, not this half-hearted attempt that was pinched from DB2 anyway
Back to top
View user's profile Send private message
Challenger
PostPosted: Mon Jun 09, 2008 11:59 pm    Post subject: Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

Quote:
You should also have your disk space monitored so you get emails when the dir is getting full and a page when the drive is getting really close to filling up.


This is a good idea Peter. I did implement a method that would kick an archive and purge off when the disk was 60% full, and we also had a cron job running every two hours that would do the same. Can anyone spot the problem that could (and did) occur there?
Back to top
View user's profile Send private message
harwinderr
PostPosted: Tue Jun 17, 2008 4:04 am    Post subject: Reply with quote

Voyager

Joined: 29 Jan 2002
Posts: 90

I faced a similar problem where the log file system was full and the Perl script will not work because the AMQ7467 & AMQ7468 messages were over-written. What we did was:

a) Move the oldest log files to a file system which has disk space and create symbolic links to them in the qmgr log directory. We used a script to do this and moved the log files till we had about 50% free disk space.

b) Run rcdmqimg to get a AMQ7467/68 message recorded.

c) Immediately after that run the Perl script to archive the old log files that are no longer needed.

The good thing about this technique was there was no outage of qmgr required.

Quote:
This is a good idea Peter. I did implement a method that would kick an archive and purge off when the disk was 60% full, and we also had a cron job running every two hours that would do the same. Can anyone spot the problem that could (and did) occur there?


Maybe, by the time your cron job started the disk space was already full. Or there was more disk space required to take a media dump. This can happen if there are still a huge number of messages sitting on the queues.
Back to top
View user's profile Send private message Yahoo Messenger MSN Messenger
Challenger
PostPosted: Tue Jun 17, 2008 11:20 am    Post subject: Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

Hi harwinderr

I like this idea - using sym links which take up no space. And it didn't involve an outage. Nice one!

The cron job .vs. run when 60% full actually both ran at the same time - the rcdmqimg seemed to work OK but it crashed the Tivoli archiving utility - so in the end I build checks to prevent more than one occurrence running at the same time
Back to top
View user's profile Send private message
Challenger
PostPosted: Thu Jun 26, 2008 11:08 am    Post subject: And the winner is...... Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

...not one, but three!

First bburson for spotting the cause of the problem within minutes of the challenge being posted

Second, harwinderr for a novel and potentially brilliant way of fixing the problem with no downtime or outage - any chance you could post that script?

And last, but not least, nigelg for the "most defensive posts" category

Please private message Mehrdad with your mailing address.

And finally, who am I?
Back to top
View user's profile Send private message
bbburson
PostPosted: Thu Jun 26, 2008 1:15 pm    Post subject: Reply with quote

Partisan

Joined: 06 Jan 2004
Posts: 378
Location: Nowhere near a queue manager

Cool! Thanks.

I guess SAFraser as the original poster. Just a hunch; probably way off base.

By the way, my answer appeared so quickly after the question was posed because I had to deal with this exact thing not once but twice over the course of several years. Unfortunately it was pre-version6 and the queue manager could not tell me which logs were no longer needed. I had to resort to the sledge hammer approach and deleted several of the older files until I got enough space to get a good rcdmqimg. Not the best thing, I know. Wish I'd thought of harwinderr's method back then.
Back to top
View user's profile Send private message
Challenger
PostPosted: Thu Jun 26, 2008 11:29 pm    Post subject: Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

SAFraser - no, wrong continent - but close
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Fri Jun 27, 2008 8:40 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

The man of the double trout ( )?
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
Challenger
PostPosted: Fri Jun 27, 2008 11:23 pm    Post subject: Reply with quote

Centurion

Joined: 31 Mar 2008
Posts: 115

Sorry, you've got me there - the reference went completely over my head.

No trouts in my family at all.

OK here's a clue - I'm a UK citizen but not resident, while the UK is mostly flat (sorry Scotland) and is surrounded by water, the LAND I live in is the exact opposite....
Back to top
View user's profile Send private message
Display posts from previous:   
This forum is locked: you cannot post, reply to, or edit topics.  This topic is locked: you cannot edit posts or make replies. Goto page Previous  1, 2, 3  Next Page 2 of 3

MQSeries.net Forum Index » Challenge Forum » Challenge Question - 06 / 2008
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.