| Author |
Message
|
| samgunddi |
 Posted: Mon Jun 09, 2008 4:05 am Post subject: |
 |
|
Acolyte
Joined: 17 May 2004
Posts: 54
|
we faced a similar issue, a qmgr crashed with lack of disk space. There was no way we could bring up the qmgrs "QUICKLY" with out loosing message. The volume group hosting the log disk had no additional space. The rdcmqimg even with -t qmgr option was displaying that the qmgr needed an oldest log file available for media recovery.
We came up with an interesting but simple solution (not in prd yet):
- Create a dummy file (big enough) in the log disk, to have working room.
- In case of a similar qmgr crash, delete the dummy file. Which brings up the qmgr.
- Stop all the channels, stop all the apps putting messages to the qmgr.
- remove and backup the messages from queues.
- the rcdmqimg will release the old logs.
The idea is to throttle the message with out loosing or damaging mq objects. We are at version 602 , there is a recorded APAR by IBM they recommend to upgrade the maintenance level http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg1IY96630
I believe there has to be a better solution to limit the number of linear logs created, so the qmgr will not crash. My be the qmgr will not accept any message from apps. IBM recommends using linear logs over circular logs. Hope IBM with come with some thing to address this scenario.
Any suggestion are most welcome.
_________________
IBM Certified System Administrator
- WebSphere Message Queue V5.3
- WebSphere Message Broker v6.0 |
|
| Back to top |
|
 |
| Nigelg |
| Posted: Mon Jun 09, 2008 4:50 am Post subject: |
|
|
Grand Master
Joined: 02 Aug 2004
Posts: 1046
|
Buy a bigger disk.
What has this got to do with IBM?
Calculate for yourself the size the log is going to grow to, and allocate sufficient disk space accordingly.
_________________
MQSeries.net helps those who help themselves.. |
|
| Back to top |
|
|
| Challenger |
| Posted: Mon Jun 09, 2008 5:29 am Post subject: |
|
|
Centurion
Joined: 31 Mar 2008
Posts: 115
|
| That's a good idea, sort of like a "reserve tank" - how did you size the dummy file - 1 or "n" times the size of one logfile? |
|
| Back to top |
|
|
| Challenger |
| Posted: Mon Jun 09, 2008 5:42 am Post subject: |
|
|
Centurion
Joined: 31 Mar 2008
Posts: 115
|
| Quote: |
| What has this got to do with IBM? |
IBM do recommend linear logging (which I personally disagree with, as a better solution is to use RAID to preserve disk integrity) but provide next to no tools for managing them. MS62 is written by a third-party company/consultant - it's only at version 6 that a command existed to interrogate active log filenames! |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Jun 09, 2008 10:04 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| Challenger wrote: |
| That's a good idea, sort of like a "reserve tank" - how did you size the dummy file - 1 or "n" times the size of one logfile? |
We use a 1GB dummy file, nice round number. You should also have your disk space monitored so you get emails when the dir is getting full and a page when the drive is getting really close to filling up. If your only alerting is happening after you're QM is down you need better monitoring in place, big time.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| Nigelg |
| Posted: Mon Jun 09, 2008 11:28 am Post subject: |
|
|
Grand Master
Joined: 02 Aug 2004
Posts: 1046
|
| Quote: |
| IBM do recommend linear logging |
Where?
_________________
MQSeries.net helps those who help themselves.. |
|
| Back to top |
|
|
| Challenger |
| Posted: Mon Jun 09, 2008 11:56 pm Post subject: |
|
|
Centurion
Joined: 31 Mar 2008
Posts: 115
|
| Quote: |
Quote:
IBM do recommend linear logging
Where? |
MQ Sys Admin guide - chapter 14 - Recovery and restart - Linear Logging:
| Quote: |
| Use linear logging if you want both restart recovery and media or forward recovery (recreating lost or damaged data by replaying the contents of the log). |
AFAIK these same words have been spoken since the mid-nineties - when RAID disks weren't in common use. Use the base OS for media recovery, not this half-hearted attempt that was pinched from DB2 anyway |
|
| Back to top |
|
|
| Challenger |
| Posted: Mon Jun 09, 2008 11:59 pm Post subject: |
|
|
Centurion
Joined: 31 Mar 2008
Posts: 115
|
| Quote: |
| You should also have your disk space monitored so you get emails when the dir is getting full and a page when the drive is getting really close to filling up. |
This is a good idea Peter. I did implement a method that would kick an archive and purge off when the disk was 60% full, and we also had a cron job running every two hours that would do the same. Can anyone spot the problem that could (and did) occur there? |
|
| Back to top |
|
|
| harwinderr |
| Posted: Tue Jun 17, 2008 4:04 am Post subject: |
|
|
Voyager
Joined: 29 Jan 2002
Posts: 90
|
I faced a similar problem where the log file system was full and the Perl script will not work because the AMQ7467 & AMQ7468 messages were over-written. What we did was:
a) Move the oldest log files to a file system which has disk space and create symbolic links to them in the qmgr log directory. We used a script to do this and moved the log files till we had about 50% free disk space.
b) Run rcdmqimg to get a AMQ7467/68 message recorded.
c) Immediately after that run the Perl script to archive the old log files that are no longer needed.
The good thing about this technique was there was no outage of qmgr required.
| Quote: |
| This is a good idea Peter. I did implement a method that would kick an archive and purge off when the disk was 60% full, and we also had a cron job running every two hours that would do the same. Can anyone spot the problem that could (and did) occur there? |
Maybe, by the time your cron job started the disk space was already full. Or there was more disk space required to take a media dump. This can happen if there are still a huge number of messages sitting on the queues. |
|
| Back to top |
|
|
| Challenger |
| Posted: Tue Jun 17, 2008 11:20 am Post subject: |
|
|
Centurion
Joined: 31 Mar 2008
Posts: 115
|
Hi harwinderr
I like this idea - using sym links which take up no space. And it didn't involve an outage. Nice one!
The cron job .vs. run when 60% full actually both ran at the same time - the rcdmqimg seemed to work OK but it crashed the Tivoli archiving utility  - so in the end I build checks to prevent more than one occurrence running at the same time |
|
| Back to top |
|
|
| Challenger |
| Posted: Thu Jun 26, 2008 11:08 am Post subject: And the winner is...... |
|
|
Centurion
Joined: 31 Mar 2008
Posts: 115
|
...not one, but three!
First bburson for spotting the cause of the problem within minutes of the challenge being posted
Second, harwinderr for a novel and potentially brilliant way of fixing the problem with no downtime or outage - any chance you could post that script?
And last, but not least, nigelg for the "most defensive posts" category 
Please private message Mehrdad with your mailing address.
And finally, who am I? |
|
| Back to top |
|
|
| bbburson |
| Posted: Thu Jun 26, 2008 1:15 pm Post subject: |
|
|
Partisan
Joined: 06 Jan 2004
Posts: 378
Location: Nowhere near a queue manager
|
Cool! Thanks.
I guess SAFraser as the original poster. Just a hunch; probably way off base.
By the way, my answer appeared so quickly after the question was posed because I had to deal with this exact thing not once but twice over the course of several years. Unfortunately it was pre-version6 and the queue manager could not tell me which logs were no longer needed. I had to resort to the sledge hammer approach and deleted several of the older files until I got enough space to get a good rcdmqimg. Not the best thing, I know. Wish I'd thought of harwinderr's method back then. |
|
| Back to top |
|
|
| Challenger |
| Posted: Thu Jun 26, 2008 11:29 pm Post subject: |
|
|
Centurion
Joined: 31 Mar 2008
Posts: 115
|
SAFraser - no, wrong continent - but close  |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Jun 27, 2008 8:40 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
The man of the double trout (  )?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Challenger |
| Posted: Fri Jun 27, 2008 11:23 pm Post subject: |
|
|
Centurion
Joined: 31 Mar 2008
Posts: 115
|
Sorry, you've got me there - the reference went completely over my head.
No trouts in my family at all.
OK here's a clue - I'm a UK citizen but not resident, while the UK is mostly flat (sorry Scotland) and is surrounded by water, the LAND I live in is the exact opposite.... |
|
| Back to top |
|
|
|
|
