Author |
Message
|
aditya.aggarwal |
Posted: Mon Nov 16, 2009 8:09 am Post subject: Active Log File Number Mismatch |
|
|
 Master
Joined: 13 Jan 2009 Posts: 252
|
Hi,
I have below log file setting for a queue manager
Log:
LogPrimaryFiles=3
LogSecondaryFiles=10
LogFilePages=16384
LogType=CIRCULAR
LogBufferPages=512
LogPath=/var/mqm/log/XYZ/
LogWriteIntegrity=TripleWrite
So the number of Active log files should not exceed 10+3....
but today when i browsed the /var/mqm/log/XYZ/active directory...
i found the 80 log files there...
bash-3.00$ ls -lrt
total 656640
-rw-rw---- 1 mqm mqm 4202496 Nov 14 01:42 S0000002.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 02:52 S0000000.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 04:01 S0000001.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 05:10 S0000003.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 06:22 S0000004.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 07:32 S0000005.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 08:38 S0000006.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 09:43 S0000007.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 10:43 S0000008.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 11:41 S0000009.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 12:21 S0000010.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 12:54 S0000011.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 13:27 S0000012.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 13:58 S0000013.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 14:31 S0000014.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 15:02 S0000015.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 15:35 S0000016.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 16:07 S0000017.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 16:39 S0000018.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 17:10 S0000019.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 17:42 S0000020.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 18:15 S0000021.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 18:46 S0000022.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 19:18 S0000023.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 19:51 S0000024.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 20:25 S0000025.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 20:57 S0000026.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 21:29 S0000027.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 22:01 S0000028.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 22:33 S0000029.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 23:05 S0000030.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 14 23:40 S0000031.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 00:20 S0000032.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 00:54 S0000033.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 01:30 S0000034.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 02:03 S0000035.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 02:37 S0000036.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 03:10 S0000037.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 03:44 S0000038.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 04:17 S0000039.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 04:51 S0000040.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 05:25 S0000041.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 05:58 S0000042.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 06:31 S0000043.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 07:05 S0000044.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 07:39 S0000045.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 08:13 S0000046.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 08:47 S0000047.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 09:20 S0000048.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 09:52 S0000049.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 10:25 S0000050.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 10:58 S0000051.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 11:31 S0000052.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 12:03 S0000053.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 12:35 S0000054.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 13:07 S0000055.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 13:41 S0000056.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 14:13 S0000057.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 14:45 S0000058.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 15:17 S0000059.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 15:49 S0000060.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 16:22 S0000061.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 16:54 S0000062.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 17:26 S0000063.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 17:59 S0000064.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 18:47 S0000065.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 19:43 S0000066.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 20:27 S0000067.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 21:11 S0000068.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 21:56 S0000069.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 22:56 S0000070.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 15 23:41 S0000071.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 16 02:56 S0000072.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 16 03:01 S0000073.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 16 03:01 S0000074.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 16 03:02 S0000075.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 16 03:02 S0000076.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 16 03:03 S0000077.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 16 03:03 S0000078.LOG
-rw-rw---- 1 mqm mqm 4202496 Nov 16 04:03 S0000079.LOG
I also got a FDC today
|
WebSphere MQ First Failure Symptom Report |
========================================= |
|
Date/Time :- Monday November 16 02:56:49 PST 2009 |
Host Name :- apldb02 (SunOS 5.10) |
PIDS :- 5724H7203 |
LVLS :- 6.0.0.0 |
Product Long Name :- WebSphere MQ for Sun Solaris |
Vendor :- IBM |
Probe Id :- HL006009 |
Application Name :- MQM |
Component :- hlgWriteLogRecord |
SCCS Info :- lib/logger/amqhlgw0.c, 1.26 |
Line Number :- 348 |
Build Date :- May 19 2005 |
CMVC level :- p000-L050519 |
Build Type :- IKAP - (Production) |
UserID :- 00000301 (mqm) |
Program Name :- amqzxma0_nd |
Addressing mode :- 64-bit |
Process :- 24910 |
Thread :- 1 |
QueueManager :- ESTGMQA |
ConnId(1) IPCC :- 2 |
ConnId(2) QM :- 2 |
ConnId(3) QM-P :- 2 |
ConnId(4) App :- 2 |
Major Errorcode :- hrcE_LOG_FULL |
Minor Errorcode :- OK |
Probe Type :- MSGAMQ6709 |
Probe Severity :- 2 |
Probe Description :- AMQ6709: The log for the Queue manager is full. |
FDCSequenceNumber :- 0 |
|
-----------------------------------------------------------------------------+
but later when i checked the amqzxma0 process..it was avialable..
bash-3.00$ ps -ef | grep amqzxma
mqm 22843 17113 0 04:18:01 pts/14 0:00 grep amqzxma
mqm 24910 1 0 02:50:01 ? 1:53 amqzxma0 -m XYZ
so i have ignored this FDC..
in qmgr logs i found...
-------------------------------------------------------------------------------
11/16/09 02:56:56 - Process(24910.1) User(mqm) Program(amqzxma0_nd)
AMQ7463: The log for queue manager ESTGMQA is full.
EXPLANATION:
This message is issued when an attempt to write a log record is rejected
because the log is full. The queue manager will attempt to resolve the problem.
ACTION:
This situation may be encountered during a period of unusually high message
traffic. However, if you persistently fill the log, you may have to consider
enlarging the size of the log. You can either increase the number of log files
by changing the values in the queue manager configuration file. You will then
have to stop and restart the queue manager. Alternatively, if you need to make
the log files themselves bigger, you will have to delete and recreate the queue
manager.
----- amqalmw0.c : 671 --------------------------------------------------------
11/16/09 03:03:32 - Process(24910.1) User(mqm) Program(amqzxma0_nd)
AMQ7233: 62997 out of 62997 in-flight transactions resolved for queue manager
'XYZ'.
EXPLANATION:
62997 transactions out of 62997 in-flight at the time queue manager XYZ
ended have been resolved.
ACTION:
None.
-------------------------------------------------------------------------------
11/16/09 03:03:33 - Process(25564.4) User(mqm) Program(amqzmuc0)
AMQ5037: The Queue Manager task 'CHECKPOINT' has started.
EXPLANATION:
The Utility Task Manager, processId(0) type(25564), has started the CHECKPOINT
task.
ACTION:
None.
-------------------------------------------------------------------------------
11/16/09 03:03:33 - Process(24910.1) User(mqm) Program(amqzxma0_nd)
AMQ7472: Object USTGBRKB, type queue damaged.
EXPLANATION:
Object USTGBRKB, type queue has been marked as damaged. This indicates that the
queue manager was either unable to access the object in the file system, or
that some kind of inconsistency with the data in the object was detected.
ACTION:
If a damaged object is detected, the action performed depends on whether the
queue manager supports media recovery and when the damage was detected. If the
queue manager does not support media recovery, you must delete the object as no
recovery is possible. If the queue manager does support media recovery and the
damage is detected during the processing performed when the queue manager is
being started, the queue manager will automatically initiate media recovery of
the object. If the queue manager supports media recovery and the damage is
detected once the queue manager has started, it may be recovered from a media
image using the rcrmqmobj command or it may be deleted.
----- amqaocua.c : 549 --------------------------------------------------------
as the xmit queue[persistant] was damaged..so i have deleted and recreated it again.... befroe damage there were around 3000 messages in to this queue...
may be this was the reason for the LOG full...and FDC alert..?
I have checked the queues and we are receving messages...and there is no complain reported from the sender...everything is up and running...
I am lil bit surprized that why the number of log files are greater then 13?..and why this number is not decreasing[ may be message traffic]...?
Note: This all happen when /var/mqm become 100% full and we received alert lately from Hardware team... |
|
Back to top |
|
 |
exerk |
Posted: Mon Nov 16, 2009 8:39 am Post subject: |
|
|
 Jedi Council
Joined: 02 Nov 2006 Posts: 6339
|
So has someone increased the number of logs because of "...The log for queue manager ESTGMQA is full...", and you, or someone else, reduced it down again? And not forgetting that changing the number of logs may not be immediately apparent (see the manual for a more detailed explanation).
If the number of your logs needed increasing because the original amount was insufficient, and you are now breaching disc limits, find out what has changed in the messaging/application sphere, and grow the log space accordingly. _________________ It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
Back to top |
|
 |
aditya.aggarwal |
Posted: Mon Nov 16, 2009 8:59 am Post subject: |
|
|
 Master
Joined: 13 Jan 2009 Posts: 252
|
I have not made any changes in the number of logs... That's why i was surprized...
Seems that someone from Application team who also have mqm access is playing around with these settings...
Regards,
Aditya |
|
Back to top |
|
 |
aditya.aggarwal |
Posted: Mon Nov 16, 2009 9:02 am Post subject: |
|
|
 Master
Joined: 13 Jan 2009 Posts: 252
|
I have not made any changes in the number of logs... That's why i was surprized...
Seems that someone from Application team who also have mqm access is playing around with these settings...
just for cursoity....thee were around 3000 persistant message in the damage xmit queue... will all these message processed automatically from active logs by QMGR after recreating this damaged queue ..
?
Regards,
Aditya |
|
Back to top |
|
 |
Vitor |
Posted: Mon Nov 16, 2009 9:16 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
aditya.aggarwal wrote: |
Seems that someone from Application team who also have mqm access is playing around with these settings... |
Now why would the application team have mqm access? I ask slightly rhetorically.
 _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
Vitor |
Posted: Mon Nov 16, 2009 9:20 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
aditya.aggarwal wrote: |
just for cursoity....thee were around 3000 persistant message in the damage xmit queue... will all these message processed automatically from active logs by QMGR after recreating this damaged queue ..
? |
Well in theory yes - this is the point of persistent messages.
But if someone's been fiddling with the queue manager settings, especially using this unsupported method, I think you should be glad this is a dev box.
(This is dev isn't it? App team doesn't have mqm rights in production?) _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
aditya.aggarwal |
Posted: Mon Nov 16, 2009 9:33 am Post subject: |
|
|
 Master
Joined: 13 Jan 2009 Posts: 252
|
Thanks Vitor...
it is a preprod box...
I don't know how the application team and some client members have mqm access ..don't know why....we recently start supporting this client..
i have escalated it earlier ..but no results... |
|
Back to top |
|
 |
zonko |
Posted: Mon Nov 16, 2009 9:52 am Post subject: |
|
|
Voyager
Joined: 04 Nov 2009 Posts: 78
|
Quote: |
thee were around 3000 persistant message in the damage xmit queue... will all these message processed automatically from active logs by QMGR after recreating this damaged queue .. |
No, not in circular logging. Media recovery is only done in linear logging, so the msgs are lost. |
|
Back to top |
|
 |
fjb_saper |
Posted: Mon Nov 16, 2009 9:36 pm Post subject: |
|
|
 Grand High Poobah
Joined: 18 Nov 2003 Posts: 20756 Location: LI,NY
|
From the output it looks like your app team might also have changed the LINEAR logs to CIRCULAR logs without recreating the qmgr... Or they reduced the number of logfiles without restarting the qmgr...
First I would run something like support Pack MS0L (if Linear logs are being used) to clear unneeded linear logs... If all logs are still needed you can reference the oldest ones through a symlink after having moved them off the main mount point... This should allow you to free enough space to stop / restart the qmgr.
Have fun  _________________ MQ & Broker admin |
|
Back to top |
|
 |
aditya.aggarwal |
Posted: Tue Nov 17, 2009 12:21 am Post subject: |
|
|
 Master
Joined: 13 Jan 2009 Posts: 252
|
Quote: |
Quote:
thee were around 3000 persistant message in the damage xmit queue... will all these message processed automatically from active logs by QMGR after recreating this damaged queue ..
No, not in circular logging. Media recovery is only done in linear logging, so the msgs are lost. |
In case of circular logging persistant message can be recovered... Message recovery and media recovery are different.. |
|
Back to top |
|
 |
mvic |
Posted: Tue Nov 17, 2009 7:28 am Post subject: |
|
|
 Jedi
Joined: 09 Mar 2004 Posts: 2080
|
|
Back to top |
|
 |
fjb_saper |
Posted: Tue Nov 17, 2009 9:35 pm Post subject: |
|
|
 Grand High Poobah
Joined: 18 Nov 2003 Posts: 20756 Location: LI,NY
|
Mvic's explanation is way more likely than mine.
That's what you get when you don't check the qmgr level (I know it was in the FDC which I skipped....) Anyway you should really upgrade to the latest level. For V 6. the first decent level was 6.0.1.0 or 6.0.1.1... and you should move to 6.0.2.8....
Have fun  _________________ MQ & Broker admin |
|
Back to top |
|
 |
|