| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| MQ Trace choking log file? |
« View previous topic :: View next topic » |
| Author |
Message
|
| elikatz |
 Posted: Fri Feb 27, 2009 2:50 pm Post subject: MQ Trace choking log file? |
 |
|
Voyager
Joined: 24 Feb 2009
Posts: 86
|
Hi, we have MQ 5.3 CSD 11 on windows 2003 cluster.
Today I had something very strange, I tried to run MQ trace for a few minutes and then stopped it.
On this point (or a few minutes later, not sure) one of the queue managers got stuck and caused the cluster to move to second node.
From the minutes of the crash i see a few messages:
- 6709 the log for the Queue manager is full and 6184 An internal WebSphere MQ error has occurred on queue manager ZZZZ.
- 7463 the log for queue manager HARMONY is full and 6125 An internal WebSphere MQ error has occurred
Is it possible it has something to do with the trace or it was just a coincidence?
If not, what can cause this? I found some possible like very big message or bunch or big messages or very big volume over those minutes that caused the queue manager to crash.
Thanks for your patience... |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Fri Feb 27, 2009 3:43 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Did an FDC get created?
Did any other app fail due to out-of-disk-space condition?
Circular logs? How big are your logs? Primary? Secondary? Look in the log directory for logs belonging to the qmgr that had the problem. Dates, sizes, match?
In future posts, please copy/paste the entire error (AMQ6709) - not just the numeric portion - and include the complete text of the error - including the user response (action), so we don't have to look up the message.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| elikatz |
| Posted: Fri Feb 27, 2009 4:14 pm Post subject: |
|
|
Voyager
Joined: 24 Feb 2009
Posts: 86
|
Apologies for the lack of data, I wasn't sure what should I upload here and what I should not.
Here is the missing info:
- There were a lot of FDC's generated.
- There wasn't lack of disk space and no other application failed on this server
- Log are circular, 256K, 6 primary and 4 secondary files all files are currently from the date and time of the incident
- There are 6 file in the active log folder marked S0000000.LOG to S0000005.LOG
Most FDC's header looks the same but there is a difference on the size of the file, here is the header:
+-----------------------------------------------------------------------------+
| |
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- Fri February 27 11:41:32 Eastern Standard Time 2009 |
| Host Name :- MACHINE NAME (Windows Ver 5.2 Build 3790: Service Pack 1) |
| PIDS :- 5724B4100 |
| LVLS :- 530.11 CSD11 |
| Product Long Name :- WebSphere MQ for Windows |
| Vendor :- IBM |
| Probe Id :- HL010004 |
| Application Name :- MQM |
| Component :- hlgReserveLogSpace |
| Build Date :- Aug 2 2005 |
| CMVC level :- p530-11-L050802 |
| Build Type :- IKAP - (Production) |
| UserID :- siteop |
| Process Name :- D:\PROGRA~1\IBM\WEBSPH~1\bin\amqzlaa0.exe |
| Process :- 00017812 |
| Thread :- 00004449 |
| QueueManager :- QMGR |
| Major Errorcode :- hrcE_LOG_FULL |
| Minor Errorcode :- OK |
| Probe Type :- MSGAMQ6709 |
| Probe Severity :- 2 |
| Probe Description :- AMQ6709: The log for the Queue manager is full. |
| FDCSequenceNumber :- 0 |
| |
+-----------------------------------------------------------------------------+
and the errors from the event viewer:
Event Type: Error
Event Source: WebSphere MQ
Event Category: None
Event ID: 6709
Date: 2/27/2009
Time: 11:41:30 AM
User: N/A
Computer: MACHINE NAME
Description:
The log for the Queue manager is full.
This message is issued when an attempt to write a log record is rejected because the log is full. The queue manager will attempt to resolve the problem.
This situation may be encountered during a period of unusually high message traffic. However, if you persistently fill the log, you may have to consider enlarging the size of the log. You can either increase the number of log files by changing the values in the queue manager configuration file. You will then have to stop and restart the queue manager. Alternatively, if you need to make the log files themselves bigger, you will have to delete and recreate the queue manager.
Event Type: Warning
Event Source: WebSphere MQ
Event Category: None
Event ID: 6184
Date: 2/27/2009
Time: 11:41:30 AM
User: N/A
Computer: MACHINE NAME
Description:
An internal WebSphere MQ error has occurred on queue manager HARMONY.
An error has been detected, and the WebSphere MQ error recording routine has been called. The failing process is process 25688.
Use the standard facilities supplied with your system to record the problem identifier, and to save the generated output files. Contact your IBM support center. Do not discard these files until the problem has been resolved.
Event Type: Error
Event Source: WebSphere MQ
Event Category: None
Event ID: 7463
Date: 2/27/2009
Time: 11:41:30 AM
User: N/A
Computer: MACHINE NAME
Description:
The log for queue manager QMGR is full.
This message is issued when an attempt to write a log record is rejected because the log is full. The queue manager will attempt to resolve the problem.
This situation may be encountered during a period of unusually high message traffic. However, if you persistently fill the log, you may have to consider enlarging the size of the log. You can either increase the number of log files by changing the values in the queue manager configuration file. You will then have to stop and restart the queue manager. Alternatively, if you need to make the log files themselves bigger, you will have to delete and recreate the queue manager.
Event Type: Error
Event Source: WebSphere MQ
Event Category: None
Event ID: 6125
Date: 2/27/2009
Time: 11:41:30 AM
User: N/A
Computer: MACHINE NAME
Description:
An internal WebSphere MQ error has occurred.
An internal error has occurred with identifier 20807417. This message is issued in association with other messages.
Use the standard facilities supplied with your system to record the problem identifier, and to save the generated output files. Contact your IBM support center. Do not discard these files until the problem has been resolved.
thanks you! |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Feb 27, 2009 4:39 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| Log are circular, 256K, 6 primary and 4 secondary |
Two things to do.
First, please refer to the MQ system admin guide for v5.3. Look up crtmqm, and read the portion that describes the log file size parameters (-lf, -lp and -ls). Read the section on "Calculating the size of the log." It's basic math from there.
Second, what message size(s) are your apps creating? Are any apps now creating bigger messages? And how many in a single UofW? How many concurrent UofWs? Any new applications? Or recently modified and recompiled applications? How many concurrent apps? Any recent growth?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| exerk |
| Posted: Sat Feb 28, 2009 12:14 pm Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
As a temporary measure increase the number of logs, both Primary and Secondary, and restart the queue manager.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sun Mar 01, 2009 8:58 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| bruce2359 wrote: |
| Quote: |
| Log are circular, 256K, 6 primary and 4 secondary |
Two things to do. |
Three things to do.
Upgrade to a supported version, preferably MQV7.0.0.1. |
|
| Back to top |
|
|
| elikatz |
| Posted: Mon Mar 02, 2009 8:51 am Post subject: |
|
|
Voyager
Joined: 24 Feb 2009
Posts: 86
|
we started a proccess of moving to MQ version 6.x (6.0.2.5) |
|
| Back to top |
|
|
| Garik |
| Posted: Tue Mar 17, 2009 6:43 pm Post subject: |
|
|
Newbie
Joined: 12 Jul 2008
Posts: 4
|
the output of trace has to be filtered otherwise your traced process can fill a disk space quickly.
If you started trace in hours of high traffic, CPU usage would be raising more that 50%.
As I understood the tracing has to be used to debug the application like "start app->start trace->stop app->stop trace->dspmqtrc /var/mqm/trace/filename.TRC"
* Start trace for every WebSphere MQ process:
strmqtrc -e
Or start trace only for one queue manager:
strmqtrc -m MY.QMGR
Or start a high detail trace for one queue manager:
strmqtrc -t all -t detail -m MY.QMGR
Or start a high detail wrapping trace and limit the file size to ~5MB:
strmqtrc -l 5 -t all -t detail -m MY.QMGR
* End all tracing:
endmqtrc -a
* Format the trace files:
dspmqtrc *.TRC
Or format wrapping trace files:
dspmqtrc *.TRC *.TRS |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
