| Author |
Message
|
| tczielke |
 Posted: Fri Jan 15, 2016 12:16 pm Post subject: Application Activity Trace stability with Windows |
 |
|
Guardian
Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA
|
I hit an issue in 2013 where turning on/off activity tracing globally with the ALTER QMGR ACTVTRC option on the Windows platform would cause the Windows queue manager to hit an internal error and become unstable. I thought I had found what led to the issue (having an mqat.ini that was “UNIX” formatted and had lines that only ended with \n and not \r\n), but I just tried using activity tracing on Windows this week with a 7.5.0.4 queue manager and hit a similar issue with the queue manager receiving an internal error and becoming unstable with a “properly” formatted mqat.ini file.
I was curious. Has anyone else experienced anything similar with using activity tracing on Windows with the queue manager hitting an internal error and becoming unstable when you globally turn activity tracing on/off? Has anyone had the opposite experience of using the activity tracing a lot with Windows and had it always work smoothly?
_________________
Working with MQ since 2010. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Fri Jan 15, 2016 12:54 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Is there an FDC thrown at the first error?
What's the AMQERR message?
In general, this is obviously something you need to open a PMR for... presumably L3 knows how to trace tracing tools...
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| tczielke |
| Posted: Fri Jan 15, 2016 1:01 pm Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA
|
| mqjeff wrote: |
Is there an FDC thrown at the first error?
What's the AMQERR message?
In general, this is obviously something you need to open a PMR for... |
There were some FDC errors thrown and I will be opening a PMR for this sometime in the next few weeks, if I can recreate it again. I need to first schedule some time with some others so it would be ok to "break" things and see if I can recreate the issue with tracing. However, I was curious to see if anyone else had either similar or positive experiences with activity tracing and Windows. I mainly use activity tracing on Unix/Linux, and have never seen anything remotely like this. Activity tracing works like a champ on those platforms.
| mqjeff wrote: |
| presumably L3 knows how to trace tracing tools... |
funny 
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
| tczielke |
| Posted: Tue Jan 19, 2016 5:58 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA
|
| mqjeff wrote: |
Is there an FDC thrown at the first error?
What's the AMQERR message?
In general, this is obviously something you need to open a PMR for... presumably L3 knows how to trace tracing tools... |
I took a look at the Event Viewer Windows Log, and it looks like this was the first error reported by MQ. Looks like an Access Violation error with a queue manager agent process.
| Code: |
Faulting application name: amqzlaa0.exe, version: 7.500.400.14219, time stamp: 0x53e357c9
Faulting module name: amqxcs2.dll, version: 7.500.400.14219, time stamp: 0x53e35046
Exception code: 0xc0000005
Fault offset: 0x000a6857
Faulting process id: 0xafc
Faulting application start time: 0x01d14f4b67db5bb5
Faulting application path: C:\Program Files (x86)\IBM\WebSphere MQ\bin\amqzlaa0.exe
Faulting module path: C:\Program Files (x86)\IBM\WebSphere MQ\bin\amqxcs2.dll
Report Id: 42432f82-bba5-11e5-b2bd-005056be4091
|
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
| latachuphal |
| Posted: Thu Feb 11, 2016 2:59 am Post subject: |
|
|
Newbie
Joined: 05 Jan 2016
Posts: 8
|
Hi,
Even I am also facing similar issues while using MQ activity tracing on Windows platform. The moment I change something in mqat.ini file, my testing seems to be broken. I then start getting 2009 error - connection broken.
Were you able to solve your issue and if so could you please share.
I want to capture message data and so for that have to alter the default mqat.ini file and the moment I edit it, I start getting the issue.
Any pointers will be helpful.
I am getting below error.
Faulting application name: amqzlaa0.exe, version: 7.500.200.13179, time stamp: 0x51ccea5d
Faulting module name: ntdll.dll, version: 6.1.7601.19110, time stamp: 0x5684255b
Exception code: 0xc0000005
Fault offset: 0x0003bfd9
Faulting process id: 0x1aac
Faulting application start time: 0x01d164b56f8976a1
Faulting application path: C:\Program Files (x86)\IBM\WebSphere MQ\bin\amqzlaa0.exe
Faulting module path: C:\Windows\SysWOW64\ntdll.dll
Report Id: 3fb645ff-d0ac-11e5-9af3-54ee75123de3 |
|
| Back to top |
|
|
| tczielke |
| Posted: Thu Feb 11, 2016 5:14 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA
|
|
| Back to top |
|
|
| latachuphal |
| Posted: Thu Feb 11, 2016 9:22 pm Post subject: |
|
|
Newbie
Joined: 05 Jan 2016
Posts: 8
|
Thanks Tim.
Could you please give the details for the PMR raised with IBM for this.
Also, I saw your presentation and found it really good. I have also downloaded amqsactz code and will try that out to get useful formatting of data. |
|
| Back to top |
|
|
| tczielke |
| Posted: Fri Feb 12, 2016 5:12 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA
|
Hi latachuphal,
For the PMR, I did the runmqras command to collect all the logs, and L2 could not find a direct hit for the errors. It is currently under investigation by L3.
I am glad you found that trace session helpful. For using amqsactz, I have recently put it into the MH06 supportpac. I am not sure if you downloaded it from the MH06 supportpac or the Capitalware web site. I would recommend using the supportpac version, as it has more user doc in it. Functionally, amqsactz is currently the same between the Capitalware web site and the MH06 supportpac.
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
| latachuphal |
| Posted: Mon Feb 22, 2016 8:22 pm Post subject: |
|
|
Newbie
Joined: 05 Jan 2016
Posts: 8
|
Hi,
I am facing some more issues with application activity tracing on Windows. When I try triggering multiple transactions within same session of activity tracing enabled, I end up getting 2009 issue of broken connection and then MQ stops responding.
Flow for the Transaction is Java --> MQ--> MB. For one transaction log are getting generated properly but if I trigger second message, problem starts coming. Any idea there?
Also, If I add any new component in the flow like : Java(MQwrite) --> MQ--> MB --> Java(MQRead) , then also getting the same issue.
SO not able to do much with activity tracing.
Thanks. |
|
| Back to top |
|
|
| tczielke |
| Posted: Tue Feb 23, 2016 5:23 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA
|
When you get the 2009 issue, do you also see FDC errors being written that suggest an underlying issue with the queue manager?
I personally have become leery of using the application activity trace on Windows, due to several instability issues I have experienced with the queue manager. If it was me, I would just use strmqtrc to debug any kind of application issues on Windows, unless you have some development server where it is not a big deal if your queue manager encounters an internal error and becomes unstable.
The MH06 supportpac has an mqtrcfrmt tool that helps format and read a strmqtrc API trace for Windows. The MQTC session I mentioned earlier is helpful in going over how to use strmqtrc and the mqtrcfrmt tool.
http://www.mqtechconference.com/sessions_v2015/MQTC_v2015_Tracing.pdf
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
| latachuphal |
| Posted: Wed Feb 24, 2016 1:22 am Post subject: |
|
|
Newbie
Joined: 05 Jan 2016
Posts: 8
|
Thanks for the reponse.
I am working on a development server and so deleting and recreating QM is currently not an issue. Hence still trying out on Windows. I agree it is a lot unstable here and I have to struggle to make it work. Parallely trying out setup on Linux too.
Also, about strmqtrc, I could not use because I am trying to get an end to end picture of what all applications are connecting to MQ and their operations, but strmqtrc can be used by one application at one time.
Also I am getting the FDC with 2009 error. Attached below:
WebSphere MQ First Failure Symptom Report |
| ========================================= | |
| Major Errorcode :- xecF_E_UNEXPECTED_SYSTEM_RC |
| Minor Errorcode :- OK |
| Probe Type :- MSGAMQ6119 |
| Probe Severity :- 2 |
| Probe Description :- AMQ6109: An internal WebSphere MQ error has occurred.
|
| FDCSequenceNumber :- 0 |
| Comment1 :- Access Violation at address 0130A000 when writing |
|
| Back to top |
|
|
| tczielke |
| Posted: Wed Feb 24, 2016 5:25 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA
|
| latachuphal wrote: |
| Also, about strmqtrc, I could not use because I am trying to get an end to end picture of what all applications are connecting to MQ and their operations, but strmqtrc can be used by one application at one time. |
I am not sure exactly what you meant by "but strmqtrc can be used by one application at one time", but do note that the following strmqtrc command would run an api trace for all applications connected to the queue manager.
strmqtrc -m qmgr -t api
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
| latachuphal |
| Posted: Thu Feb 25, 2016 10:56 pm Post subject: |
|
|
Newbie
Joined: 05 Jan 2016
Posts: 8
|
Hi Tim,
I finally got it working on Windows now. 2 Issues I faced during this:
1. Formatting of the mqat.ini file. If I create the QM in command prompt, then I was not able to modify and run the activity trace and QM used to be unstable. If QM created through toolkit while broker creation or explorer, then I was able to run active trace for one run. But second run onwards it used to fail again with 2009 error.
2. There was an APAR(IT09496:) raised with IBM for this issue and fixed in 2 versions:
IT09496: WMQ 7.5: COMMAND SERVER MAY CRASH WITH ACTIVITY TRACE ENABLED.
MQ 7.5.0.6 ( which is not yet available)
MQ8.0.0.4
http://www-01.ibm.com/support/docview.wss?uid=swg1IT09496&myns=swgws&mynp=OCSSFKSJ&mync=R&cm_sp=swgws-_-OCSSFKSJ-_-R
I have to upgrade all my MQ and MB to 8.0.0.4 to make it work in sync. Now I am able to run the trace successfully on Windows.
Thanks.. |
|
| Back to top |
|
|
| tczielke |
| Posted: Fri Feb 26, 2016 5:09 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA
|
Hi latachuphal,
Thanks for the link. I am still waiting for IBM to investigate if the specific amqzlaa0 error that I encountered matches an APAR like the one you mentioned.
Thanks,
Tim
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
|
|
