| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| amqzlaa0: trace thread to connection |
« View previous topic :: View next topic » |
| Author |
Message
|
| mgrx |
 Posted: Thu Nov 26, 2015 11:28 pm Post subject: amqzlaa0: trace thread to connection |
 |
|
Novice
Joined: 01 Oct 2015
Posts: 20
|
Hi,
I have a problem with a queue manager fronting some customers, where amqzlaa0-processes increase in a linear manner. Im pretty sure this is a bug: http://www-01.ibm.com/support/docview.wss?uid=swg1IV46353 . We are obviously going to upgrade the queue manager to a version were the problem is fixed, however we cannot do that in some time.
| Quote: |
PROBLEM SUMMARY:
If a transaction manager issued an xa_open() without issuing an
xa_close() on a channel with SHARECNV set to something other
than zero, then MQ was not implicitly issuing an xa_close() if
the channel terminated unexpectedly. This prevented threads
inside amqzlaa0 processes from ending and therefore caused a
resource leak.
|
What I would like to try is to figure out which customer/connection/server-connection that is causing this to happen, but Im not sure its possible to actually backtrace what thread in the amqzlaa0 that is bound to a socket, most likely the socket is allready gone?
| Code: |
# ps -mo THREAD -p 61145314
USER PID PPID TID ST CP PRI SC WCHAN F TT BND COMMAND
mqm 61145314 6488180 - A 0 60 78 * 240001 - - amqzlaa0 -mQM123 -fip108
- - - 109183405 S 0 60 1 f1000a00e06f8158 410400 - - -
- - - 114819401 S 0 60 1 f1000a00e06fb360 410400 - - -
- - - 12255757 S 0 60 1 f1000a00e06e9478 410400 - - -
- - - 109970359 S 0 60 1 f1000a00e06fe518 410400 - - -
- - - 29296257 Z 0 60 1 - c00001 - - -
- - - 91489811 S 0 60 1 f1000a00e06e9450 410400 - - -
- - - 112134385 S 0 60 1 f1000a00e06e5900 410400 - - -
- - - 13961533 S 0 60 1 f1000a00e06fe608 410400 - - -
- - - 108071235 S 0 60 1 f1000a00e06e5860 410400 - - -
- - - 40503885 S 0 60 1 f1000a00e0897158 410400 - - -
- - - 53479947 S 0 60 1 f1000f0a10533040 8430400 - - -
- - - 114559697 S 0 60 1 f1000a00e06ee090 410400 - - -
- - - 91818953 S 0 60 1 f1000a00e06e95b8 410400 - - -
- - - 102501139 S 0 60 1 f1000a00e06e81f8 410400 - - -
|
Take this for example, could i figure out that a thread is related to an "physical" connection?
Tried to use the amqldmpa command (amqldmpa -m QM123 -c K -f/tmp/qm123.txt), however that didnt help me..
This is MQ 7.0.1.5 on AIX 7.1,
appreciate any insight! |
|
| Back to top |
|
 |
| tczielke |
| Posted: Fri Nov 27, 2015 8:37 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
From what I have read, files (which would also include sockets) are tied to a process, and not to a specific thread for Unix based systems. So you could see what sockets a process has opened, but not what sockets a specific thread has opened. This has to do with a multi-threaded process sharing things like opened files, the virtual memory, etc. at the process level.
My understanding is that you are trying to track down which channel instances are doing an xa_open but not an xa_close. You might have to do some tracing (i.e. strmqtrc -m qmgr -t api -p amqrmppa) to track that down, as I don't believe MQ accounting and statistics would track xa calls.
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
| mgrx |
| Posted: Fri Nov 27, 2015 9:28 am Post subject: |
|
|
Novice
Joined: 01 Oct 2015
Posts: 20
|
| tczielke wrote: |
So you could see what sockets a process has opened, but not what sockets a specific thread has opened. This has to do with a multi-threaded process sharing things like opened files, the virtual memory, etc. at the process level.
|
Yes, that seems correct and what the discussions with a colleague resulted in aswell.
| tczielke wrote: |
My understanding is that you are trying to track down which channel instances are doing an xa_open but not an xa_close. You might have to do some tracing (i.e. strmqtrc -m qmgr -t api -p amqrmppa) to track that down, as I don't believe MQ accounting and statistics would track xa calls. |
OK, gonna give that a try. Just hope I don't drown in trace 
Thanks for response, appreciate it! |
|
| Back to top |
|
|
| tczielke |
| Posted: Fri Nov 27, 2015 10:09 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
The first half of this MQTC session on tracing was written to be a tutorial on how to use and understand MQ strmqtrc API tracing -> http://www.mqtechconference.com/sessions_v2015/MQTC_v2015_Tracing.pdf
The session also mentions the MH06 supportpac which is a Trace Tools supportpac that comes with the mqtrcfrmt program that can help read strmqtrc API traces. The executables come with Linux x86, Solaris SPARC, and Windows. mqtrcfrmt was never tested on an AIX strmqtrc API trace, but it is possible that it could format that type of trace with the "-o unix" switch, as long as AIX follows the same trace format as Linux and Solaris (which were identical).
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Nov 27, 2015 3:59 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Looking at the fact that your post mentions that one of the reasons is a shareconv on the channel gt 0, why not up the max channels and set shareconv to 0 and see if that fixes the problem.
Anyhow each application should have it's own SVRCONN channel. This will make it much more easier to spot. You can also set the max conn from a single host. This too could point to the culprit. But if I fear the culprit is truly the MQ code at the level you are running (need a fixpack applied) the temporary reprieve would be to try and set shareconv to 0 (and up the max channels)...
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mgrx |
| Posted: Sat Nov 28, 2015 3:53 am Post subject: |
|
|
Novice
Joined: 01 Oct 2015
Posts: 20
|
| fjb_saper wrote: |
Looking at the fact that your post mentions that one of the reasons is a shareconv on the channel gt 0, why not up the max channels and set shareconv to 0 and see if that fixes the problem.
Anyhow each application should have it's own SVRCONN channel.
|
Well, that would be a way.. I have considered it, but we have around 100+ SVRCONN (each one a different customer). Feels a bit harsh to change every single one, that would also impact performance as I understand it. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Nov 28, 2015 5:22 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| mgrx wrote: |
| fjb_saper wrote: |
Looking at the fact that your post mentions that one of the reasons is a shareconv on the channel gt 0, why not up the max channels and set shareconv to 0 and see if that fixes the problem.
Anyhow each application should have it's own SVRCONN channel.
|
Well, that would be a way.. I have considered it, but we have around 100+ SVRCONN (each one a different customer). Feels a bit harsh to change every single one, that would also impact performance as I understand it. |
My understanding is it might improve performance, unless your data is sparse...
In fact the recommended setting for MQ 8 is sharecnv=1
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
