| Author |
Message
|
| kordi |
 Posted: Tue Mar 25, 2014 2:04 pm Post subject: Unknown process consuming messages from queue |
 |
|
Centurion
Joined: 28 May 2012
Posts: 146
Location: PL
|
Hi,
I have UAT Queue Manager which is used by many applications, have many different SVRCONN and pretty much queues. I received request to check one queue because one application is not receiving all replies for their messages. Sometimes it got all replies, sometimes it got only few.
I tried to check queue status with dis qs(queue) type(handle) all but what I got is just two IP of remote Queue Manager conected to this queue by RCVR.
I assume that application connects every n seconds, opens connection, checks queue, disconnects, it may take only few ms, so it is really difficult to check it with queue status.
I asked administrator of remote QMGR to change destination queue for replies and application got all replies every time, so this is not issue of remote Queue Manager. I checked also DLQ with no sign of lost messages.
Do you have idea how I can check what other IP may consuming messages from local queue? Is there any all the time running monitoring which can detect and store all IP connecting to local queue?
I can disable getting from queue and wait for complains but it might take too much time.
Thanks in advance for your response. |
|
| Back to top |
|
 |
| Tibor |
| Posted: Wed Mar 26, 2014 1:36 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
| You can use BlockIP2 security exit for logging all the channel activity. |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Mar 26, 2014 1:49 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
You can also enable accounting stats on the queue and use MS0P to view them, this will show the application name.
Another option with v7.5 is the MQ activity trace.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| kordi |
| Posted: Wed Mar 26, 2014 11:57 am Post subject: |
|
|
Centurion
Joined: 28 May 2012
Posts: 146
Location: PL
|
Thanks a lot for your replies.
Using BlockIP doesn't make sense in this case because this queue manager is used by many applications with many SVRCONN. I need to focus only on one queue checking who is stealing messages.
Accounting is a good idea, I have to check it together with MO71.
Yesterday I wrote bash script with infinit loop which run dis qs(queue) type(handle) all all the time. I found out what IP it is but accounting seems to be much better solution.
Once again, thanks a lot guys.
Cheers |
|
| Back to top |
|
|
| shashivarungupta |
| Posted: Wed Mar 26, 2014 2:15 pm Post subject: |
|
|
Grand Master
Joined: 24 Feb 2009
Posts: 1343
Location: Floating in space on a round rock.
|
| kordi wrote: |
| .. script with infinit loop which run dis qs(queue) type(handle) all all the time... |
script with infinite loop, anytime it is a bad idea. 
cronjob/scheduler could be used if you wanted to use your script with in finite interval.. and if your script generates a log in a text file for the history of evidences. 
but yes, there are other tools which can be used as mentioned by zpat. 
_________________
*Life will beat you down, you need to decide to fight back or leave it. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Mar 26, 2014 2:26 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| shashivarungupta wrote: |
| kordi wrote: |
| .. script with infinit loop which run dis qs(queue) type(handle) all all the time... |
script with infinite loop, anytime it is a bad idea. |
Why exactly?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| tczielke |
| Posted: Wed Mar 26, 2014 6:16 pm Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
Someone earlier in the thread mentioned the Application Activity Trace (AAT). I was thinking the same thing too when I first read the post. One note, is that it is also available at 7.1. Please note, it does have a very significant bug in the earlier releases of 7.1 and 7.5 where the queue manager can get an internal error and become unstable if you use the AAT.. However, that was resolved I believe with the later fix packs in 2013. Below is a link to a very good primer on how to use this tool. I just used this tool today on a 7.5.0.2 queue manager to help review a multi-threaded java program for a developer, and it worked great.
http://www.ibm.com/developerworks/websphere/library/techarticles/1306_bushby/1306_bushby.html |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Mar 26, 2014 10:55 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
MO71 does not format accounting messages in the same way MS0P (an extension to MQ explorer) does.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| Tibor |
| Posted: Thu Mar 27, 2014 12:39 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
| zpat wrote: |
| Another option with v7.5 is the MQ activity trace. |
Just a question: how can you extract the client information from the activity trace? I've just skimmed the manual and the related article, but I didn't find any channel related information in the output. |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Mar 27, 2014 12:47 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
I'm not an expert in the AAT, just wanted to bring it to your attention.
Good article from Emma though (who was visiting us only the other week as our outgoing AVP rep).
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| tczielke |
| Posted: Thu Mar 27, 2014 12:46 pm Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
Here is an example of AAT output (from the amqsact tool) that shows channel and client information on the connect. Some data has been obfuscated:
MonitoringType: MQI Activity Trace
Correl_id:
00000000: 414D 51xx xxxx xxxx xxxx xxxx xxxx xxxx 'AMQxxxxxxxxxxxxx'
00000010: 5334 6A16 2000 9801 'S4j. ... '
QueueManager: 'xxxxxxxx'
Host Name: 'srvr1'
IntervalStartDate: '2014-03-27'
IntervalStartTime: '15:40:46'
IntervalEndDate: '2014-03-27'
IntervalEndTime: '15:40:46'
CommandLevel: 750
SeqNumber: 0
ApplicationName: 'amqscnxc'
Application Type: MQAT_UNIX
ApplicationPid: 12727
UserId: 'user1'
API Caller Type: MQXACT_EXTERNAL
API Environment: MQXE_MCA_SVRCONN
Channel Name: 'CLIENT.TO.SERVER'
ConnName: '20.200.16.222'
Channel Type: MQCHT_SVRCONN
Application Function: ''
Appl Function Type: MQFUN_TYPE_UNKNOWN
Trace Detail Level: 2
Trace Data Length: 0
Pointer size: 8
Platform: MQPL_UNIX
MQI Operation: 0
Operation Id: MQXF_CONNX
ApplicationTid: 10
OperationDate: '2014-03-27'
OperationTime: '15:40:46'
ConnectionId:
00000000: 414D 51xx xxxx xxxx xxxx xxxx xxxx xxxx 'AMQxxxxxxxxxxxxx'
00000010: 5334 6A16 2000 FFFFFF9801 'S4j. ... '
QueueManager: 'xxxxxxxx'
Completion Code: MQCC_OK
Reason Code: 0
Connect Options: 256 |
|
| Back to top |
|
|
| Tibor |
| Posted: Thu Mar 27, 2014 2:19 pm Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
Thanks  |
|
| Back to top |
|
|
| kordi |
| Posted: Fri May 09, 2014 11:56 pm Post subject: |
|
|
Centurion
Joined: 28 May 2012
Posts: 146
Location: PL
|
Thanks a lot guys for very comprehensive replies.
Trace or accounting stats have however one disadvantage. They show application name rather than IP and it is up to developer to name properly its app. So in many scenarios I would have "Java application for WebSphere MQ" app name. Therefore sometimes dis chs is better, even in infinite loop  |
|
| Back to top |
|
|
|
|
