ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » IBM MQ Performance Monitoring » Logging Put/Get Message

Post new topic  Reply to topic Goto page 1, 2  Next
 Logging Put/Get Message « View previous topic :: View next topic » 
Author Message
matth
PostPosted: Wed Oct 23, 2019 9:19 am    Post subject: Logging Put/Get Message Reply with quote

Apprentice

Joined: 13 Feb 2007
Posts: 44
Location: Brazil

Hi:

After years of support on MQ infrastructure, I ever heard same question:

"How I can view logs on message input/output (put/get) on an queue?"

and I don´t known better response to this. I read something about amqsmon, but it treats statistics/flow, not an message alone, if I´m right. Before I plan to build something like message Exits binaries to configure in channels to log this information, I believe someone had this problem before and have any sollution.

Att,
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Oct 23, 2019 9:31 am    Post subject: Re: Logging Put/Get Message Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

matth wrote:
I believe someone had this problem before and have any sollution.


My personal solution to this problem - get whoever wants logs on messages sent & retrieved from a queue to have logging put into the sending and receiving applications.

When they start to splutter and insist that I'm responsible for all the messages travelling through the MQ topology, I agree that I'll log all the messages (or all the messages on a specific route) if the network admins agree to log all the packets travelling across the network (or all the packets on a specific route).

Because as an MQ admin I have as much interest in message content as the network admins have in packet content. We have monitoring to detect if something's not working, something's working slow and so forth but that's the limit of our involvement. If you wanted to log the HTTP calls you made or received you'd log them in the application. The same is true if you're using MQ as a transport protocol rather than TCP/IP.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
matth
PostPosted: Wed Oct 23, 2019 9:41 am    Post subject: Re: Logging Put/Get Message Reply with quote

Apprentice

Joined: 13 Feb 2007
Posts: 44
Location: Brazil

Vitor wrote:
matth wrote:
I believe someone had this problem before and have any sollution.


My personal solution to this problem - get whoever wants logs on messages sent & retrieved from a queue to have logging put into the sending and receiving applications.

When they start to splutter and insist that I'm responsible for all the messages travelling through the MQ topology, I agree that I'll log all the messages (or all the messages on a specific route) if the network admins agree to log all the packets travelling across the network (or all the packets on a specific route).

Because as an MQ admin I have as much interest in message content as the network admins have in packet content. We have monitoring to detect if something's not working, something's working slow and so forth but that's the limit of our involvement. If you wanted to log the HTTP calls you made or received you'd log them in the application. The same is true if you're using MQ as a transport protocol rather than TCP/IP.


I have an special problem in this case, because I manage MQ structure only on one side of solluction. Another side is connected throught network and banking technology and is totally closed for me/us, so any type of information/logs for this side is so hard to get. I only can use an message exit to collect message incomming on receiver channel, .i.e.

Att,
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Oct 23, 2019 10:51 am    Post subject: Re: Logging Put/Get Message Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

matth wrote:
I have an special problem in this case, because I manage MQ structure only on one side of solluction. Another side is connected throught network and banking technology and is totally closed for me/us


Not so special. Been there, done that and indeed am currently doing it for one branch of my topology.

(Currently working at a financial organization that has a B2B, MQ based link to a vendor that's black boxed to us)

Nothing you have said changes my opinion, which as I said above is only my opinion.

matth wrote:
I only can use an message exit to collect message incomming on receiver channel


There's been a lot of discussion on exits in this forum over the years and the general consensus is they're a bad idea. Again speaking personally, I take the view that if the answer is an exit you're asking the wrong question.

If you elect to use an exit for this (and it's going to be your decision), I think the biggest problem will be stopping the exit becoming a choke point. If you have messages arriving at any speed, or larger than a few Kb, the exit is rapidly going to become I/O bound. Among other problems.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
matth
PostPosted: Wed Oct 23, 2019 12:54 pm    Post subject: Re: Logging Put/Get Message Reply with quote

Apprentice

Joined: 13 Feb 2007
Posts: 44
Location: Brazil

Vitor wrote:
matth wrote:
I have an special problem in this case, because I manage MQ structure only on one side of solluction. Another side is connected throught network and banking technology and is totally closed for me/us


Not so special. Been there, done that and indeed am currently doing it for one branch of my topology.

(Currently working at a financial organization that has a B2B, MQ based link to a vendor that's black boxed to us)

Nothing you have said changes my opinion, which as I said above is only my opinion.

matth wrote:
I only can use an message exit to collect message incomming on receiver channel


There's been a lot of discussion on exits in this forum over the years and the general consensus is they're a bad idea. Again speaking personally, I take the view that if the answer is an exit you're asking the wrong question.

If you elect to use an exit for this (and it's going to be your decision), I think the biggest problem will be stopping the exit becoming a choke point. If you have messages arriving at any speed, or larger than a few Kb, the exit is rapidly going to become I/O bound. Among other problems.


Re-reading my original post I miss one word:

"How I can view logs of timestamps for message input/output (put/get) on an queue?"
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Wed Oct 23, 2019 2:42 pm    Post subject: Re: Logging Put/Get Message Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2492
Location: Melbourne, Australia

matth wrote:
Re-reading my original post I miss one word:
"How I can view logs of timestamps for message input/output (put/get) on an queue?"

Reiterating Vitor's recommendation, you should log this in the applications that are putting and getting messages. MQ is 20 years old, and does not directly provide this is as a feature, for obvious performance reasons.
_________________
Glenn
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Oct 23, 2019 3:02 pm    Post subject: Re: Logging Put/Get Message Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

matth wrote:
Re-reading my original post I miss one word:

"How I can view logs of timestamps for message input/output (put/get) on an queue?"

The MQMD of a message in a queue will contain the date/time the message was MQPUT to the queue.

IBM MQ doesn’t keep a log of messages MQPUT or MQGET.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Wed Oct 23, 2019 5:02 pm    Post subject: Re: Logging Put/Get Message Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

bruce2359 wrote:
matth wrote:
Re-reading my original post I miss one word:

"How I can view logs of timestamps for message input/output (put/get) on an queue?"

The MQMD of a message in a queue will contain the date/time the message was MQPUT to the queue.

IBM MQ doesn’t keep a log of messages MQPUT or MQGET.

No but the newer versions publish one in the application activity trace... you just gotta subscribe to it and finally do something with the data collected...
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
Andyh
PostPosted: Wed Oct 23, 2019 11:49 pm    Post subject: Reply with quote

Master

Joined: 29 Jul 2010
Posts: 237

Your initial post gives an idea of what you're trying to do, but not why you're trying to do it.
In MQ V9.1 there's typically very little additional admin or performance overhead in using linear logging, and the linear logs give you a complete history of the persistent message activity on the QMgr.
For example, if you want an audit history then by arranging not to re-use the log extents until the audit period has passed (or to archive the extents for that period) then the linear log extents are the most efficient mechanism you're likely to find.
Similarly for tracking down any "reported missing" persistent messages the recovery logs are a good source of information.
It takes a little bit of skill to read the logs (even after they've been formatted by dmpmqlog) but I suspect there are third party tools out there that take the strain if you're doing this sort of thing on a regular basis.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Oct 24, 2019 4:57 am    Post subject: Re: Logging Put/Get Message Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

matth wrote:
Re-reading my original post I miss one word:

"How I can view logs of timestamps for message input/output (put/get) on an queue?"


This doesn't change my advice by one word.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Oct 24, 2019 4:59 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

Andyh wrote:
Your initial post gives an idea of what you're trying to do, but not why you're trying to do it.




Andyh wrote:
It takes a little bit of skill to read the logs (even after they've been formatted by dmpmqlog) but I suspect there are third party tools out there that take the strain if you're doing this sort of thing on a regular basis.


Still sounds easier and cheaper to get the applications to do it.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu Oct 24, 2019 6:44 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

Vitor wrote:
Still sounds easier and cheaper to get the applications to do it.

Sounds to me like the OP is being asked to disprove a negative - that a message was (or was not) MQPUT to a queue, and MQGET from the queue. The presence of message image in the MQ log doesn't guarantee that the producer/consumer app successfully processed the message data.

"MQ has lost my message" is the siren call from end-users lamenting that their apps cannot account for lost or duplicate data. In my decades supporting MQ, I've yet to see evidence that MQ internals lost or duplicated messages.

I've got lots of stories where apps MQPUT the same message multiple times, OR counted a message before it was MQPUT, only to have the MQPUT fail.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Oct 24, 2019 6:52 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

bruce2359 wrote:
Vitor wrote:
Still sounds easier and cheaper to get the applications to do it.

Sounds to me like the OP is being asked to disprove a negative - that a message was (or was not) MQPUT to a queue, and MQGET from the queue. The presence of message image in the MQ log doesn't guarantee that the producer/consumer app successfully processed the message data.

"MQ has lost my message" is the siren call from end-users lamenting that their apps cannot account for lost or duplicate data. In my decades supporting MQ, I've yet to see evidence that MQ internals lost or duplicated messages.

I've got lots of stories where apps MQPUT the same message multiple times, OR counted a message before it was MQPUT, only to have the MQPUT fail.





bruce2359 wrote:
In my decades supporting MQ, I've yet to see evidence that MQ internals lost or duplicated messages.




Messages are more likely to be lost or duplicated by a user-written exit that's intended to log message details.....


_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
RogerLacroix
PostPosted: Thu Oct 24, 2019 10:14 am    Post subject: Reply with quote

Jedi Knight

Joined: 15 May 2001
Posts: 3252
Location: London, ON Canada

<Vendor_Plug>
Have a look at MQ Auditor. It tracks/audits all MQ API calls for all queues/topics (user selectable) of a queue manager. This includes MQGET & MQPUT calls.
</Vendor_Plug>

Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter
Back to top
View user's profile Send private message Visit poster's website
PeterPotkay
PostPosted: Thu Oct 24, 2019 10:17 am    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

bruce2359 wrote:

"MQ has lost my message" is the siren call from end-users lamenting that their apps cannot account for lost or duplicate data. In my decades supporting MQ, I've yet to see evidence that MQ internals lost or duplicated messages.

You don't need a tool to convince yourself or an MQ person of this fact. But having a tool to show the apps what they did or didn't do with MQ can save a lot of time during problem resolution.

bruce2359 wrote:

I've got lots of stories where apps MQPUT the same message multiple times, OR counted a message before it was MQPUT, only to have the MQPUT fail.

Wouldn't it be nice to have a tool that quickly shows the app that they did 2 puts, not one? Or the put failed? Or the put succeeded, with a MQMD.Expiry of 1?
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » IBM MQ Performance Monitoring » Logging Put/Get Message
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.