| Author |
Message
|
| matth |
 Posted: Wed Oct 23, 2019 9:19 am Post subject: Logging Put/Get Message |
 |
|
Apprentice
Joined: 13 Feb 2007
Posts: 44
Location: Brazil
|
Hi:
After years of support on MQ infrastructure, I ever heard same question:
"How I can view logs on message input/output (put/get) on an queue?"
and I don´t known better response to this. I read something about amqsmon, but it treats statistics/flow, not an message alone, if I´m right. Before I plan to build something like message Exits binaries to configure in channels to log this information, I believe someone had this problem before and have any sollution.
Att, |
|
| Back to top |
|
 |
| Vitor |
| Posted: Wed Oct 23, 2019 9:31 am Post subject: Re: Logging Put/Get Message |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| matth wrote: |
| I believe someone had this problem before and have any sollution. |
My personal solution to this problem - get whoever wants logs on messages sent & retrieved from a queue to have logging put into the sending and receiving applications.
When they start to splutter and insist that I'm responsible for all the messages travelling through the MQ topology, I agree that I'll log all the messages (or all the messages on a specific route) if the network admins agree to log all the packets travelling across the network (or all the packets on a specific route).
Because as an MQ admin I have as much interest in message content as the network admins have in packet content. We have monitoring to detect if something's not working, something's working slow and so forth but that's the limit of our involvement. If you wanted to log the HTTP calls you made or received you'd log them in the application. The same is true if you're using MQ as a transport protocol rather than TCP/IP.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| matth |
| Posted: Wed Oct 23, 2019 9:41 am Post subject: Re: Logging Put/Get Message |
|
|
Apprentice
Joined: 13 Feb 2007
Posts: 44
Location: Brazil
|
| Vitor wrote: |
| matth wrote: |
| I believe someone had this problem before and have any sollution. |
My personal solution to this problem - get whoever wants logs on messages sent & retrieved from a queue to have logging put into the sending and receiving applications.
When they start to splutter and insist that I'm responsible for all the messages travelling through the MQ topology, I agree that I'll log all the messages (or all the messages on a specific route) if the network admins agree to log all the packets travelling across the network (or all the packets on a specific route).
Because as an MQ admin I have as much interest in message content as the network admins have in packet content. We have monitoring to detect if something's not working, something's working slow and so forth but that's the limit of our involvement. If you wanted to log the HTTP calls you made or received you'd log them in the application. The same is true if you're using MQ as a transport protocol rather than TCP/IP. |
I have an special problem in this case, because I manage MQ structure only on one side of solluction. Another side is connected throught network and banking technology and is totally closed for me/us, so any type of information/logs for this side is so hard to get. I only can use an message exit to collect message incomming on receiver channel, .i.e.
Att, |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Oct 23, 2019 10:51 am Post subject: Re: Logging Put/Get Message |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| matth wrote: |
| I have an special problem in this case, because I manage MQ structure only on one side of solluction. Another side is connected throught network and banking technology and is totally closed for me/us |
Not so special. Been there, done that and indeed am currently doing it for one branch of my topology.
(Currently working at a financial organization that has a B2B, MQ based link to a vendor that's black boxed to us)
Nothing you have said changes my opinion, which as I said above is only my opinion.
| matth wrote: |
| I only can use an message exit to collect message incomming on receiver channel |
There's been a lot of discussion on exits in this forum over the years and the general consensus is they're a bad idea. Again speaking personally, I take the view that if the answer is an exit you're asking the wrong question.
If you elect to use an exit for this (and it's going to be your decision), I think the biggest problem will be stopping the exit becoming a choke point. If you have messages arriving at any speed, or larger than a few Kb, the exit is rapidly going to become I/O bound. Among other problems.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| matth |
| Posted: Wed Oct 23, 2019 12:54 pm Post subject: Re: Logging Put/Get Message |
|
|
Apprentice
Joined: 13 Feb 2007
Posts: 44
Location: Brazil
|
| Vitor wrote: |
| matth wrote: |
| I have an special problem in this case, because I manage MQ structure only on one side of solluction. Another side is connected throught network and banking technology and is totally closed for me/us |
Not so special. Been there, done that and indeed am currently doing it for one branch of my topology.
(Currently working at a financial organization that has a B2B, MQ based link to a vendor that's black boxed to us)
Nothing you have said changes my opinion, which as I said above is only my opinion.
| matth wrote: |
| I only can use an message exit to collect message incomming on receiver channel |
There's been a lot of discussion on exits in this forum over the years and the general consensus is they're a bad idea. Again speaking personally, I take the view that if the answer is an exit you're asking the wrong question.
If you elect to use an exit for this (and it's going to be your decision), I think the biggest problem will be stopping the exit becoming a choke point. If you have messages arriving at any speed, or larger than a few Kb, the exit is rapidly going to become I/O bound. Among other problems. |
Re-reading my original post I miss one word:
"How I can view logs of timestamps for message input/output (put/get) on an queue?" |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Wed Oct 23, 2019 2:42 pm Post subject: Re: Logging Put/Get Message |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| matth wrote: |
Re-reading my original post I miss one word:
"How I can view logs of timestamps for message input/output (put/get) on an queue?" |
Reiterating Vitor's recommendation, you should log this in the applications that are putting and getting messages. MQ is 20 years old, and does not directly provide this is as a feature, for obvious performance reasons.
_________________
Glenn |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Oct 23, 2019 3:02 pm Post subject: Re: Logging Put/Get Message |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| matth wrote: |
Re-reading my original post I miss one word:
"How I can view logs of timestamps for message input/output (put/get) on an queue?" |
The MQMD of a message in a queue will contain the date/time the message was MQPUT to the queue.
IBM MQ doesn’t keep a log of messages MQPUT or MQGET.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Oct 23, 2019 5:02 pm Post subject: Re: Logging Put/Get Message |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| bruce2359 wrote: |
| matth wrote: |
Re-reading my original post I miss one word:
"How I can view logs of timestamps for message input/output (put/get) on an queue?" |
The MQMD of a message in a queue will contain the date/time the message was MQPUT to the queue.
IBM MQ doesn’t keep a log of messages MQPUT or MQGET. |
No but the newer versions publish one in the application activity trace... you just gotta subscribe to it and finally do something with the data collected... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Andyh |
| Posted: Wed Oct 23, 2019 11:49 pm Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 239
|
Your initial post gives an idea of what you're trying to do, but not why you're trying to do it.
In MQ V9.1 there's typically very little additional admin or performance overhead in using linear logging, and the linear logs give you a complete history of the persistent message activity on the QMgr.
For example, if you want an audit history then by arranging not to re-use the log extents until the audit period has passed (or to archive the extents for that period) then the linear log extents are the most efficient mechanism you're likely to find.
Similarly for tracking down any "reported missing" persistent messages the recovery logs are a good source of information.
It takes a little bit of skill to read the logs (even after they've been formatted by dmpmqlog) but I suspect there are third party tools out there that take the strain if you're doing this sort of thing on a regular basis. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Oct 24, 2019 4:57 am Post subject: Re: Logging Put/Get Message |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| matth wrote: |
Re-reading my original post I miss one word:
"How I can view logs of timestamps for message input/output (put/get) on an queue?" |
This doesn't change my advice by one word.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Oct 24, 2019 4:59 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Andyh wrote: |
| Your initial post gives an idea of what you're trying to do, but not why you're trying to do it. |
| Andyh wrote: |
| It takes a little bit of skill to read the logs (even after they've been formatted by dmpmqlog) but I suspect there are third party tools out there that take the strain if you're doing this sort of thing on a regular basis. |
Still sounds easier and cheaper to get the applications to do it. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Oct 24, 2019 6:44 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Vitor wrote: |
| Still sounds easier and cheaper to get the applications to do it. |
Sounds to me like the OP is being asked to disprove a negative - that a message was (or was not) MQPUT to a queue, and MQGET from the queue. The presence of message image in the MQ log doesn't guarantee that the producer/consumer app successfully processed the message data.
"MQ has lost my message" is the siren call from end-users lamenting that their apps cannot account for lost or duplicate data. In my decades supporting MQ, I've yet to see evidence that MQ internals lost or duplicated messages.
I've got lots of stories where apps MQPUT the same message multiple times, OR counted a message before it was MQPUT, only to have the MQPUT fail.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Oct 24, 2019 6:52 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| bruce2359 wrote: |
| Vitor wrote: |
| Still sounds easier and cheaper to get the applications to do it. |
Sounds to me like the OP is being asked to disprove a negative - that a message was (or was not) MQPUT to a queue, and MQGET from the queue. The presence of message image in the MQ log doesn't guarantee that the producer/consumer app successfully processed the message data.
"MQ has lost my message" is the siren call from end-users lamenting that their apps cannot account for lost or duplicate data. In my decades supporting MQ, I've yet to see evidence that MQ internals lost or duplicated messages.
I've got lots of stories where apps MQPUT the same message multiple times, OR counted a message before it was MQPUT, only to have the MQPUT fail. |
| bruce2359 wrote: |
| In my decades supporting MQ, I've yet to see evidence that MQ internals lost or duplicated messages. |
Messages are more likely to be lost or duplicated by a user-written exit that's intended to log message details.....
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Thu Oct 24, 2019 10:14 am Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
<Vendor_Plug>
Have a look at MQ Auditor. It tracks/audits all MQ API calls for all queues/topics (user selectable) of a queue manager. This includes MQGET & MQPUT calls.
</Vendor_Plug>
Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Oct 24, 2019 10:17 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| bruce2359 wrote: |
"MQ has lost my message" is the siren call from end-users lamenting that their apps cannot account for lost or duplicate data. In my decades supporting MQ, I've yet to see evidence that MQ internals lost or duplicated messages.
|
You don't need a tool to convince yourself or an MQ person of this fact. But having a tool to show the apps what they did or didn't do with MQ can save a lot of time during problem resolution.
| bruce2359 wrote: |
I've got lots of stories where apps MQPUT the same message multiple times, OR counted a message before it was MQPUT, only to have the MQPUT fail. |
Wouldn't it be nice to have a tool that quickly shows the app that they did 2 puts, not one? Or the put failed? Or the put succeeded, with a MQMD.Expiry of 1?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
|
|
