ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » IBM MQ Performance Monitoring » Monitoring Processed And Old Messages

Post new topic  Reply to topic
 Monitoring Processed And Old Messages « View previous topic :: View next topic » 
Author Message
liorp
PostPosted: Mon Feb 10, 2020 2:53 am    Post subject: Monitoring Processed And Old Messages Reply with quote

Newbie

Joined: 10 Feb 2020
Posts: 4

Hello Everyone,

I would like to know if there is a way to monitoring the following:

1) No. of messages processed over MQ per Min.
2) Total processing time for no. of messages over MQ.

I would also like to know if it's possible to look for old massages, that have already been processed.

Thanks!
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Mon Feb 10, 2020 3:28 am    Post subject: Re: Monitoring Processed And Old Messages Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

liorp wrote:
Hello Everyone,

I would like to know if there is a way to monitoring the following:

1) No. of messages processed over MQ per Min.
2) Total processing time for no. of messages over MQ.

I would also like to know if it's possible to look for old massages, that have already been processed.

Thanks!

If a message has been processed it should be off the queue and there is no bringing back. MQ is not a DB. The messages are stateless!

There are tools for what you are asking. Look at the performance support pack MH04
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
liorp
PostPosted: Mon Feb 10, 2020 3:40 am    Post subject: Re: Monitoring Processed And Old Messages Reply with quote

Newbie

Joined: 10 Feb 2020
Posts: 4

fjb_saper wrote:
liorp wrote:
Hello Everyone,

I would like to know if there is a way to monitoring the following:

1) No. of messages processed over MQ per Min.
2) Total processing time for no. of messages over MQ.

I would also like to know if it's possible to look for old massages, that have already been processed.

Thanks!

If a message has been processed it should be off the queue and there is no bringing back. MQ is not a DB. The messages are stateless!

There are tools for what you are asking. Look at the performance support pack MH04



Hi fjb_saper, thanks for your response!
I know that MQ is not a DB, but let me rephrase, I'm not looking for the actual data, I'm just looking for a log that indicates that specific message has been processed successfully.
Back to top
View user's profile Send private message
Vitor
PostPosted: Mon Feb 10, 2020 6:52 am    Post subject: Re: Monitoring Processed And Old Messages Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

liorp wrote:
I'm just looking for a log that indicates that specific message has been processed successfully.


You need to be a little careful with your terms. The best that MQ can tell you is that a message has been removed from the queue by an application, and that the application committed that action. If (as an example) the application is coded in such a way that the read of the message is in a different transaction / unit of work to the processing of the message, it's perfectly possible for the message to be read off and then dropped on the floor.

Moving to your question. The queue manager can produce statistics on how many messages were added and removed from a queue or queues over given time periods. My worthy associate has mentioned a support pack that assists with this. Again, you speak of processing time; MQ is recording queue time. There's no indication within the system of how long the application took to process the message once it was read off. So MQ can tell you the average, max and minimum times a message was sitting and waiting over that time period but not how long it took to process.

Now the question of if a specific message has been processed. If I had $5 for every time this question has been debated in this forum, I'd be able to afford a small island somewhere

MQ does not log individual messages. MQ is designed to provide high speed queueing and as a design decision it doesn't waste I/O writing an audit log. Imaging if you have 1,000 queues on a given queue manager (and I've seen queue managers with 5 times that number), with 200-300 messages a second, and all of those not only have to be logged for persistence but audited. That's a lot of I/O, and a lot of disc file which then has to be managed as well as interrogated by some poor sap who has to scroll through a text file looking for some message content (because I bet the application that's complaining about the missing message doesn't know the IBM-generated message id that MQ uses as a key). Write it to a database instead and you immediately have the problem of what the queue manager's supposed to do if the connection to the database is lost.

The IBM supplied solution to your question are the confirmation of delivery and confirmation of arrival messages, which can be requested by the application when a message is put. However - if you don't trust that MQ will successfully delivery a message (because you want a log proving it) how can you trust that the CoD or CoA messages will arrive? What will you do if (after a decent period) no such message arrives, you perform whatever processing you do in that event and then (a few minutes later) the CoD message turns up because the network finished burping?

The better solution with an assured delivery system such as MQ is to ensure nothing is preventing it delivering. Look for XMITQ depths (there shouldn't be any), channels in RETRY (there shouldn't be any), messages on the DLQ (there shouldn't be any), ...

See the pattern?

So, one final question - what's the actual requirement for this log? Do you actually need to prove that a given message has arrived, when it arrived and when it left the queue or does management just want an audit log because it sounds like something you should have and they lack the technical nounce to ask for something useful?
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Mon Feb 10, 2020 7:45 am    Post subject: Re: Monitoring Processed And Old Messages Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

liorp wrote:
I'm just looking for a log that indicates that specific message has been processed successfully.

You are not the first to ask for an "audit log," and you won't be the last. Over time, more than a few of my clients requested such a thing, as a requirement of Sorbanes-Oxley (Soxley) Law. Soxley, and best (recommended) practice, requires that a transaction be repeatable (replay-able). MQ doesn't drive transactions, no more than a data bases drives transactions. MQ and db's can participate in a transaction.

MQ doesn't "process" messages; rather, applications process messages. MQ is like a pipe through which messages flow.

MQ has no idea if a message was processed successfully or otherwise. That responsibility rests entirely in the application.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Mon Feb 10, 2020 11:21 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

However...

MQ can capture statistics and accounting information that may be useful in auditing application activity (MQI calls), along with queue and channel activity.

MQ is not involved with business logic (calculating payroll, for example), and thus cannot determine if a message was processed successfully.

I suspect that you will be back shortly with management concerns about MQ losing messages - a subject discussed at great length here and the UV post site. MQ assures message delivery - no loss or duplication of messages.

I've been working with MQ since v2.1, and have not seen MQ lose a message. I have seen instances where developers wrote code that MQPUT the same, identical message multiple times. I've seen app code that failed to successfully MQPUT messages (the queue was full, but the code didn't catch the error). I've seen app code that browsed messages, and then neglected to destructively consume them - blaming MQ for this.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
liorp
PostPosted: Tue Feb 11, 2020 12:03 am    Post subject: Re: Monitoring Processed And Old Messages Reply with quote

Newbie

Joined: 10 Feb 2020
Posts: 4

Vitor wrote:
liorp wrote:
I'm just looking for a log that indicates that specific message has been processed successfully.


You need to be a little careful with your terms. The best that MQ can tell you is that a message has been removed from the queue by an application, and that the application committed that action. If (as an example) the application is coded in such a way that the read of the message is in a different transaction / unit of work to the processing of the message, it's perfectly possible for the message to be read off and then dropped on the floor.

Moving to your question. The queue manager can produce statistics on how many messages were added and removed from a queue or queues over given time periods. My worthy associate has mentioned a support pack that assists with this. Again, you speak of processing time; MQ is recording queue time. There's no indication within the system of how long the application took to process the message once it was read off. So MQ can tell you the average, max and minimum times a message was sitting and waiting over that time period but not how long it took to process.

Now the question of if a specific message has been processed. If I had $5 for every time this question has been debated in this forum, I'd be able to afford a small island somewhere

MQ does not log individual messages. MQ is designed to provide high speed queueing and as a design decision it doesn't waste I/O writing an audit log. Imaging if you have 1,000 queues on a given queue manager (and I've seen queue managers with 5 times that number), with 200-300 messages a second, and all of those not only have to be logged for persistence but audited. That's a lot of I/O, and a lot of disc file which then has to be managed as well as interrogated by some poor sap who has to scroll through a text file looking for some message content (because I bet the application that's complaining about the missing message doesn't know the IBM-generated message id that MQ uses as a key). Write it to a database instead and you immediately have the problem of what the queue manager's supposed to do if the connection to the database is lost.

The IBM supplied solution to your question are the confirmation of delivery and confirmation of arrival messages, which can be requested by the application when a message is put. However - if you don't trust that MQ will successfully delivery a message (because you want a log proving it) how can you trust that the CoD or CoA messages will arrive? What will you do if (after a decent period) no such message arrives, you perform whatever processing you do in that event and then (a few minutes later) the CoD message turns up because the network finished burping?

The better solution with an assured delivery system such as MQ is to ensure nothing is preventing it delivering. Look for XMITQ depths (there shouldn't be any), channels in RETRY (there shouldn't be any), messages on the DLQ (there shouldn't be any), ...

See the pattern?

So, one final question - what's the actual requirement for this log? Do you actually need to prove that a given message has arrived, when it arrived and when it left the queue or does management just want an audit log because it sounds like something you should have and they lack the technical nounce to ask for something useful?



Hi Vitor,
First, thank you for your detailed answer!
The requirement actually was to "prove" to a certain client that massages have been indeed arrived.
Anyway, I fully understand your explanation and will definitely try the support pack you have mentioned.
Hope you'll get enough money for your small island
Back to top
View user's profile Send private message
liorp
PostPosted: Tue Feb 11, 2020 12:07 am    Post subject: Reply with quote

Newbie

Joined: 10 Feb 2020
Posts: 4

bruce2359 wrote:
However...

MQ can capture statistics and accounting information that may be useful in auditing application activity (MQI calls), along with queue and channel activity.

MQ is not involved with business logic (calculating payroll, for example), and thus cannot determine if a message was processed successfully.

I suspect that you will be back shortly with management concerns about MQ losing messages - a subject discussed at great length here and the UV post site. MQ assures message delivery - no loss or duplication of messages.

I've been working with MQ since v2.1, and have not seen MQ lose a message. I have seen instances where developers wrote code that MQPUT the same, identical message multiple times. I've seen app code that failed to successfully MQPUT messages (the queue was full, but the code didn't catch the error). I've seen app code that browsed messages, and then neglected to destructively consume them - blaming MQ for this.


Thank you Bruce!
Back to top
View user's profile Send private message
RogerLacroix
PostPosted: Thu Feb 13, 2020 3:09 pm    Post subject: Re: Monitoring Processed And Old Messages Reply with quote

Jedi Knight

Joined: 15 May 2001
Posts: 3252
Location: London, ON Canada

liorp wrote:
I'm just looking for a log that indicates that specific message has been processed successfully.

<Vendor_Plug>

You should have a look at MQ Auditor. By default, it can log every MQ API call made by local and remote applications. You can trim it back to just gets and puts.

</Vendor_Plug>

Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter
Back to top
View user's profile Send private message Visit poster's website
bruce2359
PostPosted: Thu Feb 13, 2020 3:23 pm    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

Perhaps I misunderstood the OP, but to me "processed successfully" means "met business requirements," as in accounts balanced, amounts transferred, db's updated, ... that kind of thing.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Thu Feb 13, 2020 5:47 pm    Post subject: Re: Monitoring Processed And Old Messages Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

liorp wrote:

The requirement actually was to "prove" to a certain client that massages have been indeed arrived.

_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu Feb 13, 2020 7:00 pm    Post subject: Re: Monitoring Processed And Old Messages Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

PeterPotkay wrote:
liorp wrote:

The requirement actually was to "prove" to a certain client that massages have been indeed arrived.

I must have nodded off while reading ...
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Thu Sep 21, 2023 3:38 pm    Post subject: Re: Monitoring Processed And Old Messages Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2492
Location: Melbourne, Australia

liorp wrote:
...
I know that MQ is not a DB, but let me rephrase, I'm not looking for the actual data, I'm just looking for a log that indicates that specific message has been processed successfully.

Application processing of message data occurs outside the responsibility of MQ, so you would be looking for an application log that records this processing. eg. logs the data after a successful MQGET then logs the outcome of its processing, including reasons for failure and rollback.
_________________
Glenn
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » IBM MQ Performance Monitoring » Monitoring Processed And Old Messages
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.