ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General IBM MQ Support » How to track MQ transactions ?

Post new topic  Reply to topic Goto page Previous  1, 2, 3  Next
 How to track MQ transactions ? « View previous topic :: View next topic » 
Author Message
mqjeff
PostPosted: Thu Nov 03, 2016 7:41 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

You will have a window of missed messages, between when you try and see them, and when they have been consumed.

Because of that, and because of a lot of other issues, you should either make the applications log their own messages, or use pub/sub to copy messages for storing in a database.

Regardless, you should go back to the business team and ask why you need a 90 backup of data, and what kind of security measures need to be put in place for that data... encryption at rest? encryption in transit? etc, etc., etc
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Nov 03, 2016 7:47 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

mqjeff wrote:
Because of that, and because of a lot of other issues, you should either make the applications log their own messages, or use pub/sub to copy messages for storing in a database.


This is where I came in....

Vitor wrote:
Add code to the application.

_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Thu Nov 03, 2016 8:02 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

Vitor wrote:
mqjeff wrote:
Because of that, and because of a lot of other issues, you should either make the applications log their own messages, or use pub/sub to copy messages for storing in a database.


This is where I came in....

Vitor wrote:
Add code to the application.


Yes... and I did forget the quote and i agree, but I still think I was reemphasizing your point.
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Nov 03, 2016 8:37 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

mqjeff wrote:
Vitor wrote:
mqjeff wrote:
Because of that, and because of a lot of other issues, you should either make the applications log their own messages, or use pub/sub to copy messages for storing in a database.


This is where I came in....

Vitor wrote:
Add code to the application.


Yes... and I did forget the quote and i agree, but I still think I was reemphasizing your point.



_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Muhammad Haider
PostPosted: Thu Nov 03, 2016 9:50 am    Post subject: Reply with quote

Apprentice

Joined: 14 Mar 2011
Posts: 43

mqjeff wrote:
You will have a window of missed messages, between when you try and see them, and when they have been consumed.

Because of that, and because of a lot of other issues, you should either make the applications log their own messages, or use pub/sub to copy messages for storing in a database.

Regardless, you should go back to the business team and ask why you need a 90 backup of data, and what kind of security measures need to be put in place for that data... encryption at rest? encryption in transit? etc, etc., etc


For Pub/Sub I saw this link: http://www-01.ibm.com/support/docview.wss?uid=swg21574670

But you see it mentions:
"c) NOTE: Strictly speaking, the messages received by Q1 and Q2 are NOT truly identical. Even though they will have the same payload (text string 'TEST') and same characteristics, each message will have its own unique message-id."

So when original message id can not be maintained plus the subscriber q's will also be not maintaining the source queue name then pub/sub is not filling the objective.

Client wants to maintain the full message (header+body) for audit purpose so that if its client says that i sent you a message then he is able to find it in his logs somehow.

Trying API exit seems to be the next option?
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Nov 03, 2016 9:58 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

Muhammad Haider wrote:
Trying API exit seems to be the next option?


I'll warn you one more time about the dangers of exits, then let you burn your system to the ground in peace.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
tczielke
PostPosted: Thu Nov 03, 2016 12:13 pm    Post subject: Reply with quote

Guardian

Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA

This IBM document covers the Application Activity Trace more in detail -> http://www.ibm.com/developerworks/websphere/library/techarticles/1306_bushby/1306_bushby.html

One of the use cases that IBM provides for the Application Activity Trace is the following which seems to fit your need:

"Scenario 3. Using Application Activity Trace to keep an audit trail of messages"
_________________
Working with MQ since 2010.
Back to top
View user's profile Send private message
RogerLacroix
PostPosted: Thu Nov 03, 2016 2:33 pm    Post subject: Reply with quote

Jedi Knight

Joined: 15 May 2001
Posts: 3252
Location: London, ON Canada

Muhammad Haider wrote:
Client wants to maintain the full message (header+body) for audit purpose so that if its client says that i sent you a message then he is able to find it in his logs somehow.

Trying API exit seems to be the next option?

<Vendor_Speak>
Have you looked at MQ Message Replication (MQMR) because it was designed/created for exactly your problem. Side bonus: MQMR can set message expiry of each message, so that you don't have to worry about cleaning up the queue holding the replicated messages.

Now if you don't want to use a queue as a database then have a look at MQ Auditor. You can configure it to be a message tracker (monitor just MQGET & MQPUT/1 & grab the entire message payload). The audit records are written to disk (or queue) as a plain text CSV (comma separated values) file.

Note: Both MQ Message Replication and MQ Auditor are API exits.
</Vendor_Speak>

Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter
Back to top
View user's profile Send private message Visit poster's website
mqjeff
PostPosted: Fri Nov 04, 2016 3:53 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

Muhammad Haider wrote:

"c) NOTE: Strictly speaking, the messages received by Q1 and Q2 are NOT truly identical. Even though they will have the same payload (text string 'TEST') and same characteristics, each message will have its own unique message-id."


Is the message id *that* imiportant? or is the contents of the message, and any message properties it has, including other parts of the MQMD sufficient?

And again,
mqjeff wrote:
Regardless, you should go back to the business team and ask why you need a 90 backup of data, and what kind of security measures need to be put in place for that data... encryption at rest? encryption in transit? etc, etc., etc

You *really* need to address these questions. Make sure you have answers to them, even if the client leader you are addressing says you don't need to bother with it.

If you don't, they will come back to you and blame you for not taking care of it.
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
Muhammad Haider
PostPosted: Fri Nov 04, 2016 9:15 am    Post subject: Reply with quote

Apprentice

Joined: 14 Mar 2011
Posts: 43

Thanks everyone for your feedback.

Reason for 90 days backup as given by client is that they want to maintain it for Tracking. For example if the client sends a particular message with message id abc and the external party did not receive it then he should be able to tell from his records that atleast there was no issue with that message id from his side. So keeping track of Queue/Channel activity will later help him to find if there was or what was the issue at his side.

Pub/Sub is an ideal solution which I can atleast propose to the client but it will require code change as they are working with queues currently. But again, subscribed Queues will not retain actual Queue name and Msg Id.

Apart from the commercial tools available, I tried a support pack given below, today on my windows 64 bit and MQ v 8 and found it awesome but the issue is it was last updated on 22/11/12 for MQ v 7.5 plus there is not code in the downloadable package.
MA0W: WebSphere MQ API Trace
http://www-01.ibm.com/support/docview.wss?uid=swg24010343


Last edited by Muhammad Haider on Fri Nov 04, 2016 9:28 am; edited 2 times in total
Back to top
View user's profile Send private message
Muhammad Haider
PostPosted: Fri Nov 04, 2016 9:22 am    Post subject: Reply with quote

Apprentice

Joined: 14 Mar 2011
Posts: 43

Vitor wrote:
Muhammad Haider wrote:
Trying API exit seems to be the next option?


I'll warn you one more time about the dangers of exits, then let you burn your system to the ground in peace.


Could you kindly inform what are the dangers of using exits. Commercial tools being suggested here are also based on api exits. So are those tools also not recommendable? Or writing my own API exit is dangerous because it requires good knowledge and proper time for testing.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Fri Nov 04, 2016 9:36 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

Pub/Sub doesn't require altering code.

You can create a QAlias that points to a topic. You then create administrative subscriptions that point to the two queues in question.

If the customer needs to rely on the message id of an MQ message to find out whether a given customer received it, then the data in the message is wrong.

And, again. There are usually a large number of security and privacy issues with messages at rest or being archived in a database.
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
Vitor
PostPosted: Fri Nov 04, 2016 9:44 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

Muhammad Haider wrote:
writing my own API exit is dangerous because it requires good knowledge and proper time for testing.




Exits run as part of the MQ process they're linked with, in most architectures as part of the same thread. A badly written exit will slow performance and increase resources. A really badly written exit will bring the process (like the queue manager) to the ground in flames.

Exits are not user friendly, nor are they especially fault tolerant. Even their design is a skill; I've seen exits that write to disc and make database calls. Perfectly legal but imagine what that kind of I/O costs in terms of overhead for an API call, never mind what happens when the external resources are not available.

The commercial solutions mentioned (and I apologize to Roger for any implied and unintended slight) have been designed, written and tested by teams of knowledgeable people over a period of time, are bullet proof and importantly give access to those teams in the event of a problem.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Fri Nov 04, 2016 2:47 pm    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

mqjeff wrote:
Pub/Sub doesn't require altering code.

Usually doesn't. Depends how the app was written, or the queue topology.

Consider an app that opens the queue for putting and used every random MQOO option it could find. Ya know, "just in case". Was the app coded wrong? Yes. Can they switch to an Alias to Pub Sub with no changes? Nope.

Or 2 apps that meet at one queue, one puts, one gets. Should they both have been given alias queues to begin with? Debatable. But regardless, you ain't switching to Pub Sub without one of them changing something. And apps can't change, anything, ever, remember?


Vitor wrote:
The commercial solutions mentioned (and I apologize to Roger for any implied and unintended slight) have been designed, written and tested by teams of knowledgeable people over a period of time, are bullet proof and importantly give access to those teams in the event of a problem.

And, tested by all the other customers of that exit, so collectively, inadvertently, all the customers are helping each other testing the exit together.

You will not find many people walking the planet that know more about MQ exits than Roger of Capitalware. So do yourself a favor and consider a proper vendor solution when contemplating something as dark and scary as an API Exit. Its not hard to get some kind of exit cobbled together to finally work your one happy path test case. Its a whole other ball game to make it bullet proof when it comes to performance, security, stability, etc, etc, etc....
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
Muhammad Haider
PostPosted: Sun Nov 06, 2016 8:30 am    Post subject: Reply with quote

Apprentice

Joined: 14 Mar 2011
Posts: 43

PeterPotkay wrote:


Usually doesn't. Depends how the app was written, or the queue topology.

Consider an app that opens the queue for putting and used every random MQOO option it could find. Ya know, "just in case". Was the app coded wrong? Yes. Can they switch to an Alias to Pub Sub with no changes? Nope.

Or 2 apps that meet at one queue, one puts, one gets. Should they both have been given alias queues to begin with? Debatable. But regardless, you ain't switching to Pub Sub without one of them changing something. And apps can't change, anything, ever, remember?


Is there a way I can perform a GET operation on an alias queue whose base type is topic (by setting some alias queue, topic or subscription properties) ? So that I can read what another app has PUT on this queue (or published on the base topic) ?
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page Previous  1, 2, 3  Next Page 2 of 3

MQSeries.net Forum Index » General IBM MQ Support » How to track MQ transactions ?
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.