Author |
Message
|
mqjeff |
Posted: Thu Nov 03, 2016 7:41 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 17447
|
You will have a window of missed messages, between when you try and see them, and when they have been consumed.
Because of that, and because of a lot of other issues, you should either make the applications log their own messages, or use pub/sub to copy messages for storing in a database.
Regardless, you should go back to the business team and ask why you need a 90 backup of data, and what kind of security measures need to be put in place for that data... encryption at rest? encryption in transit? etc, etc., etc _________________ chmod -R ugo-wx / |
|
Back to top |
|
|
Vitor |
Posted: Thu Nov 03, 2016 7:47 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
mqjeff wrote: |
Because of that, and because of a lot of other issues, you should either make the applications log their own messages, or use pub/sub to copy messages for storing in a database. |
This is where I came in....
Vitor wrote: |
Add code to the application. |
_________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
|
mqjeff |
Posted: Thu Nov 03, 2016 8:02 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 17447
|
Vitor wrote: |
mqjeff wrote: |
Because of that, and because of a lot of other issues, you should either make the applications log their own messages, or use pub/sub to copy messages for storing in a database. |
This is where I came in....
Vitor wrote: |
Add code to the application. |
|
Yes... and I did forget the quote and i agree, but I still think I was reemphasizing your point. _________________ chmod -R ugo-wx / |
|
Back to top |
|
|
Vitor |
Posted: Thu Nov 03, 2016 8:37 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
mqjeff wrote: |
Vitor wrote: |
mqjeff wrote: |
Because of that, and because of a lot of other issues, you should either make the applications log their own messages, or use pub/sub to copy messages for storing in a database. |
This is where I came in....
Vitor wrote: |
Add code to the application. |
|
Yes... and I did forget the quote and i agree, but I still think I was reemphasizing your point. |
_________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
|
Muhammad Haider |
Posted: Thu Nov 03, 2016 9:50 am Post subject: |
|
|
Apprentice
Joined: 14 Mar 2011 Posts: 43
|
mqjeff wrote: |
You will have a window of missed messages, between when you try and see them, and when they have been consumed.
Because of that, and because of a lot of other issues, you should either make the applications log their own messages, or use pub/sub to copy messages for storing in a database.
Regardless, you should go back to the business team and ask why you need a 90 backup of data, and what kind of security measures need to be put in place for that data... encryption at rest? encryption in transit? etc, etc., etc |
For Pub/Sub I saw this link: http://www-01.ibm.com/support/docview.wss?uid=swg21574670
But you see it mentions:
"c) NOTE: Strictly speaking, the messages received by Q1 and Q2 are NOT truly identical. Even though they will have the same payload (text string 'TEST') and same characteristics, each message will have its own unique message-id."
So when original message id can not be maintained plus the subscriber q's will also be not maintaining the source queue name then pub/sub is not filling the objective.
Client wants to maintain the full message (header+body) for audit purpose so that if its client says that i sent you a message then he is able to find it in his logs somehow.
Trying API exit seems to be the next option? |
|
Back to top |
|
|
Vitor |
Posted: Thu Nov 03, 2016 9:58 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
Muhammad Haider wrote: |
Trying API exit seems to be the next option? |
I'll warn you one more time about the dangers of exits, then let you burn your system to the ground in peace. _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
|
tczielke |
Posted: Thu Nov 03, 2016 12:13 pm Post subject: |
|
|
Guardian
Joined: 08 Jul 2010 Posts: 941 Location: Illinois, USA
|
|
Back to top |
|
|
RogerLacroix |
Posted: Thu Nov 03, 2016 2:33 pm Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001 Posts: 3258 Location: London, ON Canada
|
Muhammad Haider wrote: |
Client wants to maintain the full message (header+body) for audit purpose so that if its client says that i sent you a message then he is able to find it in his logs somehow.
Trying API exit seems to be the next option? |
<Vendor_Speak>
Have you looked at MQ Message Replication (MQMR) because it was designed/created for exactly your problem. Side bonus: MQMR can set message expiry of each message, so that you don't have to worry about cleaning up the queue holding the replicated messages.
Now if you don't want to use a queue as a database then have a look at MQ Auditor. You can configure it to be a message tracker (monitor just MQGET & MQPUT/1 & grab the entire message payload). The audit records are written to disk (or queue) as a plain text CSV (comma separated values) file.
Note: Both MQ Message Replication and MQ Auditor are API exits.
</Vendor_Speak>
Regards,
Roger Lacroix
Capitalware Inc. _________________ Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
Back to top |
|
|
mqjeff |
Posted: Fri Nov 04, 2016 3:53 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 17447
|
Muhammad Haider wrote: |
"c) NOTE: Strictly speaking, the messages received by Q1 and Q2 are NOT truly identical. Even though they will have the same payload (text string 'TEST') and same characteristics, each message will have its own unique message-id." |
Is the message id *that* imiportant? or is the contents of the message, and any message properties it has, including other parts of the MQMD sufficient?
And again,
mqjeff wrote: |
Regardless, you should go back to the business team and ask why you need a 90 backup of data, and what kind of security measures need to be put in place for that data... encryption at rest? encryption in transit? etc, etc., etc |
You *really* need to address these questions. Make sure you have answers to them, even if the client leader you are addressing says you don't need to bother with it.
If you don't, they will come back to you and blame you for not taking care of it. _________________ chmod -R ugo-wx / |
|
Back to top |
|
|
Muhammad Haider |
Posted: Fri Nov 04, 2016 9:15 am Post subject: |
|
|
Apprentice
Joined: 14 Mar 2011 Posts: 43
|
Thanks everyone for your feedback.
Reason for 90 days backup as given by client is that they want to maintain it for Tracking. For example if the client sends a particular message with message id abc and the external party did not receive it then he should be able to tell from his records that atleast there was no issue with that message id from his side. So keeping track of Queue/Channel activity will later help him to find if there was or what was the issue at his side.
Pub/Sub is an ideal solution which I can atleast propose to the client but it will require code change as they are working with queues currently. But again, subscribed Queues will not retain actual Queue name and Msg Id.
Apart from the commercial tools available, I tried a support pack given below, today on my windows 64 bit and MQ v 8 and found it awesome but the issue is it was last updated on 22/11/12 for MQ v 7.5 plus there is not code in the downloadable package.
MA0W: WebSphere MQ API Trace
http://www-01.ibm.com/support/docview.wss?uid=swg24010343
Last edited by Muhammad Haider on Fri Nov 04, 2016 9:28 am; edited 2 times in total |
|
Back to top |
|
|
Muhammad Haider |
Posted: Fri Nov 04, 2016 9:22 am Post subject: |
|
|
Apprentice
Joined: 14 Mar 2011 Posts: 43
|
Vitor wrote: |
Muhammad Haider wrote: |
Trying API exit seems to be the next option? |
I'll warn you one more time about the dangers of exits, then let you burn your system to the ground in peace. |
Could you kindly inform what are the dangers of using exits. Commercial tools being suggested here are also based on api exits. So are those tools also not recommendable? Or writing my own API exit is dangerous because it requires good knowledge and proper time for testing. |
|
Back to top |
|
|
mqjeff |
Posted: Fri Nov 04, 2016 9:36 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 17447
|
Pub/Sub doesn't require altering code.
You can create a QAlias that points to a topic. You then create administrative subscriptions that point to the two queues in question.
If the customer needs to rely on the message id of an MQ message to find out whether a given customer received it, then the data in the message is wrong.
And, again. There are usually a large number of security and privacy issues with messages at rest or being archived in a database. _________________ chmod -R ugo-wx / |
|
Back to top |
|
|
Vitor |
Posted: Fri Nov 04, 2016 9:44 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
Muhammad Haider wrote: |
writing my own API exit is dangerous because it requires good knowledge and proper time for testing. |
Exits run as part of the MQ process they're linked with, in most architectures as part of the same thread. A badly written exit will slow performance and increase resources. A really badly written exit will bring the process (like the queue manager) to the ground in flames.
Exits are not user friendly, nor are they especially fault tolerant. Even their design is a skill; I've seen exits that write to disc and make database calls. Perfectly legal but imagine what that kind of I/O costs in terms of overhead for an API call, never mind what happens when the external resources are not available.
The commercial solutions mentioned (and I apologize to Roger for any implied and unintended slight) have been designed, written and tested by teams of knowledgeable people over a period of time, are bullet proof and importantly give access to those teams in the event of a problem. _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
|
PeterPotkay |
Posted: Fri Nov 04, 2016 2:47 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001 Posts: 7717
|
mqjeff wrote: |
Pub/Sub doesn't require altering code.
|
Usually doesn't. Depends how the app was written, or the queue topology.
Consider an app that opens the queue for putting and used every random MQOO option it could find. Ya know, "just in case". Was the app coded wrong? Yes. Can they switch to an Alias to Pub Sub with no changes? Nope.
Or 2 apps that meet at one queue, one puts, one gets. Should they both have been given alias queues to begin with? Debatable. But regardless, you ain't switching to Pub Sub without one of them changing something. And apps can't change, anything, ever, remember?
Vitor wrote: |
The commercial solutions mentioned (and I apologize to Roger for any implied and unintended slight) have been designed, written and tested by teams of knowledgeable people over a period of time, are bullet proof and importantly give access to those teams in the event of a problem. |
And, tested by all the other customers of that exit, so collectively, inadvertently, all the customers are helping each other testing the exit together.
You will not find many people walking the planet that know more about MQ exits than Roger of Capitalware. So do yourself a favor and consider a proper vendor solution when contemplating something as dark and scary as an API Exit. Its not hard to get some kind of exit cobbled together to finally work your one happy path test case. Its a whole other ball game to make it bullet proof when it comes to performance, security, stability, etc, etc, etc.... _________________ Peter Potkay
Keep Calm and MQ On |
|
Back to top |
|
|
Muhammad Haider |
Posted: Sun Nov 06, 2016 8:30 am Post subject: |
|
|
Apprentice
Joined: 14 Mar 2011 Posts: 43
|
PeterPotkay wrote: |
Usually doesn't. Depends how the app was written, or the queue topology.
Consider an app that opens the queue for putting and used every random MQOO option it could find. Ya know, "just in case". Was the app coded wrong? Yes. Can they switch to an Alias to Pub Sub with no changes? Nope.
Or 2 apps that meet at one queue, one puts, one gets. Should they both have been given alias queues to begin with? Debatable. But regardless, you ain't switching to Pub Sub without one of them changing something. And apps can't change, anything, ever, remember?
|
Is there a way I can perform a GET operation on an alias queue whose base type is topic (by setting some alias queue, topic or subscription properties) ? So that I can read what another app has PUT on this queue (or published on the base topic) ? |
|
Back to top |
|
|
|