| Author |
Message
|
| viks |
 Posted: Tue Oct 15, 2013 9:30 pm Post subject: Tracking and archiving all messages |
 |
|
Newbie
Joined: 12 Jun 2013
Posts: 8
|
Hi,
We are having a scenario where we need to track and archive all the messages flowing through our queue manager. So for that I need to replicate all the messages to an another queue while the existing message transfer doesn't get affected. I've seen number options, but not sure which is the best ( or most used ). Of course since applications are already started using the environment, we don't want to change the publishing/consuming queues if possible.
While doing some research here the options that I came across are given below. I hope these are correct analysis. Correct me if I'm wrong.
1) using an API exit , MirorQ:
Advantages: No need to change the queue names. Easier to setup.
Disadvantage: Not sure MirrorQ is an ideal one to sue in production. Only found 32-bit version. Performance could be an issue
2) using MQ triggers
Advantages: No need to change the queue names.
Disadvantage: Performance could be an issue.
3) using pub/sub
( based on this thread : http://www.mqseries.net/phpBB2/viewtopic.php?t=51319 )
Advantages: More native MQ solution.
Disadvantages: Publishers need to change the queue name, message id will get changed after pub/sub.
We don't want to use a Broker just for this.
Please let me know if any other options are available. Also which is the better way to achieve this, in terms of performance.
I'm sure there are some organizations which does the same thing.
Appreciate your help on this. |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Tue Oct 15, 2013 9:56 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20697
Location: LI,NY
|
The fastest in performance would be for the consuming application to log each and every message received..., this being said, you'll have to prototype and take your own key measures... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Wed Oct 16, 2013 3:04 am Post subject: Re: Tracking and archiving all messages |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| viks wrote: |
| We don't want to use a Broker just for this. |
Why not ? What's wrong with using a Broker to perform what Brokers do ?
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Oct 16, 2013 3:49 am Post subject: Re: Tracking and archiving all messages |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| lancelotlinc wrote: |
| viks wrote: |
| We don't want to use a Broker just for this. |
Why not ? What's wrong with using a Broker to perform what Brokers do ? |
Perhaps they don't have broker at this site?
Perhaps they don't want to pay the licensing fee?
Who knows eh?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Oct 16, 2013 4:07 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7716
|
Have a look at Capitalware's Message Multiplexer
http://www.capitalware.biz/mmx_overview.html
| Quote: |
(MMX) application will get a message from a queue and output it to one or more queues. Context information is maintained across the message put(s). Basically, this program is a message cloner.
|
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| elkinsc |
| Posted: Wed Oct 16, 2013 4:42 am Post subject: Are the messages flowing thru a z/OS queues manager? |
|
|
Centurion
Joined: 29 Dec 2004
Posts: 138
Location: Indy
|
| And are they persistent? If so, you can use the log utility to extract them. this has the advantage of being async, and will not impact current message throughput. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Oct 16, 2013 4:44 am Post subject: Re: Are the messages flowing thru a z/OS queues manager? |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| elkinsc wrote: |
| And are they persistent? If so, you can use the log utility to extract them. this has the advantage of being async, and will not impact current message throughput. |
Is that the log utility that only comes with zOS?
Or is that some other log utility? |
|
| Back to top |
|
|
| elkinsc |
| Posted: Wed Oct 16, 2013 4:58 am Post subject: Yes of course |
|
|
Centurion
Joined: 29 Dec 2004
Posts: 138
Location: Indy
|
| I did ask if they were flowing thru z/OS, so I assumed that the operating system was 'given' for the remainder of the post. Stupid me |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Oct 16, 2013 5:36 am Post subject: Re: Yes of course |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| elkinsc wrote: |
| I did ask if they were flowing thru z/OS, so I assumed that the operating system was 'given' for the remainder of the post. Stupid me |
Here? or in another thjread by the same user? |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Oct 16, 2013 6:10 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9405
Location: US: west coast, almost. Otherwise, enroute.
|
Tracking and archiving are two different activities. What exactly do you mean by 'tracking?'
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Andyh |
| Posted: Wed Oct 16, 2013 9:59 am Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 237
|
The requirement stated here is rather vague and hence it's difficult to give good advice.
If this activity relates to message integrity, and hence only applies to persistent messages handled inside syncpoint, then these messages are guaranteed to be written to disk on the MQ recovery log at MQPUT time and their removal recorded at MQGET time.
If you're using distributed MQ with linear logging then it's possible the dmpmqlog command might enable support of your tracking and archiving requirements.
The recovery log records the physical queue file updates and as such, for example, matching MQGET's to MQPUT's isn't quite as simple as you might expect and doesn't directly identify the application consuming the message.
However all the information is there to demonstrate what messages were put to which queues and when, whether those messages were subsequently consumed, and what related artifacts were also updated. |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Wed Oct 16, 2013 2:08 pm Post subject: Re: Tracking and archiving all messages |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3254
Location: London, ON Canada
|
| viks wrote: |
| We are having a scenario where we need to track and archive all the messages flowing through our queue manager. |
As Peter mentioned, have a look at the FREE open source project called Message Multiplexer (MMX):
http://www.capitalware.biz/mmx_overview.html
Not only can it copy the message (including MQMD) to "n" other queues, it has an Archive flag, so that the messages get written to a 'archive' file.
Now, if you want more of the tracking side of MQ then have a look at MQ Auditor:
http://www.capitalware.biz/mqa_overview.html
Its whole purpose is the 5 Ws (who, what, where when & how) plus if you really want to, you can include the message data in the Audit information.
Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Wed Oct 16, 2013 2:36 pm Post subject: Re: Tracking and archiving all messages |
|
|
Jedi
Joined: 25 Mar 2003
Posts: 2498
Location: Melbourne, Australia
|
| viks wrote: |
Hi,
We are having a scenario where we need to track and archive all the messages flowing through our queue manager.
....Please let me know if any other options are available. Also which is the better way to achieve this, in terms of performance.
I'm sure there are some organizations which does the same thing.
Appreciate your help on this. |
MQ does not have any built in features to track and archive all messages. Any MQ "internal" or "band-aid" solution you can think of will have weaknesses in one or more areas.
It is far easier and more reliable for an application to write MQ messages to a database or an application log immediately after producing or consuming each message.
What are you going to do with the messages?
_________________
Glenn |
|
| Back to top |
|
|
| viks |
| Posted: Wed Oct 16, 2013 6:40 pm Post subject: |
|
|
Newbie
Joined: 12 Jun 2013
Posts: 8
|
Thanks PeterPotkay, RogerLacroix for the MMX. This is definitely something similar to what I'm looking for.
| bruce2359 wrote: |
| Tracking and archiving are two different activities. What exactly do you mean by 'tracking?' |
The requirement is to have a copy of all the messages passing through the queue manager in database plus the tracking details like, when the message arrived.
| gbaddeley wrote: |
It is far easier and more reliable for an application to write MQ messages to a database or an application log immediately after producing or consuming each message. |
In our case we have 100 or more producer and consumer applications, so it' easier to do this ourselves than asking each application to do it.
| gbaddeley wrote: |
What are you going to do with the messages?
|
There is an application which needs all these messages that is been archived, for certain calculation/processing.
I do agree that in theory this is more off a pub/sub requirement, but then we need to redesign the setup where the current applications at least needs to change their end points. So trying to achieve this with no or minimal changes on the connecting applications.
Thanks everyone for their feedback. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Oct 16, 2013 9:26 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| Quote: |
I do agree that in theory this is more off a pub/sub requirement, but then we need to redesign the setup where the current applications at least needs to change their end points. So trying to achieve this with no or minimal changes on the connecting applications.
|
What about using Alias Queues to avoid having all the clients change their endpoints?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
|
|
