| Author |
Message
|
| lyntongrice |
 Posted: Tue Oct 25, 2011 10:50 pm Post subject: Is "Message Identifier Bytes" in MQMD unique per m |
 |
|
Acolyte
Joined: 26 Sep 2010
Posts: 70
|
Hi there,
I have an application that takes messages off queues using transactions and would like to know if the "message identifier bytes" in the MQMD is guaranteed to be unique per message? I am assuming this is populated by WMQ by default if not set by the user?
I want to treat it as a GUID type reference to guarantee that my application does not process messages twice...
Thanks for the help
Lynton |
|
| Back to top |
|
 |
| rekarm01 |
| Posted: Wed Oct 26, 2011 12:20 am Post subject: Re: Is "Message Identifier Bytes" in MQMD unique p |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
|
| Back to top |
|
|
| exerk |
| Posted: Wed Oct 26, 2011 12:26 am Post subject: Re: Is "Message Identifier Bytes" in MQMD unique p |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| lyntongrice wrote: |
| I want to treat it as a GUID type reference to guarantee that my application does not process messages twice. |
Why would your application process messages twice? Surely if the processing fails your application will roll back everything?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Oct 26, 2011 4:42 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Used properly, MQ will never deliver the same message twice. Even with syncpoint rollback, the backout count is incremented so that you are aware of this.
Once and once only is the basic maxim of MQ. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Oct 26, 2011 5:02 am Post subject: Re: Is "Message Identifier Bytes" in MQMD unique p |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| lyntongrice wrote: |
| I want to treat it as a GUID type reference to guarantee that my application does not process messages twice... |
You're better off coding your application to properly exploit WMQ's assured once-and-once-only delivery mechanisms. You're paying a lot of license money for that IBM wheel so why reinvent it?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| lyntongrice |
| Posted: Wed Oct 26, 2011 11:11 pm Post subject: |
|
|
Acolyte
Joined: 26 Sep 2010
Posts: 70
|
Hi there,
Thanks for your comments......but let me paint a quick scenario of what i am concerned about...
1. Application takes message off queue using MQGET with SYNCPOINT....
2. Application processes message and sends it to target system...and commit work happens in target system....
3. Now network goes down .....and application had not yet commited it off the queue....so WMQ rolls it back...
4. Application picks up the same message again on next round and now we have a duplicate ...
I am not wanting to implement XA in this scenario as the target system is SAP.....
That is why I am wanting to know if the "message identifier bytes" is ALWAYS unique by default so that I can implement my own internal "guarantee" that the message will only be processed once.....
Am I correct in what I am saying here?
Thanks for the help 
Lynton |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Oct 26, 2011 11:25 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Do the MQPUT in the same SYNCPOINT (UoW) as the MQGET.
You can detect rollback by inspecting the MQMD.Back value and should always do this with syncpointed gets,
If you are unable to two phase commit and must use non-MQ resources in your original transaction then you should decide whether to risk data loss or data duplication.
If it's easier to detect duplication, then you could do what you suggested but only after using MQ's transactional features to the maximum possible extent.
Data loss can be detected using sequence numbers, if you have them. In fact data duplication can also use this method. Ideally the incoming message would have a sequence number. If it doesn't you could potentially increment and store such a number in a MQ message that you get and put in the same transaction to a queue that is used just for this purpose.
Downstream transactions are the responsibility of the downstream system.
Once you commit - your transaction is over.
The uniqueness of the queue manager generated MQMD.msgid is documented (App Prof Ref or Guide or infocentre). Essentially it will be unique for messages generated on that queue manager.
But IBM can't stop you having two queue managers with the same name!
Last edited by zpat on Thu Oct 27, 2011 12:12 am; edited 1 time in total |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Oct 27, 2011 12:09 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
It took me 20 seconds to find this.
| Quote: |
A MsgId generated by the queue manager consists of a 4-byte product identifier (AMQ␢ or CSQ␢ in either ASCII or EBCDIC, where ␢ represents a blank), followed by a product-specific implementation of a unique string. In WebSphere® MQ this contains the first 12 characters of the queue-manager name, and a value derived from the system clock. All queue managers that can intercommunicate must therefore have names that differ in the first 12 characters, in order to ensure that message identifiers are unique. The ability to generate a unique string also depends on the system clock not being changed backward. To eliminate the possibility of a message identifier generated by the queue manager duplicating one generated by the application, the application must avoid generating identifiers with initial characters in the range A through I in ASCII or EBCDIC (X'41' through X'49' and X'C1' through X'C9'). However, the application is not prevented from generating identifiers with initial characters in these ranges.
|
The putting application should clear the MQMD.msgid field in its storage area after each MQPUT to avoid the risk of this being re-used on the next MQPUT instead of being re-generated by the queue manager. Failure to do this is why you may see a series of identical msgids on queues.
For this reason alone I would be cautious in relying on it for application purposes unless you are confident of the programming in the sender. However you might use the put date/time to make it more likely to be unique. But incrementing your own sequence numbers seems like a better approach to me. |
|
| Back to top |
|
|
| lyntongrice |
| Posted: Thu Oct 27, 2011 2:22 am Post subject: |
|
|
Acolyte
Joined: 26 Sep 2010
Posts: 70
|
Hi there,
Thanks for the comments, and I would have liked to use the PUT date / time but I see there are no "microseconds"...;-( Or at least not that I know of?
I have no control of the sending application....
I will brainstorm this more and will keep you posted on what I decide to do.....
Thanks again
Lynton |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Oct 27, 2011 4:02 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
You could concatenate the put date/time to the msgid (I didn't mean use instead of it)....
Get the sender to send a number of messages and see if the msgids are different, it should be obvious.
However bear in mind that MQ re-using the msgid is only going to happen for multiple MQPUTs in the same execution of the sender's program (and where they don't clear the msgid field in-between). |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Oct 27, 2011 4:21 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9470
Location: US: west coast, almost. Otherwise, enroute.
|
| lyntongrice wrote: |
... so that I can implement my own internal "guarantee" that the message will only be processed once.....
Am I correct in what I am saying here?
|
Search here for "well-behaved application".
WMQ assures message delivery. With MQPMO_NEW_MSGID, the putting qmgr will generate a unique (to this qmgr) message id. It is possible (as my colleague stated above) that an application can issue two MQPUTs of the same exact message (MQMD and app data), or a message with a new MsgID with duplicated application data. Both are application coding errors.
As you've described it, your application encompasses three separate and distinct WMQ units of work: 1) a request message is created; 2) it is transported across the network to another qmgr; 3) the message is consumed.
If you have no control of the requesting (sending) application, then all that you can do is keep a DB of the messages you've received, and reject any that are duplicates. Duplicates here means either or both MQMD and application data have been duplicated.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
