| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| MQClient getting persistent messages - Chance of duplicates? |
« View previous topic :: View next topic » |
| Author |
Message
|
| PeterPotkay |
 Posted: Fri Jul 12, 2002 11:12 am Post subject: MQClient getting persistent messages - Chance of duplicates? |
 |
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
QM1 has Q1, a triggered on first queue. A remote application puts persistent message to this queue, causing an MQClient app on another machine to start up and connect to QM1, open Q1 and get the message under syncpoint. While the client is processing the message, the server housing QM1 goes down. Meanwhile the client finishes processing the message and goes to close and disconnect, only to find the QM not there anymore.
What happened to that message? It was persistent and gotten under syncpoint, but a commit (either implicit or explicit) was never done. Will it be there when the QM comes back up? And will the client app have to be coded to handle duplicate messages?
If the message will be there, is this a documented feature? In other words could/should the client code say unless I get a good RC on my MQCOMMIT or my MQDISC call, I assume all messages in this UOW will be there again, so undo everything I just did to avoid duplicates?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
 |
| bduncan |
| Posted: Fri Jul 12, 2002 3:14 pm Post subject: |
|
|
Padawan
Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley
|
I'm going to have to sit down and do a little reading before I can give a full answer to this one, but right away I can point out that the MQMD has a field called BackoutCount which keeps track of the number of times a message has been removed from the queue with an MQGET under syncpoint, and then subsequently rolled back. In the past, I've coded applications that actually looked at this value. If it was nonzero, I knew that my application had been triggered in the past, got this message, and then rolled it back (or it was automatically rolled back by the queue manager) for some reason. At this point, additional logic could kick in to see if this was a poison message, which everytime I tried to process it, it would crash my application.
As far as I know, if your queue manager goes down, the message in question will automatically rolled back, and then it is up to you (the application) to notice that the backout count is 1, and then check the database to see if the transaction was actually carried out. If so, it's safe to throw out the message. If not, you need to reprocess it...
_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Jul 12, 2002 4:22 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Huh, I completly forgot about that Backoutcount field. I will have to test this, as I wonder if it will actually go up by one in this case. I've done lots of tests and have coded applications to take advantage of this field, but its always been because the app did a backout, or the app abended. I have never killed a server/QM while outstanding GETS are still around. Will the QM be smart enough/quick enough to up the count before it crashes? Or as it comes back up? Or does it not work in this case? (I can't test this for a few days so I am curious as to what the answer is)
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| nimconsult |
| Posted: Sun Jul 14, 2002 10:40 pm Post subject: |
|
|
Master
Joined: 22 May 2002
Posts: 268
Location: NIMCONSULT - Belgium
|
A difficult subject, mixing the complexity of transaction management, and reliability of client connections. Let's try some small explanations about the question:
Scenario:
MQCONN (RC=0)
MQOPEN (RC=0)
MQGET under syncpoint (RC=0)
- queue manager crashes
MQCMIT (RC=connection broken)
First question: does the message goes back on the queue?
Yes the message goes back on the queue, assuming that the MQGET was under syncpoint.
Second question: is the backout-count of the message increased?
Yes. In fact the pending unit of work is rolled back on queue manager restart.
- If the queue has the HARDENBO set, the backout-count value of the message will be increased by one after queue manager recovery on restart.
- If the queue does not have the HARDENBO set, the backout-count value of the message will be 1 after queue manager recovery on restart. (which is by the way a good demonstration of the fact that roll-back of the transaction happened on restart).
Third question: how do you explain the behaviour of a client application?
You split the call to a MQ Series function into three phases:
- phase 1 (before the call): you know for sure that nothing happened on the queue manager.
- phase 2 (during the call): this is the "in-doubt" phase of the call; in case of crash, you do not know whether the function was executed or not (and successfully or not) on the queue manager. If your application has crashed you stop here; if the queue manager itself crashes, you reach phase 3 with RC connection broken.
- phase 3 (after the call): you know that the function was executed and the RC indicates the success of the operation.
Illustration 1: MQGET under syncpoint
- phase 1: nothing.
- phase 2: in-doubt. In case of crash you do not loose the message, but potentially the backout-count of the message will be increased.
- phase 3: if RC=0, the message is retrieved under unit of work. In case of failure after this point the message goes back on the queue and the backout-count is increased.
Illustration 2: MQCMIT (after a successfull MQGET under SYNCPOINT)
- phase 1: the UOW is pending
- phase 2: in-doubt. In case of crash the UOW may be syncpointed or rolled-back.
- phase 3: if RC=0, the UOW is closed and the message is definitely out of the queue.
Conclusion: there is always some "in-doubt" phase, this is why you should follow the advice of Brandon and implement some recovery logic in case you read a message with a backout-count greater than zero.
If you run in binding mode, you can ask MQ Series to act as a global transaction coordinator or run MQ Series under a transaction coordinator (COM+, CICS, Tuxedo...). In which case transaction coordination and recovery will be performed automatically with databases (such as DB2, Orable, Sybase) or other (XA-compliant) resource managers. Read the system administration manual for more information on this subject.
(Wow this was a long one  )
Regards,
_________________
Nicolas Maréchal
Senior Architect - Partner
NIMCONSULT Software Architecture Services (Belgium)
http://www.nimconsult.be |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Jul 15, 2002 7:34 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Thanks for the detailed explanation.
Just wanna clarify on the Hardened backout count attribute.
(From the programming referance manual)
The message backout count survives restarts of the queue manager. However, to ensure that the count is accurate, information has to be "hardened" (recorded on disk or other permanent storage device) each time a message is retrieved by an MQGET call within a unit of work for this queue. If this is not done, and a failure of the queue manager occurs together with backout of the MQGET call, the count may or may not be incremented.
Hardening information for each MQGET call within a unit of work, however, imposes a performance overhead, and the HardenGetBackout attribute should be set to MQQA_BACKOUT_HARDENED only if it is essential that the count is accurate.
On Compaq OpenVMS Alpha, OS/2, OS/400, Compaq NonStop Kernel, UNIX systems, and Windows, the message backout count is always hardened, regardless of the setting of this attribute.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| pcrparimi |
| Posted: Tue Jul 16, 2002 6:30 am Post subject: |
|
|
Apprentice
Joined: 09 Oct 2001
Posts: 43
Location: NY
|
We have an MQclient client program(java) puts the message in a persistant queue under syncpoint. problem is after put if network goes down, the put is not getting rolled back and the message becomes permanent. Is there a way that i can avoid this.
Regards,
Poorna. |
|
| Back to top |
|
|
| bduncan |
| Posted: Tue Jul 16, 2002 7:54 am Post subject: |
|
|
Padawan
Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley
|
Is the queue manager failure happening before or after your application is issuing the MQCMIT?
_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator |
|
| Back to top |
|
|
| pcrparimi |
| Posted: Wed Jul 17, 2002 5:08 am Post subject: |
|
|
Apprentice
Joined: 09 Oct 2001
Posts: 43
Location: NY
|
| Queue manger is not going down. I am intentionally disconnecting the network between client and QM, after MQPUT, but before MQCMIT. |
|
| Back to top |
|
|
| nimconsult |
| Posted: Wed Jul 17, 2002 10:47 pm Post subject: |
|
|
Master
Joined: 22 May 2002
Posts: 268
Location: NIMCONSULT - Belgium
|
Is the application still running, with connection opened, queue opened, MQPUT under syncpoint pending when you kill the connection?
Have you been waiting until the server queue manager realizes that the connection is lost?
How to check this: the open output count of the queue should pass from 1 to 0 when the server queue manager cleans the connection. At that time only will the transaction be rolled back.
If yes then the transaction should be rolled back as soon as the server queue manager realizes that the client connection is lost.
_________________
Nicolas Maréchal
Senior Architect - Partner
NIMCONSULT Software Architecture Services (Belgium)
http://www.nimconsult.be |
|
| Back to top |
|
|
| mqonnet |
| Posted: Tue Jul 23, 2002 5:47 am Post subject: |
|
|
Grand Master
Joined: 18 Feb 2002
Posts: 1114
Location: Boston, Ma, Usa.
|
Poorna what OS are we talking of. What version of MQ. And what level, GA/CSD.
Cheers.
Kumar
_________________
IBM Certified WebSphere MQ V5.3 Developer
IBM Certified WebSphere MQ V5.3 Solution Designer
IBM Certified WebSphere MQ V5.3 System Administrator |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
