| Author |
Message
|
| JavaDev135 |
 Posted: Wed Dec 24, 2014 8:09 am Post subject: The 2009 exception happens on commit but put succeeds |
 |
|
Newbie
Joined: 24 Dec 2014
Posts: 4
|
We have a strange issue that happens sporadically and rarely but has pretty significant impact.
The following MQ.NET code places the XML messages on the queue. In rare instances, this code fails with an 2009 MQ Code exception right in the middle of commit, but the message is still placed on the queue. This happened a couple of times in the past 2 months. Our program assumes there was a failure because of this MQException and tries to place the message again resulting in the duplication of messages. Any idea on how to fix this from client code perspective?
The 2009 issue in itself is not a problem if placing of the message actually failed (our infrastructure guys are looking into the reason for the 2009 issue happening). The fact that that the put succeeds while the commit operation throws an MQException is the big problem because we can't know that the MQ Put operation succeeded.
key portion of the code in the class/method called CustomMQUtility.WriteMsg:
| Code: |
if (queueManager.IsConnected) {
queue = queueManager.AccessQueue(queueName, MQC.MQOO_OUTPUT + MQC.MQOO_FAIL_IF_QUIESCING);
queueMessage = new MQMessage();
queueMessage.CharacterSet = 1208;
var utf8Enc = new UTF8Encoding();
byte[] utf8String = Encoding.UTF8.GetBytes(strInputMsg);
queueMessage.WriteBytes(Encoding.UTF8.GetString(utf8String).ToString());
queuePutMessageOptions = new MQPutMessageOptions();
queuePutMessageOptions.Options += MQC.MQGMO_SYNCPOINT;
queue.Put(queueMessage, queuePutMessageOptions);
queueManager.Commit();
queueManager.Disconnect();
queue.Close();
} |
Exception:
MQ Reason code: 2009, Exception: MQRC_CONNECTION_BROKEN
StackTrace: at IBM.WMQ.MQQueueManager.Commit()
at CustomMQUtility.WriteMsg(String strInputMsg, MessageDocument msgDoc) |
|
| Back to top |
|
 |
| tczielke |
| Posted: Wed Dec 24, 2014 9:26 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
| Yes, a commit can get a failure return code, and the commit can still be successful. One thing you can do is include another queue as a consistency check within your unit of work, where you track whether the commit was successful or not. For example, you can do your PUT on your XML queue and then PUT some message (i.e. a message that includes the current timestamp) on your consistency queue under the same unit of work, and do your commit. If the commit fails, you can then do a browse on your consistency queue and see if that message with that timestamp is still there. If so, you know your PUT to the XML queue was committed, too. |
|
| Back to top |
|
|
| McueMart |
| Posted: Wed Dec 24, 2014 3:06 pm Post subject: |
|
|
Chevalier
Joined: 29 Nov 2011
Posts: 490
Location: UK...somewhere
|
|
| Back to top |
|
|
| tczielke |
| Posted: Wed Dec 24, 2014 3:23 pm Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Dec 24, 2014 4:49 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Emphasis on
| Quote: |
| ...it is possible that the unit of work was successfully committed. |
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| JavaDev135 |
| Posted: Wed Dec 24, 2014 4:55 pm Post subject: |
|
|
Newbie
Joined: 24 Dec 2014
Posts: 4
|
| McueMart wrote: |
| I guess when the client application gets the 2009 exception, it could check to see if the QMGR returned a confirmation message, indicating that it did actually successfully receive the message. |
I will look into this option - thanks for this suggestion.
Any other ideas on how to solve this pickle would be appreciated. Frankly, I expected that the commit command would either fail or succeed. |
|
| Back to top |
|
|
| JavaDev135 |
| Posted: Wed Dec 24, 2014 4:58 pm Post subject: |
|
|
Newbie
Joined: 24 Dec 2014
Posts: 4
|
| bruce2359 wrote: |
Emphasis on
| Quote: |
| ...it is possible that the unit of work was successfully committed. |
|
Understood, so the obvious question is if there is a way to find out whether it succeeded or not, because we want to make sure there is guaranteed delivery of the message, or if the delivery failed we would try to send the message again. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Dec 24, 2014 5:20 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
The app will need to reconnect to the qmgr. Search for 'reconnecting to a qmgr'. It's been discussed here frequently.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| tczielke |
| Posted: Thu Dec 25, 2014 5:11 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
| JavaDev135 wrote: |
| bruce2359 wrote: |
Emphasis on
| Quote: |
| ...it is possible that the unit of work was successfully committed. |
|
Understood, so the obvious question is if there is a way to find out whether it succeeded or not, because we want to make sure there is guaranteed delivery of the message, or if the delivery failed we would try to send the message again. |
So you are dealing here with a potential false negative condition on the MQCMIT on the distributed platform. The MQCMIT is telling you that it failed, but potentially it did succeed. You need to account for this in your application design in some way. If your application cares about not potentially sending duplicate messages, then the following are a few options when the MQCMIT fails in this way:
1. Mark these potential duplicate messages with some flag that they may be duplicates and resend. Design your receiving application to handle this potential for duplicate messages. This may be difficult or impractical to implement, depending on your application requirements.
2. As I mentioned before, include some kind of consistency or state tracking queue in your unit of work that can keep track if the MQCMIT was successful, or not. If your putting application gets this type of MQCMIT failure, you can now check your consistency queue to see if the MQCMIT was successful or not. The channel application (MCA) does this, per a previous LISTSERV thread I had on this topic with Paul Clarke. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Dec 25, 2014 1:24 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
It seems to me that this false positive might be linked in some way to the default commit behavior for the platform.
Depending on the platform a few things happen on disconnect.
- open (uncommitted) units of work are rolled back
- open (uncommitted) units of work are committed
- open (uncommitted) units of work are left in pending state (JMS)
May be we should just ask for the behavior to be consistent across platforms.
At least I would like to know what the rationale was behind treating a disconnect as a commit... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Dec 25, 2014 2:27 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Look at the rc 2009 programmer response here http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_7.0.1/com.ibm.mq.csqsao.doc/fm12160_1.htm . Frequently, the rc 2009 is caused by a (brief) network failure between the MQ Client software and the qmgr. Less often (in my experience) is the failure (abnormal termination) of the qmgr itself. Search here or google for other causes.
The MQ Client is not your application; rather, it is the IBM-supplied MQ Client layer of software installed on the client platform that interfaces between your client-bindings app, the network, and the qmgr.
At a minimum, for each MQI call from your client-bindings application there are two network flows. The first flow is the MQI call transmitted by the MQ Client across the network from your application to qmgr; the second flow is the qmgr transmitting the Reason-Code/Completion-Code (CC/RC) back to the the MQ Client, which are then passed back to your application.
In the case of an MQCMIT call, it is possible that the network failed AFTER the MQCMIT call was transmitted to the qmgr, and AFTER the qmgr committed messages, but BEFORE the successful (or unsuccessful) CC/RC are transmitted from the qmgr to, and received by the MQ client.
The MQ Client layer waits for the CC/RC to be returned from the mqgr. If no CC/RC is returned from the qmgr, the MQ Client returns 2009 to your application.
The application should try to reconnect to the qmgr, and determine if the MQCMIT was successful, and take remedial action if not.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Dec 26, 2014 7:50 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
The good part of JMS is the external transaction manager that keeps a resource identifier for the transaction and can request the transaction state from the resource manager.
For the case (non JMS) where the disconnect prevents the return code of the commit from reaching the client, I wish there was the same kind of inquiry possible for the transaction state.
Like the actual MQI call containing a transaction identifier in the return information... If that transaction identifier could then be queried on reconnect it would help to resolve that situation without round about strategy...
It should also allow to specifically commit or rollback the transaction using the identifier, if the transaction is still in pending state.
Well may be a RFE somebody?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sat Dec 27, 2014 10:43 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| McueMart wrote: |
Interesting post. I had never actually considered that a Client application could get an Exception on the PUT, but the operation succeed...
One thing that could be used (although I have never seen it used!) is MQs ability to produce Confirmation-of-arrival reports : http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_7.5.0/com.ibm.mq.dev.doc/q022870_.htm%23q022870___Report_messages
I guess when the client application gets the 2009 exception, it could check to see if the QMGR returned a confirmation message, indicating that it did actually successfully receive the message. |
If memory serves, RC 2009 is possible for all MQI calls, and not limited to client-binding apps. A quick read of the APR manual (or equivalent) will confirm this.
One of the causes of 2009 is the qmgr terminating. As such, COA/COD report messages depend on the qmgr being alive and well to produce them.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| McueMart |
| Posted: Sun Dec 28, 2014 3:46 am Post subject: |
|
|
Chevalier
Joined: 29 Nov 2011
Posts: 490
Location: UK...somewhere
|
| So it seem that the only truly safe method is to design all consuming applications with idempotence in mind (using some kind of unique message id). I have rarely seen this done in practice. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sun Dec 28, 2014 6:36 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| McueMart wrote: |
| So it seem that the only truly safe method is to design all consuming applications with idempotence in mind (using some kind of unique message id). I have rarely seen this done in practice. |
A well-behaved application should be able to recognize and deal with (tolerate) missing or duplicated requests/replies. I've seen very few of these, as well.
"Well-behaved" means that a requester app should keep track of request messages it has sent, and replies received. Likewise, a well-behaved responder app should keep track of requests it has received, and replies it has sent. Both requester and responder should recognize loss and duplication of transaction data.
MQ internals assure that messages are delivered once, with no loss or duplication; but but misbehaving applications can lose or duplicate data.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
