ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General IBM MQ Support » Transaction Management in IIB10 and Standalone MQ

Post new topic  Reply to topic
 Transaction Management in IIB10 and Standalone MQ « View previous topic :: View next topic » 
Author Message
vmanel
PostPosted: Tue Mar 12, 2019 5:13 pm    Post subject: Transaction Management in IIB10 and Standalone MQ Reply with quote

Acolyte

Joined: 12 Mar 2019
Posts: 57

IIB10 and MQ7 are installed in different servers, and MQEndpoint policy is used by IIB to connect to MQ.
All the queues present(input node queue, output node queue and queue present in failure terminal of input node) in the flows are created in this queue manager. All the queues are persistent and Transaction property of Input Node in message flow is set to 'Yes'.

What happens if the message is picked by the message flow and connectivity to Queue Manager is lost, and flow is not able to put message to any of the queues?

Will the message be lost or will it be processed once the connectivity is restored? Under what circumstances message will be lost for this kind of setup?
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Mar 13, 2019 5:07 am    Post subject: Re: Transaction Management in IIB10 and Standalone MQ Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

vmanel wrote:
What happens if the message is picked by the message flow and connectivity to Queue Manager is lost, and flow is not able to put message to any of the queues?


The MQOutput node will receive an appropriate reason code from the MQ layer (probably a 2059 or 2009) and throw an exception.

vmanel wrote:
Will the message be lost or will it be processed once the connectivity is restored?


Depends entirely on what you've coded the flow to do in the event of a failure. Certainly nothing out of the box will retry delivery but you code such a mechanism. What would happen if (for example) the inbound message contained bad data and caused an exception in one of the processing nodes?

vmanel wrote:
Under what circumstances message will be lost for this kind of setup?


Circumstances where the flow code allows or causes the message to be lost.

Before someone else says it, queues are not persistent. Messages are persistent or not persistent. Just because the queue has a default of "persistent" does not prevent an application writing a non-persistent message to it.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
vmanel
PostPosted: Wed Mar 13, 2019 6:20 am    Post subject: Re: Transaction Management in IIB10 and Standalone MQ Reply with quote

Acolyte

Joined: 12 Mar 2019
Posts: 57

Vitor wrote:

The MQOutput node will receive an appropriate reason code from the MQ layer (probably a 2059 or 2009) and throw an exception.

Depends entirely on what you've coded the flow to do in the event of a failure. Certainly nothing out of the box will retry delivery but you code such a mechanism. What would happen if (for example) the inbound message contained bad data and caused an exception in one of the processing nodes?


The catch terminal of MQ inpit node is connected to the throw node and failure terminal of MQ Input node is connected to MQ Output Node(which acts as backout queue).

My question is incase of connectivity failure since it will not be able to put message to queue in failure terminal as well, will the message be lost?

How to ensure that message is not lost?

Since Queues are persistent and Transaction of Input Node is set to 'Yes', doesnt MQ and IIB ensure that the message is not lost until that transaction is complete? What is the use of "persistent" property of queues?
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Mar 13, 2019 6:32 am    Post subject: Re: Transaction Management in IIB10 and Standalone MQ Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

vmanel wrote:
The catch terminal of MQ inpit node is connected to the throw node


Odd choice - why?

vmanel wrote:
and failure terminal of MQ Input node is connected to MQ Output Node(which acts as backout queue).


Which will work for anything except an MQ problem

vmanel wrote:
My question is incase of connectivity failure since it will not be able to put message to queue in failure terminal as well, will the message be lost?


No, because the MQOutput node putting to the backout queue will throw an MQ error as well, which will flow to the MQInput node, which will flow to MQOutput node, which will fail with an MQ error, etc., etc. and the CPU usage of the IIB server will spike. The message may be lost when the exception tree becomes so large the memory is exhausted and the flow crashes.

vmanel wrote:
How to ensure that message is not lost?


Don't rely on an external system (which may be inaccessible) to store data that can't be processed because an external system is inaccessible.

vmanel wrote:
Since Queues are persistent and Transaction of Input Node is set to 'Yes', doesnt MQ and IIB ensure that the message is not lost until that transaction is complete?


Queues are not persistent, messages are. See above.

It does ensure that, which is why you get the loop I outline above. The flow can't discard a persistent message but can't process it.

vmanel wrote:
What is the use of "persistent" property of queues?


It's not a "persistent" property. It's called "default persistence" and allows an application programmer to delegate the persistence setting of a message to the administrator. The programmer must choose to do this by putting the message with a persistence setting of "persistence as queue default" rather than "yes" or "no".

You'll find a lively debate in this forum on the pros and cons of persistence being determined by administrators v persistence being determined by the application. It's one of those questions where if you ask 5 MQ experts in a room, you get 6 opinions.......
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
vmanel
PostPosted: Wed Mar 13, 2019 7:33 am    Post subject: Reply with quote

Acolyte

Joined: 12 Mar 2019
Posts: 57

What we are planning to handle this situation is to have another MQ in same server as IIB, and the flows will have output queues and queue present in failure terminal pointing to this MQ, but those will be remote queues which will in turn point to Standalone MQ present in different server.
Whereas input queue will be still in Standalone MQ present in different server.
In this case what will happen if the connection is lost? The IIB will put message to local MQ and message will be in transmission queue until connectivity is restored. Will Standalone MQ assume transaction is incomplete, and perform the same transaction twice once the connectivity is restored?

What can be the other possible issues if this approach is used?
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Mar 13, 2019 9:15 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

vmanel wrote:
Will Standalone MQ assume transaction is incomplete, and perform the same transaction twice once the connectivity is restored?


Well the flow won't be able to commit the get from the input queue (because connectivity has been lost) so that won't think the transaction is complete. I suspect (but don't know) the flow transaction will hang (with the message put to the transmit queue but not committed because the transaction can't be completed). Once connectivity is restored, you're going to be in an interesting position with 2 outstanding MQ operations. My best guess is that the put to the remote queue will succeed, the MQInput node will read the next available message (because the original connection handle has an open transaction on the first message), continue to process down the queue and eventually reprocess the first message when the handle times out. Remember that when the connection comes back, the node will establish a new connection to the queue, not reopen the previous one. That's standard MQ client behavior.

Here's what I don't get - why are you, from a design standpoint, trying to send a properly processed message into the back out queue on the original queue manager?With all the processing failures? Why, again design wise, trying to report a problem with MQ using MQ? This makes no sense to me.

If you have an MQ on the same server as IIB (and presumably associated with IIB), why not build a retry mechanism to continuously attempt to put the message in the event connectivity is restored?

Also you need to accept that there could be a prolonged outage and you can't retry forever. Sooner or later you need to store the message someplace and wait for a fix.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
vmanel
PostPosted: Wed Mar 13, 2019 10:40 am    Post subject: Reply with quote

Acolyte

Joined: 12 Mar 2019
Posts: 57

Vitor wrote:

Well the flow won't be able to commit the get from the input queue (because connectivity has been lost) so that won't think the transaction is complete. I suspect (but don't know) the flow transaction will hang (with the message put to the transmit queue but not committed because the transaction can't be completed). Once connectivity is restored, you're going to be in an interesting position with 2 outstanding MQ operations. My best guess is that the put to the remote queue will succeed, the MQInput node will read the next available message (because the original connection handle has an open transaction on the first message), continue to process down the queue and eventually reprocess the first message when the handle times out. Remember that when the connection comes back, the node will establish a new connection to the queue, not reopen the previous one. That's standard MQ client behavior.


What do you mean by "eventually reprocess the first message when the handle times out"? Will it process first message twice?

Vitor wrote:
Here's what I don't get - why are you, from a design standpoint, trying to send a properly processed message into the back out queue on the original queue manager?With all the processing failures? Why, again design wise, trying to report a problem with MQ using MQ? This makes no sense to me.


Properly processed message will be put to output queue on the Original Queue Manager through remote queue in IIB Queue Manager. Only failed message will be put to backout queue. This was the design suggested as only MQ needs to be used.


Vitor wrote:
If you have an MQ on the same server as IIB (and presumably associated with IIB), why not build a retry mechanism to continuously attempt to put the message in the event connectivity is restored?


What kind of retry mechanism are you suggesting? Can you elaborate a bit on that?

Vitor wrote:
Also you need to accept that there could be a prolonged outage and you can't retry forever. Sooner or later you need to store the message someplace and wait for a fix.


Do you mean that even in this approach, since the message is in transmission queue, the retry would cause spike in IIB server CPU usage?

One more question, are all these relevant for both persistent and non persistent messages?
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Mar 13, 2019 10:59 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

vmanel wrote:
What do you mean by "eventually reprocess the first message when the handle times out"? Will it process first message twice?


If the scenario I guessed at is right. The first message was read off (while there was connectivity) and is sitting waiting for the read to be committed. Nothing else can read it because the queue manager knows it was read off (remember the MQInput node uses an MQGet; IBM is very clear that MQGet returns the first available message not the first message). After the failed connection times out, that message will become available again and will be read off.

vmanel wrote:
Properly processed message will be put to output queue on the Original Queue Manager through remote queue in IIB Queue Manager. Only failed message will be put to backout queue.


Okay, re-reading this from the top your thread is a little misleading. You started with:

vmanel wrote:
What happens if the message is picked by the message flow and connectivity to Queue Manager is lost, and flow is not able to put message to any of the queues?


then switched to:

vmanel wrote:
What we are planning to handle this situation is to have another MQ in same server as IIB, and the flows will have output queues and queue present in failure terminal pointing to this MQ


The penny didn't drop my end that you'd changed tack, and the flow will always be able to put the output messages (because they're local). So you'll never get a failure to put a processed message and the problem becomes redundant.

vmanel wrote:
Vitor wrote:
If you have an MQ on the same server as IIB (and presumably associated with IIB), why not build a retry mechanism to continuously attempt to put the message in the event connectivity is restored?


What kind of retry mechanism are you suggesting? Can you elaborate a bit on that?


Again, the penny failed to drop. I was referring to the IIB retry, which requires an associated queue manager. The simpler answer is to abandon client connections, have MQ move the messages to the IIB server for the MQInout node to read (like it's moving output messages) and let the MQ channels worry about connection issues.

vmanel wrote:
Vitor wrote:
Also you need to accept that there could be a prolonged outage and you can't retry forever. Sooner or later you need to store the message someplace and wait for a fix.


Do you mean that even in this approach, since the message is in transmission queue, the retry would cause spike in IIB server CPU usage?


No, because I failed to properly notice you'd moved the output queues to be local to the IIB instance. So the point is moot. Just have IIB read/write all the messages to locally hosted queue instances (which will be remote queues for output) and walk away. The reason to use client connections is to avoid the need / cost / maintenance of a local MQ instance. As you have one, you don't need clients and with that, you don't have a problem as the IIB transaction is entirely local and immune to connection issues.

vmanel wrote:
One more question, are all these relevant for both persistent and non persistent messages?


Don't decide message persistence based on the ability of code to recover. But a non-persistent message can be discarded by the system in extremis, like to break a poison loop.

But you don't actually have to worry about any of this once you stop using your unnecessary client connection.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
vmanel
PostPosted: Thu Mar 14, 2019 6:42 am    Post subject: Reply with quote

Acolyte

Joined: 12 Mar 2019
Posts: 57

Thank you very much for the response, I understand that its confusing, but that was the design that was suggested. I will get back to you if I have any more questions.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Mar 14, 2019 6:53 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

vmanel wrote:
that was the design that was suggested.


Who by? Are they the sort of people who enjoy practical jokes? Do you owe them money?


_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu Mar 14, 2019 10:07 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

Vitor wrote:
vmanel wrote:
that was the design that was suggested.


Who by? Are they the sort of people who enjoy practical jokes? Do you owe them money?


... and where can i get some of the drugs they are taking?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » General IBM MQ Support » Transaction Management in IIB10 and Standalone MQ
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.