ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General IBM MQ Support » Error handling in MB/MQ

Post new topic  Reply to topic Goto page 1, 2  Next
 Error handling in MB/MQ « View previous topic :: View next topic » 
Author Message
jnoubi
PostPosted: Fri Nov 02, 2007 3:37 am    Post subject: Error handling in MB/MQ Reply with quote

Novice

Joined: 26 Sep 2006
Posts: 24

I am working on an error handling strategy for our environment which uses MQ 6.0 for transport and MB 6.0 for routing and transformation.
One of the strategies is to roll back messages into the source system in case the link with the target system is down.

So basically, in case the link between the MB and the target system is down (for any reason such as channel down, network issues,..), my strategy is to roll back any messages to the source system's queue (no DLQ is specified for the queue manager of the message broker and retry count is set to 5 for example). This works well because I am ensuring that incoming messages will stay in sequence and the source system will be forced to stop sending once their queue is full.

However, the problem that I am facing is how to restart the process of MB picking up the messages from the incoming queue once the link to the target system is restored?

So basically, the recovery strategy is not clear. If I manually remove the first message from the incoming queue (for ex using rfhutil), the flow is restarted. However, because one of my requirements is not to lose any message and not to lose the order of the messages, I can't manually remove messages.

Any suggestions to restart the flow without any loss of messages or order?
Back to top
View user's profile Send private message
jefflowrey
PostPosted: Fri Nov 02, 2007 4:11 am    Post subject: Reply with quote

Grand Poobah

Joined: 16 Oct 2002
Posts: 19981

You should restart the flow in the opposite manner to how you stopped it.

But most everything about your design is not best practices.
_________________
I am *not* the model of the modern major general.
Back to top
View user's profile Send private message
JosephGramig
PostPosted: Fri Nov 02, 2007 4:15 am    Post subject: Reply with quote

Grand Master

Joined: 09 Feb 2006
Posts: 1230
Location: Gold Coast of Florida, USA

Good question, but you put it in the wrong topic. Someone will move this to the MQSI/WMB Forum.

Hmmm, I assume you want the flow to automatically stop and start when connectivity is restored.

Wouldn't it be better to create the XMIT size to hold all of the messages until the connectivity is restored and let the channel handle this issue?
_________________
Joseph
Administrator - IBM WebSphere MQ (WMQ) V6.0, IBM WebSphere Message Broker (WMB) V6.1 & V6.0
Solution Designer - WMQ V6.0
Solution Developer - WMB V6.1 & V6.0, WMQ V5.3
Back to top
View user's profile Send private message AIM Address
jnoubi
PostPosted: Fri Nov 02, 2007 6:27 am    Post subject: Reply with quote

Novice

Joined: 26 Sep 2006
Posts: 24

Thanks for the replies so far.

Restarting the flow did not help. The queue is still taking in messages but not releasing any to MB. The only I was able to get the queue to start functioniong properly is by removing the first message (that was retried few times).

Regarding the increase of the XMIT queue size: I assume you are referring to the xmit queue on the message broker tier: that still does not solve the problem of restarting the reception of messages from the source application queue, it just buys more time until the queue starts overflowing.
Back to top
View user's profile Send private message
bower5932
PostPosted: Fri Nov 02, 2007 6:28 am    Post subject: Reply with quote

Jedi Knight

Joined: 27 Aug 2001
Posts: 3023
Location: Dallas, TX, USA

jnoubi wrote:
TRestarting the flow did not help. The queue is still taking in messages but not releasing any to MB. The only I was able to get the queue to start functioniong properly is by removing the first message (that was retried few times).


What is the backout count on the first message? Is it possible that the flow was actually receiving it and kept rolling it back?
Back to top
View user's profile Send private message Send e-mail Visit poster's website AIM Address Yahoo Messenger
JosephGramig
PostPosted: Fri Nov 02, 2007 7:20 am    Post subject: Reply with quote

Grand Master

Joined: 09 Feb 2006
Posts: 1230
Location: Gold Coast of Florida, USA

Well, the channel is designed to get the messages from point A to point B and knows how to recover and restart in the event of a communications failure.

So, if you size the XMITQ to hold as many messages as will be generated during the outage, then the flow will not need to stop and therefore not need to be restarted.
_________________
Joseph
Administrator - IBM WebSphere MQ (WMQ) V6.0, IBM WebSphere Message Broker (WMB) V6.1 & V6.0
Solution Designer - WMQ V6.0
Solution Developer - WMB V6.1 & V6.0, WMQ V5.3
Back to top
View user's profile Send private message AIM Address
jnoubi
PostPosted: Fri Nov 02, 2007 7:46 am    Post subject: RE: Reply with quote

Novice

Joined: 26 Sep 2006
Posts: 24

To bower5932: The backout count was 147 . So apparently it retried few times and then stopped?

To JosephGramig: I think you mis-understood the case I am trying to resolve, so let me explain it in a different way:
The source App A has put a bunch of messages on a remote queue that points to the message broker queue manager. The message broker has a local queue LQA that receives messages from App A. The message broker puts these messages on remote queue RQB to send them to Application B. But when the link with application B is down, the message can not put anymore messages and will start rolling back messages to LQA which in turn starts accumulating messages until it reaches its max allowable depth. Application A will not be able to send anymore messages because LQA is full. The first message on the LQA queue will have a certain backout count. Application A will abend. So far so good.
However, the problem is how to restart the processing of messages once the link with Application B is restored?
It does not restart automatically.
Back to top
View user's profile Send private message
bower5932
PostPosted: Fri Nov 02, 2007 7:57 am    Post subject: Re: RE: Reply with quote

Jedi Knight

Joined: 27 Aug 2001
Posts: 3023
Location: Dallas, TX, USA

jnoubi wrote:
To bower5932: The backout count was 147 . So apparently it retried few times and then stopped?


I'm not sure that it stopped. It might still be reading it and rolling it back. You need to check again to see if the count is increasing. If it is, your problem is what is wrong with the message and not why is the broker not picking up messages. Have you looked at any of the error logs? Have you turned on tracing?
Back to top
View user's profile Send private message Send e-mail Visit poster's website AIM Address Yahoo Messenger
SAFraser
PostPosted: Fri Nov 02, 2007 11:03 am    Post subject: Reply with quote

Shaman

Joined: 22 Oct 2003
Posts: 742
Location: Austin, Texas, USA

Please let me be sure I understand.

Broker puts message to RQB, destined for a queue manager that services application B.

The link to the application B queue manager goes down, so the sender channel goes down. The transmit queue that services the channel to application B gets full.

The broker cannot put any more messages to the RQB because the transmit queue is full. So it rolls messages back to LQA.

Is this correct? If yes, then I agree with all others.... Resize the transmit queue that services the channel to application B. The broker's service will not be interrupted.

One of the strong points of MQ is its ability to decouple the sender and the receiver applications. A way to do this is to size your transmit queues to allow for channel downtime. Some of my transmit queues are really quite large, for this very reason.

Am I understanding your situation correctly?

Thanks,
Shirley
Back to top
View user's profile Send private message
jnoubi
PostPosted: Fri Nov 02, 2007 12:13 pm    Post subject: RE: Reply with quote

Novice

Joined: 26 Sep 2006
Posts: 24

SAFraser: Part of the problem is to handle the downtime which by increasing the queue depth as you described. That is fine.
But what I am trying to document for our support team is the recovery strategy -> what are the steps that are needed in order to get back to normal operation: what I mean by this is when the connection to App B is up, what do I need to do to get the flow in MB to restart picking up messages from LQA? it does not do this automatically when the connection to B is restored. The queue is still blocked even though the problem with App B is resolved. I found that the only to resolve the problem is to manually remove the first message from LQA (the one that has a high backout retry count).
Back to top
View user's profile Send private message
Michael Dag
PostPosted: Fri Nov 02, 2007 12:37 pm    Post subject: Re: RE: Reply with quote

Jedi Knight

Joined: 13 Jun 2002
Posts: 2602
Location: The Netherlands (Amsterdam)

jnoubi wrote:
SAFraser: Part of the problem is to handle the downtime which by increasing the queue depth as you described. That is fine.
But what I am trying to document for our support team is the recovery strategy -> what are the steps that are needed in order to get back to normal operation: what I mean by this is when the connection to App B is up, what do I need to do to get the flow in MB to restart picking up messages from LQA? it does not do this automatically when the connection to B is restored. The queue is still blocked even though the problem with App B is resolved. I found that the only to resolve the problem is to manually remove the first message from LQA (the one that has a high backout retry count).


If your queuedepth is deep enough... you'll never get to this point and no recovery of 'broker' flows is needed...
_________________
Michael



MQSystems Facebook page
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
JosephGramig
PostPosted: Fri Nov 02, 2007 12:46 pm    Post subject: Reply with quote

Grand Master

Joined: 09 Feb 2006
Posts: 1230
Location: Gold Coast of Florida, USA

So what we are saying is, don't get into the XMITQ full condition and you don't have to restart the flow because it won't stop.

The broker will not process the message because it has exceeded its retry count and you have no DLQ for it to put the message to. The flow is clearly in transactional mode.

Of course, I highly recommend you search this site about DLQs and read the information.
_________________
Joseph
Administrator - IBM WebSphere MQ (WMQ) V6.0, IBM WebSphere Message Broker (WMB) V6.1 & V6.0
Solution Designer - WMQ V6.0
Solution Developer - WMB V6.1 & V6.0, WMQ V5.3
Back to top
View user's profile Send private message AIM Address
jnoubi
PostPosted: Fri Nov 02, 2007 1:18 pm    Post subject: RE: Reply with quote

Novice

Joined: 26 Sep 2006
Posts: 24

Thanks for the replies.

Regarding the DLQ, I purposefully removed the DLQ because I want to ensure the sequence of my messages. If one of the messages end up in the DLQ, and the message broker processes the next one then I lost the sequence of messages.

I guess the best solution is to try to avoid this situation by having very deep queues.
Back to top
View user's profile Send private message
bower5932
PostPosted: Fri Nov 02, 2007 2:13 pm    Post subject: Re: RE: Reply with quote

Jedi Knight

Joined: 27 Aug 2001
Posts: 3023
Location: Dallas, TX, USA

jnoubi wrote:
I guess the best solution is to try to avoid this situation by having very deep queues.


The best solution would be to avoid having to have your messages delivered in sequence. Is it possible to send your messages in groups and then only get them when they are all there?
Back to top
View user's profile Send private message Send e-mail Visit poster's website AIM Address Yahoo Messenger
fjb_saper
PostPosted: Fri Nov 02, 2007 3:21 pm    Post subject: Re: RE: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

jnoubi wrote:
Thanks for the replies.

Regarding the DLQ, I purposefully removed the DLQ because I want to ensure the sequence of my messages. If one of the messages end up in the DLQ, and the message broker processes the next one then I lost the sequence of messages.

I guess the best solution is to try to avoid this situation by having very deep queues.

You have designed for message affinity... Everybody will tell you this is a bad thing and kills scalability...

Back to the drawing board !
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » General IBM MQ Support » Error handling in MB/MQ
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.