| Author |
Message
|
| Boomn4x4 |
 Posted: Fri Dec 09, 2011 5:36 am Post subject: Handling undeliverable messages? |
 |
|
Disciple
Joined: 28 Nov 2011
Posts: 172
|
What is the appropriate way to handle undeliverable messages?
I'm in the early stages of developing a new C API which utilizes MQ messaging. I have a program monitoring a local queue and when a message shows up, the program does a "get" from the queue, then "puts" it to a clustered queue. The program works great, but I'm trying to understand what I would do with the message if the clustered queue becomes unavailable.
My simple thought is to simply have the program keep retrying until it can be delivered, but in doing so, there is an exposure to losing the data should something happen to the program while it is retrying. |
|
| Back to top |
|
 |
| exerk |
| Posted: Fri Dec 09, 2011 5:40 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
Do the GET in syncpoint and roll it back if you don't get a successful PUT?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Dec 09, 2011 5:41 am Post subject: Re: Handling undeliverable messages? |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Boomn4x4 wrote: |
| I have a program monitoring a local queue and when a message shows up, the program does a "get" from the queue, then "puts" it to a clustered queue. |
Why? If the message has made it to a queue manager with access to the cluster why do you need a program?
| Boomn4x4 wrote: |
| My simple thought is to simply have the program keep retrying until it can be delivered, but in doing so, there is an exposure to losing the data should something happen to the program while it is retrying. |
How is your application going to know the clustered queue(s) are unavailable? It can't tell from the MQPut.
There is no exposure to data loss (subject to message persistence & the other usual rules) because the messages will sit in the transmission queue until the target is available. This is as true with a cluster as it is with point to point.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Dec 09, 2011 5:45 am Post subject: Re: Handling undeliverable messages? |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Boomn4x4 wrote: |
| What is the appropriate way to handle undeliverable messages? |
Also in strict WMQ terms an "undeliverable" message is one where there is no target. Because WMQ is an assured delivery system, the fact that the target is valid but not currently available does not make it undeliverable just not yet delivered.
This may sound like pedantic hair splitting but where the WMQ documentation talks about undeliverable messages it's talking in WMQ terms, and the software has built in capabilities to handle such messages. Which it does not in the sense you mean undeliverable.
I act only to ward off confusion.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| exerk |
| Posted: Fri Dec 09, 2011 5:46 am Post subject: Re: Handling undeliverable messages? |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| Vitor wrote: |
| ...How is your application going to know the clustered queue(s) are unavailable? It can't tell from the MQPut... |
Good point...which is why I stay away from programming generally. I must have had an attack of the vapours.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Dec 09, 2011 5:48 am Post subject: Re: Handling undeliverable messages? |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| How is your application going to know the clustered queue(s) are unavailable? It can't tell from the MQPut. |
This is the important bit.
In every case when writing to a queue that is not a qlocal, the message is always instead put to the transmission queue.
The only time you should ever get an MQRC from the MQPUT that indicates an issue with transmitting the message is when the transmission queue is blocked, full, or unavailable for other reasons.
So focus on identifying what you need to do with your messages when you get back an MQRC that is not zero, rather than on what to do when your message can't be delivered. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Dec 09, 2011 5:52 am Post subject: Re: Handling undeliverable messages? |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| exerk wrote: |
| Vitor wrote: |
| ...How is your application going to know the clustered queue(s) are unavailable? It can't tell from the MQPut... |
Good point...which is why I stay away from programming generally. I must have had an attack of the vapours. |
It happens when you get to a certain age. Or because there's something leaking in the dungeon again. 
I remember my first senior moment. Dimly. So, so long ago.... 
Of course, this does highlight the need for proper infrastructure monitoring with WMQ.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| exerk |
| Posted: Fri Dec 09, 2011 5:56 am Post subject: Re: Handling undeliverable messages? |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| Vitor wrote: |
..It happens when you get to a certain age. Or because there's something leaking in the dungeon again.
I remember my first senior moment. Dimly. So, so long ago....
Of course, this does highlight the need for proper infrastructure monitoring with WMQ. |
True. I wish I'd been as clear in my answer as mqjeff, which was what I was alluding to, cough, cough...
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Dec 09, 2011 5:58 am Post subject: Re: Handling undeliverable messages? |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| exerk wrote: |
| I wish I'd been as clear in my answer as mqjeff |
We all wish that.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Boomn4x4 |
| Posted: Fri Dec 09, 2011 6:14 am Post subject: |
|
|
Disciple
Joined: 28 Nov 2011
Posts: 172
|
| So if I'm understanding correctly, using a program to get the message and put it on a clustered queue is not the correct approach? I should be putting a message on a remote queue definition with a local transmission queue and forwarding the message directly to the clustered queue? |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Dec 09, 2011 6:22 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Boomn4x4 wrote: |
| I should be putting a message on a remote queue definition with a local transmission queue and forwarding the message directly to the clustered queue? |
The question I asked was why use this application? If there's a good reason fine, but if this message has arrived from this point you've not mentioned by a route you've not mentioned onto a local queue on this queue manager which is clearly part of the necessary cluster then why put it on the local queue only to move it off rather than directly onto the clustered queue?
If I read between the lines of your post correctly and this message is coming from a non-clustered queue manager then your assertion is broadly correct. You'll find a section in the Clusters manual titled "Putting From a Queue Manager Outside The Cluster" worth reading.
If it's being put locally then you don't need anything apart from the putting application using the clustered queue!
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Boomn4x4 |
| Posted: Fri Dec 09, 2011 6:37 am Post subject: |
|
|
Disciple
Joined: 28 Nov 2011
Posts: 172
|
| Vitor wrote: |
| Boomn4x4 wrote: |
| I should be putting a message on a remote queue definition with a local transmission queue and forwarding the message directly to the clustered queue? |
The question I asked was why use this application? If there's a good reason fine, but if this message has arrived from this point you've not mentioned by a route you've not mentioned onto a local queue on this queue manager which is clearly part of the necessary cluster then why put it on the local queue only to move it off rather than directly onto the clustered queue? |
Where my concern lies, is what does the sending program.... the program that is generating a message in the first place....do if that clustered queue isn't there, where does the message go?
The message itself will be physically getting to the clustered queue over a satellite network connection. This type of network is inherently susceptible to environmental conditions... snow, rain, heavy cloud cover, etc. and inevitably, that connection link will be down, potentially, for an extended period of time. Hurricanes, in the past, have taken communications down for weeks.
My understanding is that the initiating program will need a place to put those messages as the local physical location cannot afford to simply wait it out, potentially for weeks at a time. My proposed solution to this is to have a local staging queue where messages can be placed so that the local application can operate independently of external network communications. This way, when the network is down, it can still be 'business-as-usual'. Once the network comes back, the messages would then be relayed offsite. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Dec 09, 2011 7:00 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Boomn4x4 wrote: |
| Where my concern lies, is what does the sending program.... the program that is generating a message in the first place....do if that clustered queue isn't there, where does the message go? |
The sending program doesn't do anything. It won't know the clustered queue isn't there any more than it would know about any other queue. As to where the message goes, it goes into the transmission queue. Again, this is true for clustered or non-clustered.
| Boomn4x4 wrote: |
| The message itself will be physically getting to the clustered queue over a satellite network connection. This type of network is inherently susceptible to environmental conditions... snow, rain, heavy cloud cover, etc. and inevitably, that connection link will be down, potentially, for an extended period of time. Hurricanes, in the past, have taken communications down for weeks. |
And how will the message get to the local queue? Does it have a different network to use? Or is it actually local to the queue manager?
| Boomn4x4 wrote: |
| My understanding is that the initiating program will need a place to put those messages as the local physical location cannot afford to simply wait it out, potentially for weeks at a time. My proposed solution to this is to have a local staging queue where messages can be placed so that the local application can operate independently of external network communications. This way, when the network is down, it can still be 'business-as-usual'. Once the network comes back, the messages would then be relayed offsite. |
Why have you bought WMQ? You seem to be building your own solution to all the things WMQ fixes. WMQ is an assured delivery system designed to be resilient in the face of communication difficulties. WMQ provides a "local staging queue" for messages in transit; this is basic, basic WMQ.
What is your percieved issue with this?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Dec 09, 2011 7:08 am Post subject: Re: Handling undeliverable messages? |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Boomn4x4 wrote: |
| What is the appropriate way to handle undeliverable messages? |
I'm going to take a different view of the OP. Perhaps this is the intended question:
How can an application determine if a message was successfully put to the appropriate destination queue?
WMQ has built-in mechanisms to deal with this.
Two possible reasons that a message hasn't arrived at the intended local queue on a qmgr down the network:
1) there is no message. If the local transmission queue on the qmgr where the application is running doesn't exist, then the app will receive an unknown-object failure at MQOPEN. If the transmission queue exists, but is not available (put-inhibited, queue full, etc.), then the application will receive an MQPUT failure. In both cases, THERE IS NO MESSAGE.
This is all documented in the WMQ Application Programming Reference manual.
2) there is a message (the MQPUT succeeded), but it can't be delivered to the down-stream qmgr. Reasons for this include
- the channel doesn't exist
- the channel is not started
- the channel has failed
The application receives NO ReasonCode to indicate this type of failure (delay).
If the channel is functioning, the down-stream qmgr is functioning, then other reasons why the message cannot be delivered include:
- the destination queue doesn't exist
- the destination queue doesn't exist, or
- the destination queue is full
- the destination queue is put-inhibited
- the message is too big for the destination queue
The application recieves NO indication of this - unless the application has been coded to ask for, and deal with COA and COD report messages from down-stream qmgr components that the MQPUT there failed.
The usual approach to dealing with un-deliverable messages is to define a dead-letter queue where the un-deliverable message should be MQPUT to.
A dead-letter handler utility is provided that can process the message out of the dead-letter queue to some other application-oriented destination (queue or qmgr).
This is all documented in the Intercommunications manual.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Dec 09, 2011 7:26 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Boomn4x4 wrote: |
| So if I'm understanding correctly, using a program to get the message and put it on a clustered queue is not the correct approach? I should be putting a message on a remote queue definition with a local transmission queue and forwarding the message directly to the clustered queue? |
No.
Every time you execute an MQPUT on a queue that is not defined as a QLOCAL on the queue manager you are connected to, WebSphere MQ will automatically and behind the scenes actually write the message to the relevant transmission queue that is linked to the channel that identifies the route to the queue manager that you have named in your MQOPEN that created the Hobj that you are passing to MQPUT.
MQ will then be responsible for doing all of the right things to ensure that the message is successfully moved to the end destination based on the quality of service you selected (persistant/non-persistent for example).
You likely do not want to use an MQ Cluster to connect queue managers across unreliable network links. This can lead to messages sitting on the SYSTEM.CLUSTER.TRANSMIT.QUEUE for a long period of time, which can lead to impact on messages going to other queue managers in the same cluster.
But your code makes NO NOTICE of this. |
|
| Back to top |
|
|
|
|
