| Author |
Message
|
| new_to_wmb8 |
 Posted: Tue Oct 21, 2014 4:05 am Post subject: Maintain the Sequence of messages |
 |
|
Centurion
Joined: 28 May 2013
Posts: 127
Location: Hyderabad, India
|
Hi All,
I have an requirement in which when the output queue is full, we don't loose the messages.
we also have to maintain the sequence because when the queue get empty later on then it will process the message in same sequence as the input queue processes.
any idea ??
i have tried it using MQ out put queue in which when output queue is full then it will put the message on temporary queue and using MQGET , we will pick it but in that case as well my sequence doesn't maintain.
Please provide any suggestion if you have
Thanks in advance. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Tue Oct 21, 2014 5:26 am Post subject: Re: Maintain the Sequence of messages |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| new_to_wmb8 wrote: |
| I have an requirement in which when the output queue is full, we don't loose the messages. |
Make the messages persistent then WMQ will do this for you.
| new_to_wmb8 wrote: |
| we also have to maintain the sequence because when the queue get empty later on then it will process the message in same sequence as the input queue processes. |
If you have to do this because the processing of a message relies on the message processed immediately prior to it, this is called "message affinity" and is a bad practice because it requires you to maintain sequence which, as you've found, is really hard. So if you're using message affintiy you're in a trap of your own making, if you're not using it then see my answer above.
| new_to_wmb8 wrote: |
| i have tried it using MQ out put queue in which when output queue is full then it will put the message on temporary queue and using MQGET , we will pick it but in that case as well my sequence doesn't maintain. |
Set the messages to persistent and let WMQ perserve them for you. When your flow gets a queue full error it should abend and have the sending system retry. When the queue empties things will go back to normal.
If you have a situation where you must maintain delivery sequence (i.e. single threaded and step dependant), the price is that when one step fails the whole process has to stop until the problem is resolved. This is unacceptable in the majority of real world situations and hence message affinity is outlawed in the majority of real world situations.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| new_to_wmb8 |
| Posted: Tue Oct 21, 2014 6:10 am Post subject: |
|
|
Centurion
Joined: 28 May 2013
Posts: 127
Location: Hyderabad, India
|
Hi,
Thanks for the suggestion.
Actually the situation is some thing like we are hitting a server in one message and if the server is down , we have to retry that message in definite time till the issue is resolved because these messages are high priority message and we cant loose these messages.
So when server is down the output queue of the prior application which is the input queue for this flow get full and we start loosing message.
We cannot simply put the message on holding or temporary queue that will disturb the sequence.
Any idea to achieve this??? |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Oct 21, 2014 6:28 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| new_to_wmb8 wrote: |
| Any idea to achieve this??? |
Use persistent messages with an HA solution (or a WMQ cluster, having reviewed the many posts in this forum on the advantages & disadvantages of using a cluster for HA).
You've also failed to clarify why the sequence of the messages is important. Is it important because the messages must be processed in sequence (message affinity) or is it important because you need to process the oldest first (message SLA).
| new_to_wmb8 wrote: |
| we have to retry that message in definite time till the issue is resolved because these messages are high priority message |
If the queue is full because the downstream system that processes them is out because of a hardware failure, you can retry as much as you like in definite time and it's not going to help you. You need to have a much more holistic HA strategy.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| new_to_wmb8 |
| Posted: Tue Oct 21, 2014 8:33 am Post subject: |
|
|
Centurion
Joined: 28 May 2013
Posts: 127
Location: Hyderabad, India
|
| We have to send the message in sequence to third party. they need them in sequence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Oct 21, 2014 8:59 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| new_to_wmb8 wrote: |
| We have to send the message in sequence to third party. they need them in sequence. |
So it's the message affinity reason - now you know why this is such a bad idea.
I stand by all my comments above.
So all you can do is stack the messages in a queue (with their persistence attribute set) and once the queue fills, your process will abend. So make sure the queue max depth is nice and high, you have bags of disk space and enjoy the Hell of message affinity.
Be sure to thank the third party for their stupid design.
It's also worth asking the third party why they need them in sequence. It may well be that you've always sent them in sequence, and a previous version of their system needed them in sequence, but the current version does not. Equally they may request them in sequence for no better reason than they use sequence numbers to ensure no records are missing, and WMQ has much better faciltities than that built in.
Or you can just sit and suffer. Your system, your choice. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Oct 22, 2014 6:40 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Why would it abend when theputting application experiences a MQRC 2053?
Sequencing is not unusual. Use of the WMB node re-sequence can help.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Oct 22, 2014 6:52 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| zpat wrote: |
| Why would it abend when theputting application experiences a MQRC 2053? |
Because the MQOutput node throws it and that leads to an abend.
| zpat wrote: |
| Sequencing is not unusual. Use of the WMB node re-sequence can help. |
You would still need to store the un-puttable messages as the OP has found, and using the resequencing node relies on there being something in the message that can be used as a sequence rather than just the order of arrival.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Oct 22, 2014 7:29 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
By abend you mean transactional rollback?
There are sometimes, unfortunately, real abends in broker - the sort that cause core dumps and currently are the bane of my life.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Oct 22, 2014 7:34 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| zpat wrote: |
| By abend you mean transactional rollback? |
In this context I do - perhaps "exception" would have been a better choice of words.
| zpat wrote: |
| There are sometimes, unfortunately, real abends in broker - the sort that cause core dumps and currently are the bane of my life. |
Not in my brokers there aren't. And if there are, whoever made the last change to OS/SAN/etc is found and punished. Or a random selection of people are blamed and punished.
The principle holds though - the flow gets a 2053 or encounters other problem, rolls back, has to put message someplace while preserving order of arrival. This problem is compounded if the flow is still receiving inbound messages.
I notice that this:
| Vitor wrote: |
| It's also worth asking the third party why they need them in sequence. It may well be that you've always sent them in sequence, and a previous version of their system needed them in sequence, but the current version does not. Equally they may request them in sequence for no better reason than they use sequence numbers to ensure no records are missing, and WMQ has much better faciltities than that built in |
did not receive a response as yet.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Oct 22, 2014 9:54 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
There are a few of things that you might like to consider.
1) Setup your monitoring system to start all the alarmbells ringing loud and clear when the queue gets say 40% full.
2) Make sure that the Queue can hold enough messages for at least 2.5 times the receiving applications SLA for downtime.
3) Think about what happens in your flow when you get a 2053. Do you block processing of any more messages (difficult if running more than one instance but you shouldn't be anyway if you want to preserve the sequence) for ever and a day until you no longer get a 2053. You will need to impliment a retry mechanism but beware about looping too much or your flow may well abend/crash.
4) Don't be afraid to try a load of things out until you are comfortable with your solution.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
|
|
