| Author |
Message
|
| sandeep9678 |
 Posted: Tue May 08, 2012 10:41 pm Post subject: Message Sequencing |
 |
|
Apprentice
Joined: 04 Aug 2008
Posts: 41
Location: India
|
We are designing the flow for in which message should be delivered in the sequence as they arrive. This requirement is important as our production line sends different updates on same message. Can we have message flow deployed on multiple Brokers which are in cluster and still achieve message Sequencing?
Is it possible with Resequence node while we get Timestamp as part of payload for ensuring Sequencing?
Or do we need to design flows with single threaded and deploy on one broker & execution group only? |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Tue May 08, 2012 10:57 pm Post subject: Re: Message Sequencing |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
What you are describing is 'Message Affinity'. There are numerous posts in this forum talking about why this is a bad thing.
| sandeep9678 wrote: |
Or do we need to design flows with single threaded and deploy on one broker & execution group only? |
This is probably the way to do it. You could have the flow deployed on more than one broker but as long as you don't send any messages to it then you are good to go especially if the first broker/system fails for some reason.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| sandeep9678 |
| Posted: Tue May 08, 2012 11:07 pm Post subject: Re: Message Sequencing |
|
|
Apprentice
Joined: 04 Aug 2008
Posts: 41
Location: India
|
Thanks smdavies99 for quick reply.
| smdavies99 wrote: |
| You could have the flow deployed on more than one broker but as long as you don't send any messages to it then you are good to go especially if the first broker/system fails for some reason. |
We have plan to have flows in stopped state on other brokers. I will search on this forum for related messages for more information.
We were trying to find out if any ways preset to have flows deployed on multiple brokers and still achieve message Sequencing.
_________________
Cheers,
Sandeep |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed May 09, 2012 2:17 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Have you thought about processing off a DB?
- Phase 1 insert into DB with id and counter or timestamp
- Phase 2 check all records with id and counter that have not been processed
- Phase 3 lock all records with id =x and counter between y and z (that are ready for processing ). Individually lock all single records. If there is a gap consider locking operation failed and roll back...
- Phase 4 process locked records
- Phase 5 mark locked records as processed
That should allow you to have multiple instances across multiple hosts trying to process.
Even if you have only a single flow processing and no DB, I would still recommend the use of a resequencing node as you are not guaranteed by the clustering network for the messages to arrive in order...
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed May 09, 2012 5:00 am Post subject: Re: Message Sequencing |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| sandeep9678 wrote: |
| We were trying to find out if any ways preset to have flows deployed on multiple brokers and still achieve message Sequencing. |
You're trying to have your cake and eat it too. If you have message affinity you can't distribute the message workload.
The design suggested by my most worthy associate is a viable solution is that you remove the affinity from the messages and "resequence" them in a database. You'll need to exercise a small amount of caution to ensure that the time you save having the messages processed in a distributed way isn't lost in database I/O.
But message affinity is a bad thing, for this and many other reasons.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Esa |
| Posted: Wed May 09, 2012 5:10 am Post subject: Re: Message Sequencing |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| Vitor wrote: |
| If you have message affinity you can't distribute the message workload. |
You can distribute the transformation workload, but the transformation flow instances need to feed a resequencing flow that must run in one single instance.
But it seems this is not what the OP is after. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed May 09, 2012 5:21 am Post subject: Re: Message Sequencing |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Esa wrote: |
| Vitor wrote: |
| If you have message affinity you can't distribute the message workload. |
You can distribute the transformation workload, but the transformation flow instances need to feed a resequencing flow that must run in one single instance. |
Well yes, but it's fairly seldom that you can (or want to) split tranformation & processing. Also in the kind of scenario you describe you need to cater for the situation where message 3 is transformed faster than message 2 as well as the situation where the transformation of message 2 fails utterly due to non-message related circumstances. Hence the use of a database as both storage & a means of identifying gaps (where the affinity is not the length of the message stream).
Once you've built all that, you've almost certainly lost all the time you saved fanning out.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed May 09, 2012 5:42 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You could use ExtremeScale to hold the 'next' sequence number to be processed, and have every distributed flow sync against that, and push anything that's not the 'next' one to be processed into a delay queue....
  |
|
| Back to top |
|
|
| Esa |
| Posted: Wed May 09, 2012 5:57 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| mqjeff wrote: |
You could use ExtremeScale to hold the 'next' sequence number to be processed, and have every distributed flow sync against that, and push anything that's not the 'next' one to be processed into a delay queue....
|
Sounds like a very funny way of 'scaling'. Disallowing the distributed instances to run simultaneously, if I understand correctly  |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed May 09, 2012 6:37 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Esa wrote: |
| mqjeff wrote: |
You could use ExtremeScale to hold the 'next' sequence number to be processed, and have every distributed flow sync against that, and push anything that's not the 'next' one to be processed into a delay queue....
|
Sounds like a very funny way of 'scaling'. Disallowing the distributed instances to run simultaneously, if I understand correctly |
Presumably more than one sequence would be processed at once. So that sequence one is processed in parallel with sequence two - but the individual messages within each sequence are processed ... umm... sequentially.... |
|
| Back to top |
|
|
| Esa |
| Posted: Wed May 09, 2012 6:44 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| mqjeff wrote: |
Presumably more than one sequence would be processed at once. So that sequence one is processed in parallel with sequence two - but the individual messages within each sequence are processed ... umm... sequentially.... |
OK, so it would scale throughput only, not response time or processing time per sequence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed May 09, 2012 7:32 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Esa wrote: |
| mqjeff wrote: |
Presumably more than one sequence would be processed at once. So that sequence one is processed in parallel with sequence two - but the individual messages within each sequence are processed ... umm... sequentially.... |
OK, so it would scale throughput only, not response time or processing time per sequence. |
Which brings us back full circle to "message affinity prevents distribution".
You can dance round this truth in various ways, but the solution is to design (or re-design) out the affinity.
YMMV but I've found down the years a lot of "message affinity" is caused by the system originally being designed to read a flat file & just converted, or being designed by people used to flat files. A lot of "message affinity" is just habit. Overlapping updates (which sounds like what the OP is describing) can be handled.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| sunny_30 |
| Posted: Thu May 10, 2012 8:14 am Post subject: |
|
|
Master
Joined: 03 Oct 2005
Posts: 258
|
you could deploy multiple mirror copy flows listening on different inputs.
A driver flow can use "Modulus" to determine where a particular message needs to route to, there by not losing sequence for updates to a 'same message'
This solution is scalable too, depending on your workload
example:
assume they are mq messages
say there are total 100 different messages that can be grouped into 10 different groups
a group say can be identified by its primary INT key (say PK)
route each message as per PK (modulus) 5. result will always be in range:0 to 4
1 driver flow, 5 child threads for parallel processing
5 child flows are indexed / identified by the modulus result (0 to 4)
eg: groupId- 100002021 will go to queue1. 100002024 will go to queue4. 100002020 will land up always in queue0 etc
so each thread with this setup processes only two groups (as we have total 100 messages, 10 groups, 5 destination queues: queue0 to queue4)
sequence within a group will never be lost. messages pertaining to a particular group will always use same queue
you could even set FIFO at mq level for each thread/ child flow
its scalable: you could have 10 or 20 or 30 flows to process 10 or 100 or 200 different groups |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu May 10, 2012 9:45 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| sunny_30 wrote: |
you could deploy multiple mirror copy flows listening on different inputs.
A driver flow can use "Modulus" to determine where a particular message needs to route to, there by not losing sequence for updates to a 'same message'
This solution is scalable too, depending on your workload
example:
assume they are mq messages
say there are total 100 different messages that can be grouped into 10 different groups
a group say can be identified by its primary INT key (say PK)
route each message as per PK (modulus) 5. result will always be in range:0 to 4
1 driver flow, 5 child threads for parallel processing
5 child flows are indexed / identified by the modulus result (0 to 4)
eg: groupId- 100002021 will go to queue1. 100002024 will go to queue4. 100002020 will land up always in queue0 etc
so each thread with this setup processes only two groups (as we have total 100 messages, 10 groups, 5 destination queues: queue0 to queue4)
sequence within a group will never be lost. messages pertaining to a particular group will always use same queue
you could even set FIFO at mq level for each thread/ child flow
its scalable: you could have 10 or 20 or 30 flows to process 10 or 100 or 200 different groups |
If the messages can be grouped like this, you can achieve the same thing by using WMQ groups without the need for an intermediate flow.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| sunny_30 |
| Posted: Thu May 10, 2012 10:00 am Post subject: |
|
|
Master
Joined: 03 Oct 2005
Posts: 258
|
| Vitor wrote: |
| If the messages can be grouped like this, you can achieve the same thing by using WMQ groups without the need for an intermediate flow. |
yes MQ grouping works if the sending application beforehand can determine where the group should logically end. This usually is not the case with "update events". Updates to an initial insert via middlelayer can happen any number of times.
My solution applies better & can be multi-threaded for this kind of example scenario: All the records of the same color must be processed in order, but the purples, blues, and reds can be processed in parallel.
Last edited by sunny_30 on Thu May 10, 2012 10:05 am; edited 1 time in total |
|
| Back to top |
|
|
|
|
