| Author |
Message
|
| lenar |
 Posted: Tue Sep 13, 2011 2:25 am Post subject: using MQ as events broker |
 |
|
Newbie
Joined: 13 Sep 2011
Posts: 3
|
Hi
I want to use MQ queue as synchronization of events from parallel processes.
I need something like this:
application ProducerA put message DoneA to queue with transaction TID,
application ProducerB put message DoneB to queue with transaction TID,
application ProducerC put message DoneC to queue with transaction TID,
application Consumer wait until there are three messages in queue DoneA,DoneB,DoneC with the same TID and then get these messages in UOW and communicate: all work with TID is done.
Producer X produce messages DoneX with various TID.
Is it possible to be done by using groups? For example first ProducerC put message with groupID=TID , sequence=3 and flag: LAST_MSG_IN_GROUP, next ProducerA put message with groupID=TID , sequence=1, next ProducerB put message with groupID=TID , sequence=2.
Consumer wait with flag: ALL_MSG_IN_GROUP.
I set flag LAST_MSG_IN_GROUP in ProducerC even if ProducerC first finishes his work and first put message to queue - he does not know about it. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Tue Sep 13, 2011 4:19 am Post subject: Re: using MQ as events broker |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| lenar wrote: |
| I set flag LAST_MSG_IN_GROUP in ProducerC even if ProducerC first finishes his work and first put message to queue - he does not know about it. |
This strikes me as a bit of a problem - if C signals the group is complete before A & B have finished it's going to be a bit bad.
IHMO you're better off having the consumer application read the individual messages as they arrive, store them (database?) and apply whatever logic is necessary to decide it's got all the messages for a given transaction. This would also cater for scenarios where A produces 2 messages for a TID giving 4 messages not 3.
If you had WMB (and you've given no indication that you have) then a Collector node would do this for you out of the box.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| lenar |
| Posted: Tue Sep 13, 2011 5:42 am Post subject: |
|
|
Newbie
Joined: 13 Sep 2011
Posts: 3
|
Ok, so when someone put message with flag LAST_MSG_IN_GROUP QueueManagers decide that the group is full.
Another option is to decide that group is full when all seq numbers from 1 to seq number with LAST_MSG_IN_GROUP are filled, so if you put MSG: seq 1, MSG: seg 3 LAST_MSG_IN_GROUP then queue manager will wait until MSG: seq 2 arrive - but it is not like that? |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Sep 13, 2011 5:55 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| lenar wrote: |
| Another option is to decide that group is full when all seq numbers from 1 to seq number with LAST_MSG_IN_GROUP are filled, so if you put MSG: seq 1, MSG: seg 3 LAST_MSG_IN_GROUP then queue manager will wait until MSG: seq 2 arrive - but it is not like that? |
Well it's like that if you specifically write application code to make that happen but the queue manager won't enforce it. It's a sequence number, not a serial number; you can have a group with 1,2,5,6,12,56 if you want.
And note that the queue manager won't even enforce the sequence unless you ask it.
If you do write application code it's going to be tricky, inefficient and will eat CPU. That's why I suggested a database.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Sep 13, 2011 5:57 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
The LAST_MSG_IN_GROUP flag is set by the programmer that creates the message. The purpose is to tell the consuming application that there are no more messages in the group. The qmgr doesn't know, and doesn't care about this - and most other fields in the MQMD.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Sep 13, 2011 5:59 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
You best shot is to log the messages in a DB.
Have an index on unprocessed messages and work on messages with key having count(*) = total expected # of messages.
There is no good solution as there is no time limit as to when all messages will be available... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Sep 13, 2011 6:25 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Or at least don't try to use grouping this way.
just have the Consumer wait until it's received the correct number of DoneX messages. |
|
| Back to top |
|
|
| lenar |
| Posted: Tue Sep 13, 2011 6:37 am Post subject: |
|
|
Newbie
Joined: 13 Sep 2011
Posts: 3
|
|
| Back to top |
|
|
| gbaddeley |
| Posted: Tue Sep 13, 2011 5:25 pm Post subject: Re: using MQ as events broker |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| Vitor wrote: |
IHMO you're better off having the consumer application read the individual messages as they arrive, store them (database?) and apply whatever logic is necessary to decide it's got all the messages for a given transaction. This would also cater for scenarios where A produces 2 messages for a TID giving 4 messages not 3.
If you had WMB (and you've given no indication that you have) then a Collector node would do this for you out of the box. |
I agree with Vitor's suggestions. MQ is designed for asynchronous messaging but the OP is trying to implement a synchronous messaging paradigm with message affinity from independent sources. Its better to implement this design requirement in the application rather than trying to use MQ features (message grouping) which are not designed for this particular use case.
_________________
Glenn |
|
| Back to top |
|
|
|
|
