| Author |
Message
|
| wyatt |
 Posted: Wed Feb 14, 2007 7:27 am Post subject: Message Processing Design Question |
 |
|
Voyager
Joined: 28 Nov 2004
Posts: 76
|
Hi
We are reviewing available options to assemble multiple MQseries messages into 1 logical message.
For Example: Multiple (external) entities are sending data to a single queue. As per interchange agreements, external entities are allowed to send a logical message that spans a physical message; so a series of messages may look like this
Msg 1 of 4=Envelope START, transaction1, transaction2, transaction3…
Msg 2 of 4=transaction3 (cont’d) transaction4, transaction5, transaction6
Msg3 of 4=transaction7, Envelope END, Envelope START
Msg4 of 4=transaction1, Envelope END
The messages are not part of a MQSeries group, the only thing that identifies the message as having an affinity is identical message id. In this example, it necessary to assembly the 3 physical messages into a single logical message for parsing purposes. Approximately 2% of messages will be received in this manner. When the message exceeds a specific length, the process would check to see if additional related messages exist.
Are original solution was to use the MQGet Node available in Message Broker V6, but, this node does not allow MQInput node and MQGet node to read from the same queue (PMR 31071 487 000). PMR resolution will not occur in a timely manner.
As always, comments/thoughts are appreciated. |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Wed Feb 14, 2007 7:36 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Aggregation.
Create a dummy aggregation replier that will always immediately send back the original message.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| wyatt |
| Posted: Wed Feb 14, 2007 11:09 am Post subject: |
|
|
Voyager
Joined: 28 Nov 2004
Posts: 76
|
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Feb 14, 2007 11:13 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Oh, crud. I keep forgetting that Aggregation is based on one input message, not many input messages.
You can use the MQGet node in the way you wanted, but you'll have to create another flow.
Build one flow who's job it is to forward the input messages to another queue, (which will be processed by MQGet) and send control messages to a different queue (which will be processed by MQInput).
Then build the second flow that is initiated by the control messages and uses MQGet in the way you wanted to in the first place.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| daveeason |
| Posted: Wed Feb 14, 2007 11:28 pm Post subject: MQ message segmentation |
|
|
Novice
Joined: 14 Jul 2005
Posts: 18
Location: Canberra, Australia
|
I just had to perform a task very similar to this where it was necessary to combine several messages that were part of one logical batch into a single message. I was able to do this by using the MQ feature of message segmentation. The solution is in two parts;
1) Receive seperate messages and set the MQ segmentation parameters accordingly for each message to indicate that it is part of a larger message, then write message out to a temporary staging queue. When last message received mark it as "last_segment".
2) A message flow reads from this temporary staging queue (WMB uses the MQ GMO option of "GET COMPLETE MESSAGE" which is handy because it is required to make the segmentation solution work  ), segments build up on the input queue as they are received, then when the "last_segment" arrives on the queue WMB retrieves all segments as one complete message and you can process this as desired.
This is not the intended use for MQ segmentation but it is a nice solution to this type of problem.
_________________
Dave Eason
Addis |
|
| Back to top |
|
|
| wyatt |
| Posted: Thu Feb 15, 2007 5:53 am Post subject: |
|
|
Voyager
Joined: 28 Nov 2004
Posts: 76
|
Ok
I don’t believe either scenario will work in this case.
jefflowrey: This might result in a race condition whereby the message flow servicing the "primary" input queue consumes message(s) that have an affinity before the second flow has an opportunity to do so?
daveeason: If messages are interleaved on the input queue, how can I reliably mark the messages correctly for segmentation; e.g.
physical msg1=part 1 of 2
physical msg2=complete message, no segmentation required
physical msg3=part 2 of 2
Are you suggesting that some type of state information be maintained between messages?
Thanks in advance |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Feb 15, 2007 6:11 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Dave's is the better solution, actually. Or you could use logical groups instead of segements.
In either case, what we are talking about is rebatching your transactions into single transaction batches.
If you're saying that each batch has to be processed in order, but that segments of the batch may not come in logical order (for example, you might receive them in exactly the reverse order they need to be processed) ....
Then you can't do this with plain MQ, and you have allowed someone to design a message affinity problem of epic proportions. Message Affinitiy = BAD, and now you know why.
You will have to involve a datastore, and a staging and unstaging process to solve this problem. Then you insert each segment into the datastore as you receive it. A separate process scans the datastore for "the next completed batch", and processes that one.
And note, also, MQ <> datastore.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| wyatt |
| Posted: Thu Feb 15, 2007 6:52 am Post subject: |
|
|
Voyager
Joined: 28 Nov 2004
Posts: 76
|
LOL: re message affinity problems of EPIC proportions.
That is basically what I have been saying but here I am stuck with this problem.
Best |
|
| Back to top |
|
|
| daveeason |
| Posted: Thu Feb 15, 2007 2:29 pm Post subject: |
|
|
Novice
Joined: 14 Jul 2005
Posts: 18
Location: Canberra, Australia
|
wyatt,
To use the solution that I have suggested it is necessary to save state between the messages, you can do this by including the state in the messages themselves (this is nice because there is no overhead of external storage)...
i.e. Each message contains two fields 1) Batch_id, 2) Total_Num_Recs_In_Batch
If you have this information then handling interleaved messages is no problem. I actually do it in my solution. However, trying to re-combine the messages in some particular order other than the order they have been received is another issue. In that case I would think about doing what Jeff has described and using an external datastore to re-combine the messages.
If you are interested in the solution I mentioned previously it is discussed in this article http://www.mqseries.net/phpBB2/viewtopic.php?t=33919&highlight=segmentation+mqmd
_________________
Dave Eason
Addis |
|
| Back to top |
|
|
| daveeason |
| Posted: Thu Feb 15, 2007 2:55 pm Post subject: |
|
|
Novice
Joined: 14 Jul 2005
Posts: 18
Location: Canberra, Australia
|
One further thing, I am actually storing state in an external database that records how many recs I have received at any point in time. There are alternatives to doing this atleast in WMBv6 but my solution does not have a high performance requirement 
_________________
Dave Eason
Addis |
|
| Back to top |
|
|
|
|
