MQSeries.net :: View topic - [newbie] Pls Help me review my design on aggregate

MQSeries.net

Tech Exchange

Education

Certifications

Library

Info Center

SupportPacs

FAQÂ Â

Usergroups

RSS Feed - WebSphere MQ Support

RSS Feed - Message Broker Support

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » [newbie] Pls Help me review my design on aggregate

[newbie] Pls Help me review my design on aggregate

« View previous topic :: View next topic »

Author

Message

khlim2005

Posted: Thu Sep 22, 2011 12:02 am Post subject: [newbie] Pls Help me review my design on aggregate

Newbie

Joined: 20 Sep 2011
Posts: 4

Hi All,

Iâ€™m new to this forum as well as MQ, hope those experience can help. Thanks alot. This is my first task to use MQ/WB as a solution to below scenario.

1.An existing application will send (in xml format) to (starting) input queue. (XML_IN)

2.My broker needs to send/spread out this xml message to four external applications. (I use aggregate node â€˜fan outâ€™ for this purpose). (TO_APP_1/TO_APP_2 as example)

3.This four external applications will â€˜extractâ€™ the require info at their end and response with related data. (COLLECT_ALL_APP_REPLY)

4.Iâ€™ll have to combine all returning responses and send out to the final receiving application. (I â€˜fan inâ€™ with aggregateReply) (BuildReply).

5. In events that not every application reply, I build a filter node â€˜process_partial_msgâ€™ and check the response if it contain any reply, for example â€˜<replyFromApp2>â€™, if yes, continue, else send to fail/timeout queue.

6.I build a simulation reply (in red box) that function like above (3), manually adding information to xml

I set aggregate timeout to be 10 minutes and in the compute node (Add_Content / Add_Content_2) I set in ESQL expiry of message to be 5 minutes. I test the message flow and it works, from input to output.

Just wondering, can anyone points out if I miss anything important (other than error handling) in the flow or feel inform me how i can enhance or improve my design.

By the way, one important question I canâ€™t understand. If for example, thereâ€™s multiple instances of sender application sending the original xml to my first input, will (application) replies node connecting to my aggregateReply (COLLECT_ALL_APP_REPLY) get confuse, which response message belong to which instances?

Thanks alot!

Last edited by khlim2005 on Thu Sep 22, 2011 1:23 am; edited 1 time in total

fjb_saper

Posted: Thu Sep 22, 2011 12:36 am Post subject: Re: [newbie] Pls Help me review my design on aggregate

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY

khlim2005 wrote:

Hi All,

Iâ€™m new to this forum as well as MQ, hope those experience can help. Thanks alot. This is my first task to use MQ/WB as a solution to below scenario.

1.An existing application will send (in xml format) to (starting) input queue. (XML_IN)

2.My broker needs to send/spread out this xml message to four external applications. (I use aggregate node â€˜fan outâ€™ for this purpose). (TO_APP_1/TO_APP_2 as example)

3.This four external applications will â€˜extractâ€™ the require info at their end and response with related data. (COLLECT_ALL_APP_REPLY)

4.Iâ€™ll have to combine all returning responses and send out to the final receiving application. (I â€˜fan inâ€™ with aggregateReply) (BuildReply).

6.I build a simulation reply (in red box) that function like above (3), manually adding information to xml

First let me congratulate you on an excellently described case. Not every newbie manages that.

Going on: you showed all flows in one picture. I hope that this really displays 3 flows:

Fan out flow
Application simulation flow
Fan in flow

Let's have a look at what is missing...

The fan out flow does not contain any information as to where to send the response. You would ideally need to serialize the MQMD of the request message, 64 bit encode it and send it to a "dummy" app, or manipulate before agg req node, and have it show up in the aggregation fan in. This way the fan in knows where to send the reply to (as everything is done by correlation Id).
Typically I do this as one of the branches of the fan out. For all MQOutput nodes in the aggregation request a new msgid is generated and no correlation id is set. This design has no problem with multiple instances.
From the picture is does look that you are using MQOutput nodes and not MQReply nodes. On my design the application receives a request / reply type message where the reply is sent to the aggregation fan in queue. The application replies and the correlationId is set to the messageId of the message being handled by the app.
In the same way the fan in flow extracts the MQMD of the original message from the aggregation, 64bit decodes it, parses it and uses the information thereon to populate the response MQMD before sending the message to the MQReply node.
if you have the aggregation timeout set at 10 mins, why would you set only a 5 min time out on the request?
Assuming that the request time out is passed forward to the applications in the fan out, an aggregation time out of 310 seconds should be sufficient (5 mins 10 secs)
finally and to make sure your flow does perform, with timeout like those, you need to have the fanout and fanin be in 2 separate, unrelated flows in the same execution group...

Hope this answers most of your questions.

_________________
MQ & Broker admin

mqjeff

Posted: Thu Sep 22, 2011 1:49 am Post subject: Re: [newbie] Pls Help me review my design on aggregate

Grand Master

Joined: 25 Jun 2008
Posts: 17447

fjb_saper wrote:

First let me congratulate you on an excellently described case. Not every newbie manages that.

fjb_saper wrote:

The fan out flow does not contain any information as to where to send the response. You would ideally need to serialize the MQMD of the request message, 64 bit encode it and send it to a "dummy" app, or manipulate before agg req node, and have it show up in the aggregation fan in. This way the fan in knows where to send the reply to (as everything is done by correlation Id).
Typically I do this as one of the branches of the fan out. For all MQOutput nodes in the aggregation request a new msgid is generated and no correlation id is set. This design has no problem with multiple instances.

I want to take a minute to comment on this point. What fjb_saper is saying is that the Fan out flow does not, at least as shown, preserve the necessary information about where the AGGREGATED response needs to go. He is assuming that the Fan In flow is called in a request-reply paradigm, and thus you need to store the ReplyTO queue and MsgId/CorrelID for the final response so the original requesting application can correlate the total response with it's original request.

One way to do this is to forward the necessary data as another "request" that is then part of the aggregation.

Another way to do this may be the way you are taking already, that the each request to each aggregated service contains the needed information, and this is then returned back as part of the partial reply.

khlim2005

Posted: Thu Sep 22, 2011 1:56 am Post subject: Re: [newbie] Pls Help me review my design on aggregate

Newbie

Joined: 20 Sep 2011
Posts: 4

fjb_saper wrote:

First let me congratulate you on an excellently described case. Not every newbie manages that.

Going on: you showed all flows in one picture. I hope that this really displays 3 flows:

Fan out flow

Application simulation flow

Fan in flow

I also have to thanks you for your kind advice and very fast response. I indeed have three flow (out, reply,in .msgflow files) as I follow closely to the IBM aggregate (and airline reservation) sample.

Please allow me more time to understand your 5 points but will try to answers some that I know.

Quote:

if you have the aggregation timeout set at 10 mins, why would you set only a 5 min time out on the request?
Assuming that the request time out is passed forward to the applications in the fan out, an aggregation time out of 310 seconds should be sufficient (5 mins 10 secs)

I get you, I overly caution to buffer 5 minutes overhead.

Quote:

From the picture is does look that you are using MQOutput nodes and not MQReply nodes. On my design the application receives a request / reply type message where the reply is sent to the aggregation fan in queue. The application replies and the correlationId is set to the messageId of the message being handled by the app.

If I get you correctly, you are refering to my middle flow in the red color box last MQOutput node (COLLECT_ALL_APP_REPLY)

The reason why I dun use MQReply node is I'm trying to better understand/learn how aggregate work so I manually simulate a reply function. In the compute nodes (Process_1 / Process_2), I write the following ESQL statement (actually I copy the code from the sample,

Code:

SET OutputRoot.Properties = InputRoot.Properties;
SET OutputRoot.MQMD.MsgId=InputRoot.MQMD.MsgId;
SET OutputRoot.MQMD.CorrelId =InputRoot.MQMD.MsgId;
SET OutputRoot.MQMD.Report = MQRO_PASS_CORREL_ID;
SET OutputRoot.MQMD.ReplyToQ=InputRoot.MQMD.ReplyToQ;

Thanks.

khlim2005

Posted: Thu Sep 22, 2011 2:07 am Post subject: Re: [newbie] Pls Help me review my design on aggregate

Newbie

Joined: 20 Sep 2011
Posts: 4

mqjeff wrote:

fjb_saper wrote:

First let me congratulate you on an excellently described case. Not every newbie manages that.

Thanks alot mqjeff!

I'll try to understand your advice. If I get you correctly, In my application simulation flow, on both compute odes, I've these code:

Code:

Kindly advice me if I get you, thans very much

khlim2005

Posted: Tue Sep 27, 2011 1:44 am Post subject:

Newbie

Joined: 20 Sep 2011
Posts: 4

Hi All, I've completed m design based on fjb_saper and mqjeff advice (thanks alot) and get it working correctly. However, in this process, i encounter some question which puzzle me, and hope somebody can enlighten me.

1.)
I read on aggregate and know it required request/reply message; as also quote by jefflowrey in this post

Quote:

You need to go AggRequest->Compute->MQReply.

Every message that shows up in the Fan-In flow must be a reply message.

The easiest way to build a reply message is to use an MQReply node.

http://www.mqseries.net/phpBB/viewtopic.php?t=39123&highlight=unknown+message+timeout

Maybe I misunderstood 'Reply' because when I set my reply msg as Datagram to the fan-in MQInput node, it still get propagates by AggregateReply node.

Using rfhutil. In this screen, I also set the Correl Id to the same as MsgId

So message type doesnt affect?

2.)
I dun fully understand this post

http://www.mqseries.net/phpBB/viewtopic.php?t=17057&highlight=unknown+message+timeout

Quote:

So I assume that if there are multiple message flow instances in an execution group (or even multiple execution groups with the same flow) that the AggReply node will "queue" any reply messages it encounters, even if they are really for another instance of the flow

You're right!.

Am i correct to assume that AggregateReply ignore instances and avoid multiple instance to my flow?

fjb_saper

Posted: Tue Sep 27, 2011 6:08 am Post subject:

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY

I have no problems with multiple instances of the flow (x fan out + x fan in).
The start message is of type request and not datagram.
The MQMD holds the replyto information.

Have fun

_________________
MQ & Broker admin

Display posts from previous:

Page 1 of 1

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » [newbie] Pls Help me review my design on aggregate

Jump to:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

Protected by Anti-Spam ACP