| Author |
Message
|
| v6newbie |
 Posted: Mon Mar 12, 2007 9:26 pm Post subject: Aggregate Node timeout question. |
 |
|
Newbie
Joined: 12 Mar 2007
Posts: 5
|
Hi All,
I've look at the sample Airline reservation that comes with WMB v6 to get a understanding of how the aggregate nodes work.
I've modified it slightly by reducing the timeout in RequestDetails to 3
seconds and changed the unknown timeout in AggregateReplies to 0 secs.
I have stopped XML_FlightQueryReply.msgflow in my execution group to force a timeout.
I would expect that the time between the request and reply would take between 3-4 secs based on my timeout of 3 secs but what I've noticed is that it varies between 3-8 secs, which to me seems quite strange.
(i'm basing this on the put times of the request and reply message)
Can someone explain this behaviour? Is my assumption of the reply message being put on the queue within 1 sec of the timeout interval not realistic?
Thanks in advance. |
|
| Back to top |
|
 |
| vk |
| Posted: Mon Mar 12, 2007 10:08 pm Post subject: |
|
|
Partisan
Joined: 20 Sep 2005
Posts: 302
Location: Houston
|
The time between request and reply messages is not based on the timeout set in the AggregateControl node. It depends on how much time the intermediate applications takes to process the requests and send the corresponding replies.
Regards,
VK. |
|
| Back to top |
|
|
| v6newbie |
| Posted: Mon Mar 12, 2007 10:54 pm Post subject: |
|
|
Newbie
Joined: 12 Mar 2007
Posts: 5
|
Hi vk,
In the airline reservation sample I stopped the XML_FlightQueryReply.msgflow so in effect my intermediate application is not responding at all.
In this this case I would've expected the AggregateReply to "pop" immediately after the timeout interval of 3 secs, and construct a reply message.
So I'm not sure why I'm seeing these erratic response times of up to 8 secs. |
|
| Back to top |
|
|
| elvis_gn |
| Posted: Mon Mar 12, 2007 11:45 pm Post subject: |
|
|
Padawan
Joined: 08 Oct 2004
Posts: 1905
Location: Dubai
|
Hi guys,
| vk wrote: |
| The time between request and reply messages is not based on the timeout set in the AggregateControl node. It depends on how much time the intermediate applications takes to process the requests and send the corresponding replies. |
So do u mean that the Aggregate Reply node will wait for all the time that the intermediate application takes to send the response ? If so, then that understanding is wrong I feel....The Aggregate Control node will tell the Aggregate Reply node of how long it should wait for the responses, suppose the responses never arrive ?....in simple words, Aggregation timing will not depend on another external app...
@ v6newbie:
In MB v5, the Agg Control node sent a message to the Agg Reply node which I think immediately triggered the count of how long to wait for the responses...
In v6, this is done internally using MQ. I think what is happening is that the message is being put into the queue and by the time the put happens and the count starts some seconds are lost.
I could be wrong, actually most probably wrong, but you can verify this my making Aggregation work the way it worked in v5(you need to make some setting in Environment Variables)....and then check is the 3secs is being honored.
Regards. |
|
| Back to top |
|
|
| vk |
| Posted: Tue Mar 13, 2007 12:31 am Post subject: |
|
|
Partisan
Joined: 20 Sep 2005
Posts: 302
Location: Houston
|
The time for which the AggregateReply node waits for replies to come is dependent on the Timeout value set in the AggregateControl node. What I meant was, the Timeout property has no impact on the time taken by the intermediate flow to process a request and send the reply to the AggregateReply flow.
I have not tested the Airline reservation sample. But in my aggregation flow that I migrated from WMQI v2.1, these are the things whcih I observed.
1. When a message passes the AggregateControl node, a Control message is automatically put into the queue SYSTEM.BROKER.AGGR.CONTROL. This will have the aggragation group ID and the timeout value set in AggregateControl node.
2. For each message that passes an AggregateRequest node, the message is stored in the queue SYSTEM.BROKER.AGGR.REQUEST, with the aggregation group ID and the individual message ID.
3. Whenever the AggregateReply node receives a reply, it checks the message ID of the reply with the message ID of the messages in SYSTEM.BROKER.AGGR.REQUEST. If a match is found, the corresponding control message is retrieved from SYSTEM.BROKER.CONTROL.QUEUE and the reply count is updated. The reply message is stored in the SYSTEM.BROKER.AGGR.REPLY queue.
4. The node waits for all replies to come. When the last reply comes, it propagates all the reply messages to the Out Terminal. All messages are deleted from SYSTEM.BROKER.AGGR.CONTROL, REQUEST and REPLY queues.
5. If no reply comes within the Timeout value set in AggregateControl node, then a blank message is sent out of the Timeout terminal. The control message is deleted from the SYSTEM.BROKER.CONTROL.QUEUE, but the messages in SYSTEM.BROKER.AGGR.REQUEST queues remain undeleted. Any reply which comes after Timeout interval will be treated as unknown and sent to the Unknown terminal.
6. If some replies come before Timeout interval, then AggregateReply node waits indefinitely. No timeout is happening in this case. This looks like a defect. Ideally, all the replies which came before Timeout should have been propagated to the Timeout terminal (which was the bahavior in WMQI v2.1). When replies come later after the timeout, the aggregation happens correctly and the message is sent to the Out terminal.
This is my understanding and findings. Please correct me if I am wrong.
Regards,
VK. |
|
| Back to top |
|
|
| v6newbie |
| Posted: Tue Mar 13, 2007 9:03 pm Post subject: |
|
|
Newbie
Joined: 12 Mar 2007
Posts: 5
|
Hi vk,
I have to agree with you up to point 6. If the aggregateReply waited indefinitely I be alittle concerned about using these nodes, since I could not guarantee on an external application responding.
To test this out I modifed the airline sample and removed the connection between GetPassengerDetails and MQMD_ReplyToQ, so I would get back a response for GetFlightDetails and not GetPassengerDetails.
In this case I don't see the AggregateReply node waiting indefinitely. The timeout "pops" but erratically, and the output message does contain the GetFlightDetails details.
Has anyone implemented the aggregate nodes in a production environment and seen this behaviour as well?
elivs_vn,
I've looked at the documentation and it doesn't recommend using the control node
I"n Version 5.0 of the product, fan-out and fan-in flows were also associated by sending control messages from the AggregateControl node to the AggregateReply node. This is no longer necessary. For optimum performance it preferable not to connect the AggregateControl and AggregateReply node." |
|
| Back to top |
|
|
| elvis_gn |
| Posted: Tue Mar 13, 2007 10:44 pm Post subject: |
|
|
Padawan
Joined: 08 Oct 2004
Posts: 1905
Location: Dubai
|
Hi v6newbie,
| v6newbie wrote: |
| I've looked at the documentation and it doesn't recommend using the control node |
If I made an advanced version of a tool, would I ask you to go back and use my older version ?
| v6newbie wrote: |
| I"n Version 5.0 of the product, fan-out and fan-in flows were also associated by sending control messages from the AggregateControl node to the AggregateReply node. This is no longer necessary. For optimum performance it preferable not to connect the AggregateControl and AggregateReply node." |
I did not ask you to use the older concept, I just asked you to verify....Since you did so much reading perhaps you know the reason why they say the performace is better in v6, what the current implementation has become,...and why perhaps ur getting the time discrepancy that ur getting ?
Regards. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Mar 14, 2007 2:13 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Well the time out node does not pop up erratically...
What you need to do is minimize the potential of a reply before the control information ... See documentation...
This also synchronizes your requests... (hint the fan out flow is under syncpoint)
Everything works fine for us this way...
Now if you have problems it could be because your fan out setup is not ideal for aggregation:
- before the control node remove any information about correl ids by setting it to MQI_NONE.
- On the output nodes (just before the request node) make it generate a new message Id and nothing else.
- don't know if it will handle a reply coming in as a logical message (multiple single messages like in segmentation or grouping)
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| sunny_30 |
| Posted: Wed Mar 14, 2007 11:46 am Post subject: |
|
|
Master
Joined: 03 Oct 2005
Posts: 258
|
Hi Saper,
I am using the Aggregate functionality in my flows and everything looks to be working fine for me.
Exactly as you said: Before each request the CorrelId is set to MQI-NONE and also a New MsgId is generated for each request (in the Output).
The Out & Timeout terminals of the AggregateReply are kicking up fine in all scenarios.
But, I looked at the discussion of this same topic in the other threads, which state that the end application that receives the requests need to copy the MsgId to the CorrelId of the Reply. Is this a needed functionality and what cd be the potential loss of not doing so?. This is not happening for the replies in my case. For me, in all the replies the CorrelId is also coming out to be MQI_NONE i.e all zeros. And also, strangely, the MsgIds for all the replies are all different from the Request MsgIds. I just want to know that how my aggregation part is working technically?
For each request, I am successfully able to get multiple-responses back using Aggregate. After the Output node of the flow and before the AggregateRequest node, I inserted a compute node to add n-1 extra dummy 'WrittenDestination's in the OutputLocalEnvironment which did the trick. This way I am able to get 'n' responses back for each single-Agg-Req sent (i.e to the out terminal of the AggregateReply instead of the unknown terminal). These all replies are coming with a same MsgId and still look to be working fine. With the setup I have when I deploy multiple instances of the same flow, can I guarentee that the replies are matched to their corresponding requests? Will they be overlapping for my scenario? Do I have to make any more changes?
Thanks and please respond back to my question.
-Sunny. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Mar 14, 2007 2:15 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| sunny_30 wrote: |
Hi Saper,
I am using the Aggregate functionality in my flows and everything looks to be working fine for me.
Exactly as you said: Before each request the CorrelId is set to MQI-NONE and also a New MsgId is generated for each request (in the Output).
The Out & Timeout terminals of the AggregateReply are kicking up fine in all scenarios.
But, I looked at the discussion of this same topic in the other threads, which state that the end application that receives the requests need to copy the MsgId to the CorrelId of the Reply. Is this a needed functionality and what cd be the potential loss of not doing so?. This is not happening for the replies in my case. For me, in all the replies the CorrelId is also coming out to be MQI_NONE i.e all zeros. And also, strangely, the MsgIds for all the replies are all different from the Request MsgIds. I just want to know that how my aggregation part is working technically? |
From my experience the aggregation only worked correctly if the replies had flipped the messageId to the correlId. You can use a compute node to do that before putting the reply on the FanIn input queue if you have to.
Our services are well behaved and will flip it when creating the response...
| sunny_30 wrote: |
For each request, I am successfully able to get multiple-responses back using Aggregate. After the Output node of the flow and before the AggregateRequest node, I inserted a compute node to add n-1 extra dummy 'WrittenDestination's in the OutputLocalEnvironment which did the trick. This way I am able to get 'n' responses back for each single-Agg-Req sent (i.e to the out terminal of the AggregateReply instead of the unknown terminal). These all replies are coming with a same MsgId and still look to be working fine. With the setup I have when I deploy multiple instances of the same flow, can I guarentee that the replies are matched to their corresponding requests? Will they be overlapping for my scenario? Do I have to make any more changes?
Thanks and please respond back to my question.
-Sunny. |
Don't know about your setup.
what I have seen though is that if the correlId of the reply msg does not match the msgid of the request msg the message folder being picked could be arbitrary and other unwanted side effects especially in multiple instances where the messages would go cross instances...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| v6newbie |
| Posted: Thu Mar 15, 2007 12:55 am Post subject: |
|
|
Newbie
Joined: 12 Mar 2007
Posts: 5
|
Hi elvis_gn,
point taken. I reverted back to using the control messages by set the environment variable, wiped the correlid and set ticked the new msg id's of the request messages and it still takes between 3-7 secs to reply.
I varied the timeout value to 10,20, 30 secs and each time it appears as though the reply takes between 1-4 secs extra to produce the response as it leaves the aggregateReply node.
I'm running the test on a windows XP machine with P4 2.8Ghz and 2Gig RAM, so I'm not sure if it's a limitation of my PC.
Will need to try on our AIX machine to see if it produces the same result. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Mar 15, 2007 3:03 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| v6newbie wrote: |
I varied the timeout value to 10,20, 30 secs and each time it appears as though the reply takes between 1-4 secs extra to produce the response as it leaves the aggregateReply node.
I'm running the test on a windows XP machine with P4 2.8Ghz and 2Gig RAM, so I'm not sure if it's a limitation of my PC.
Will need to try on our AIX machine to see if it produces the same result. |
How big is your aggregated message? Could it be the case where the retrieval from MQ affects your timing? Retrieving huge messages will slow you down... (and yes size matters)
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| v6newbie |
| Posted: Thu Mar 15, 2007 3:27 pm Post subject: |
|
|
Newbie
Joined: 12 Mar 2007
Posts: 5
|
Hi fjb_saper,
the reply for the airline reservation sample is quite small, less than 1K when I remove the connection for the GetPassengerDetails.
I can definitely see the control message removed off the queue immediately after the timeout period of 3secs, so my previous statement of the timeout popping is incorrect.
I'm not sure why it would take the 1-4 secs to simply construct the final message given that it is so small.
With your aggregate flows when a timeout occurs do you see it construct a message within a second of the timeout, or do you see the same behaviour as me where it takes a few secs? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Mar 15, 2007 8:18 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
We have no control message(V6) linking fan out and fan in. The fan out and the fan in are in 2 different flows. The fan out is under syncpoint. The overall timeout for the flow is 2 seconds.
The providers will have to answer in under 1 second which is the aggregation timeout. (min agg timeout , add 1 sec to aggregate the message, wrap in soap etc...)
We see very few time outs. Don't know where you need the extra 2 seconds from...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
