| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Messages staying in SYSTEM.BROKER.AGGR.UNKNOWN queue |
« View previous topic :: View next topic » |
| Author |
Message
|
| joebuckeye |
 Posted: Wed May 31, 2017 9:13 am Post subject: Messages staying in SYSTEM.BROKER.AGGR.UNKNOWN queue |
 |
|
Partisan
Joined: 24 Aug 2007
Posts: 365
Location: Columbus, OH
|
Seeing some "strange" aggregation issue since our migration from v8 to IIB 10.0.0.7 running on RHEL Server 7.3. This may have been happening under v8 but every error now is being "blamed" on the migration to IIB 10, so I have been investigating.
We have an IIB application that contains an aggregation that has three fan outs, two are service calls and the third is the original message. The application has an HTTP Input node so the original message also contains the HTTP Context variable (ie RequestIdentifier) so the HTTP Reply node can send a response to the caller. The fan out and fan in functionality are in their own flows.
The aggregation timeout is 9 seconds while the Unknown message timeout is 1 second.
When the application is under load we see messages stay in the SYSTEM.BROKER.AGGR.UNKNOWN queue.
I have a test setup to send 100 requests to the application in one second. Sometimes 100 responses are sent back and sometimes up to 50 responses are not sent back. The number of responses not sent back matches the number of messages that are stuck in the SYSTEM.BROKER.AGGR.UNKNOWN queue.
The problem is around the original message that contains the HTTP Context information being stuck in the SYSTEM.BROKER.AGGR.UNKNOWN queue. Since that message is never being released the aggregation reply message that is passed out of the Timeout terminal of the Agg Reply node does not contain the HTTP Context so no response can be sent back to the consumer.
Investigating this further lead me to bump up the Unknown message timeout to 3 seconds but the inconsistent behavior is still being seen.
We are curious why the messages are staying in the SYSTEM.BROKER.AGGR.UNKNOWN queue since we have an Unknown timeout message value set. That means the message should either be part of the aggregation response message or sent out the Unknown terminal (if the message arrives before the Aggregate Request node finishes it's work). This is based on the details here - https://www.ibm.com/support/knowledgecenter/SSMKHH_10.0.0/com.ibm.etools.mft.doc/ac12320_.htm. This leads us to believe that there really shouldn't be any messages staying on the SYSTEM.BROKER.AGGR.UNKNOWN queue.
I am just checking in here to see if anyone has any idea why we could be seeing this behavior. |
|
| Back to top |
|
 |
| joebuckeye |
| Posted: Wed May 31, 2017 9:37 am Post subject: |
|
|
Partisan
Joined: 24 Aug 2007
Posts: 365
Location: Columbus, OH
|
Of course I did this in reverse order, I posted my message and then searched to see if others had a similar issue.
I found a thread (http://www.mqseries.net/phpBB2/viewtopic.php?p=401783&sid=3c1a7d3e93dd89b5f4460dc919bcbbd3) where it was suggested that you set the Transaction mode to 'Yes' (as opposed to 'Automatic') on the MQ Output nodes in the fan out flow.
I have done that and now have had 4 straight test runs with 100 out of 100 response messages being sent back. And no growth in the number of messages in the SYSTEM.BROKER.AGGR.UNKNOWN queue.
Thanks fjb_saper for your suggestion in that thread.
I would say that the Info Center page talking about this refers to the input node on your fan out flow and not the output nodes before the the Aggregation Request node. That threw me. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Jun 01, 2017 4:25 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
You're welcome. 
What is not enough emphasized, is the behavior of the aggregation output, when there is no transaction present. As the output order is mostly random, there is no guarantee that the original message arrives in time at the aggregation input node when under load. I believe that is what you were seeing.
Setting the transaction to Yes ensures that the outbound messages are all available at the same time. This would ensure that your original message is available within the unknown time interval.
Glad you found your solution 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| joebuckeye |
| Posted: Thu Jun 01, 2017 4:37 am Post subject: |
|
|
Partisan
Joined: 24 Aug 2007
Posts: 365
Location: Columbus, OH
|
| fjb_saper wrote: |
| What is not enough emphasized, is the behavior of the aggregation output, when there is no transaction present. As the output order is mostly random, there is no guarantee that the original message arrives in time at the aggregation input node when under load. I believe that is what you were seeing. |
Yep. We were mystified by the aggregation reply node outputting an empty message from the Timeout terminal. And then seeing (most of the time) all the separate pieces being sent through the Unknown terminal.
I still think there might be a bug in aggregation handling because in many cases our original messages were stuck in the AGGR.UNKNOWN queue. I don't believe this should be happening. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Jun 01, 2017 4:48 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| joebuckeye wrote: |
| fjb_saper wrote: |
| What is not enough emphasized, is the behavior of the aggregation output, when there is no transaction present. As the output order is mostly random, there is no guarantee that the original message arrives in time at the aggregation input node when under load. I believe that is what you were seeing. |
Yep. We were mystified by the aggregation reply node outputting an empty message from the Timeout terminal. And then seeing (most of the time) all the separate pieces being sent through the Unknown terminal.
I still think there might be a bug in aggregation handling because in many cases our original messages were stuck in the AGGR.UNKNOWN queue. I don't believe this should be happening. |
It mostly has to do with the behavior under load. As I said the order of the fan out is random. Now imagine the original message comes last. And you might have swapping of context because of the load... So your messages are gone and answered while you are still struggling to output the original message...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
