ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker SupportManaging High load failure scenarios for the Aggregation nod

Post new topicReply to topic
Managing High load failure scenarios for the Aggregation nod View previous topic :: View next topic
Author Message
eai_developer123
PostPosted: Fri Jun 01, 2018 9:14 am Post subject: Managing High load failure scenarios for the Aggregation nod Reply with quote

Newbie

Joined: 21 May 2018
Posts: 6

A common scenario in WMB message flows is when the aggregation message flows call multiple validation services which are supposed to return a reply within 2 minutes.

In normal scenarios, 10 to 50 requests are handled per minute.

In high volume load scenarios, more than 100 requests are received per minute which leads to the failure of aggregation message flows. The aggregate replies from validation services will pile up on the internal broker aggregation reply queues making any new requests impossible.

The piled up requests have to be cleared and only then the failed requests can be retried using a retry message flow which reads the aggregation requests from the input backout queue and processes them once again.

This issue requires attention from IBM and can be fixed by having queue depth monitors or triggers on the internal aggregate request and reply queues. These piled up messages should be cleared automatically and after the timeout of 2 minutes which was mentioned in the aggregation message flows.

I would like to bring this issue to the notice of IBM and a PMR has to be raised for this. We faced this issue in production and would like to request a fix for the issue.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Fri Jun 01, 2018 6:21 pm Post subject: Reply with quote

Grand Poobah

Joined: 18 Nov 2003
Posts: 19820
Location: LI,NY

Do your fan-out requests have a time-out set on 2 mins? Do your fan-in responses respect that time-out?
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
eai_developer123
PostPosted: Sun Jun 03, 2018 9:54 pm Post subject: Reply with quote

Newbie

Joined: 21 May 2018
Posts: 6

yes they do. infact the avg reply time is 10 to 15 seconds max.
its only when any one of the validation service goes down that the failure happes.
Back to top
View user's profile Send private message
joebuckeye
PostPosted: Mon Jun 04, 2018 6:19 am Post subject: Re: Managing High load failure scenarios for the Aggregation Reply with quote

Partisan

Joined: 24 Aug 2007
Posts: 339
Location: Columbus, OH

eai_developer123 wrote:
I would like to bring this issue to the notice of IBM and a PMR has to be raised for this. We faced this issue in production and would like to request a fix for the issue.


Well the PMR will bring this issue to IBM's notice.

Let us know their answer and if they release a fix for this issue.

I will say that the numbers you are talking about seem low for an issue to occur though. We have aggregations with higher volumes that work just fine.

We did have an aggregation issue last year where messages were arriving on the fan in queues before the aggregation request node had finished recording the message and therefore the quickly arriving messages were marked as unknown. We had to make the fan out requests a transaction to ensure the aggregation was recorded before the messages could arrive on the fan in queues.
Back to top
View user's profile Send private message
eai_developer123
PostPosted: Mon Jun 04, 2018 8:56 pm Post subject: Reply with quote

Newbie

Joined: 21 May 2018
Posts: 6

The messages were big in the range of 1-10 Mb so 100 msgs were a high volume load and caused a failure.
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexWebSphere Message Broker SupportManaging High load failure scenarios for the Aggregation nod
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.