ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker SupportSYSTEM.BROKER.AGGR Messages getting replayed

Post new topicReply to topic
SYSTEM.BROKER.AGGR Messages getting replayed View previous topic :: View next topic
Author Message
mqlover
PostPosted: Mon Dec 19, 2016 3:01 am Post subject: SYSTEM.BROKER.AGGR Messages getting replayed Reply with quote

Disciple

Joined: 25 Jul 2010
Posts: 176

Hi,

There were few messages in SYSTEM.BROKER.AGGR.TIMEOUT and SYSTEM.BROKER.AGGR.UNKNOWN Q and were getting replayed and the messages were going to Dead letter q.
Later I cleared the SYSTEM.BROKER.AGGR.* queues and 0 queue depth.
I see that still the messages are getting replayed and pushing messages to DLQ and donno from where.
Could somebody make me understand how this is happening even though i don find the messages anywhere in any of the queues.

Thanks in advance
Back to top
View user's profile Send private message
smdavies99
PostPosted: Mon Dec 19, 2016 3:41 am Post subject: Reply with quote

Jedi Council

Joined: 10 Feb 2003
Posts: 6077
Location: Somewhere over the Rainbow this side of Never-never land.

The DLQ will contain the details of what queue does not exist

Then look at the flows that use aggregation and find out what ones use that queue.
Also look at any changes to flows that are done via barfile overrides. PErhaps the override points to a pre-prod queue and you are running in prod.

If this is production then it seems that the movement of the flow + queue definitions +++ from the test environment didn't work
Did this work in Dev? in Test? In Systest? In Pre-prod? What changes were made to enable the flow to move from environment to environment
(As an aside, this is one of the reasons I personally don't like system dependent queue names but that is a discussion for another day and time)
There is a wealth of data on your system that is just waiting for you to look at.
Have you looked at the system logs? the MQ logs?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995

Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions.
Back to top
View user's profile Send private message
mqlover
PostPosted: Mon Dec 19, 2016 7:29 am Post subject: Reply with quote

Disciple

Joined: 25 Jul 2010
Posts: 176

Well, thank you very much.
I would like to reframe my question.
There were some old messages in the system queues which couldnt be delivered to other parties for known reasons.
They were later cleared manually. Despite of that, the messages are getting repayed and I am unable to find the reason. There is nothing much in the system logs as it states that messages are getting replayed from system timeout queues but no messages are there in those system defined queues.
Back to top
View user's profile Send private message
mqlover
PostPosted: Tue Dec 20, 2016 10:46 pm Post subject: Reply with quote

Disciple

Joined: 25 Jul 2010
Posts: 176

HI,

Can anybody help me on this?As to what has to be done?
Back to top
View user's profile Send private message
smdavies99
PostPosted: Wed Dec 21, 2016 5:22 am Post subject: Reply with quote

Jedi Council

Joined: 10 Feb 2003
Posts: 6077
Location: Somewhere over the Rainbow this side of Never-never land.

Have you looked at the logs?
Have you looked at the DLQ reason code (Rfhutil is your friend here)?
Have you raised a PMR?

What flows use Aggregation? Could they possibly be at fault?
Have you discussed this with ther developers?
Why is the destination queue not defined? Was it defined in your test system?

Have you done any further investigation at all?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995

Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions.
Back to top
View user's profile Send private message
mqlover
PostPosted: Wed Dec 21, 2016 6:09 pm Post subject: Reply with quote

Disciple

Joined: 25 Jul 2010
Posts: 176

Thanks will investigate all these.
Back to top
View user's profile Send private message
mqlover
PostPosted: Wed Dec 21, 2016 10:59 pm Post subject: Reply with quote

Disciple

Joined: 25 Jul 2010
Posts: 176

Hi,

I found the following information in MQ logs and broker logs.
Will refreshing the execution group helpful?

MQ Log :
AMQ7487: Application DataFlowEngine was preventing log space from being
released.

EXPLANATION:
A long running transaction was detected, this message is intended to help
identify the application associated with this long running transaction. Message
AMQ7469 or AMQ7485 has been issued indicating if the transaction was rolled
back or rolled forward in the log to allow the log space to be released.
Message AMQ7486 has been issued identifying the transaction context of the
transaction that was rolled back or rolled forwards. The application associated
with this transaction was running with Pid 12345678, Tid 112760, under
application name DataFlowEngine, and under user identifier mqbrokerid. The
following application context may also be useful in identifying the application
causing this behaviour: <No further context>. This message can be correllated
with the previous AMQ7486 message in the queue manager error logs.
ACTION:
Identify the application responsible for the long running unit of work and
ensure this application is creating and completing transactions in a timely
manner. If the application is working as expected it may be appropriate to
increase the size of the queue manager recovery log.
Broker Log :
Dec 22 14:03:20 abcaix1 user:info IIB[18677844]: IBM Integration Bus v10002 (broker1.EG_TEST) [Thread 6526] (Msg 1/3) BIP3901I: Exception condition detected on input node '**.AggregateReply'.
Dec 22 14:03:20 abcaix1 user:err|error IIB[18677844]: IBM Integration Bus v10002 (broker1.EG_TEST) [Thread 6526] (Msg 2/3) BIP4424E: Unable to retrieve control message in AggregateReply node. Queue='SYSTEM.BROKER.AGGR.TIMEOUT'.
Dec 22 14:03:20 abcaix1 user:err|error IIB[18677844]: IBM Integration Bus v10002 (broker1.EG_TEST) [Thread 6526] (Msg 3/3) BIP4630E: An MQGET operation was unsuccessful. Completion Code 2; Reason Code 2003.
Dec 22 14:04:20 abcaix1 user:err|error IIB[18677844]: IBM Integration Bus v10002 (broker1.EG_TEST) [Thread 6526] (Msg 1/1) BIP2648E: Message backed out to a queue; message flow node 'AggregateReply'.
Dec 22 14:04:20 abcaix1 user:err|error last message repeated 6 times
Back to top
View user's profile Send private message
adubya
PostPosted: Thu Dec 22, 2016 12:33 am Post subject: Reply with quote

Partisan

Joined: 25 Aug 2011
Posts: 377
Location: GU12, UK

Was broker still running when you cleared down the SYSTEM queues ?

I'd shutdown broker and probably MQ also, then restart MQ, clear down the aggr SYSTEM queues and then restart broker.

The error message mentions the sizing of the MQ log space, have you reviewed that and sized it appropriately ? That may well be the underlying cause of your issues.
_________________
Independent Middleware Consultant
andy@knownentity.com
Back to top
View user's profile Send private message Send e-mail
mqlover
PostPosted: Thu Dec 22, 2016 1:20 am Post subject: Reply with quote

Disciple

Joined: 25 Jul 2010
Posts: 176

Thanks a ton for your reply.
Yeah the broker was running when cleared the SYSTEM queues.
Will the mqsireload be of any help?
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Thu Dec 22, 2016 2:43 am Post subject: Reply with quote

Grand Poobah

Joined: 18 Nov 2003
Posts: 19923
Location: LI,NY

Code:
Dec 22 14:03:20 abcaix1 user:info IIB[18677844]: IBM Integration Bus v10002 (broker1.EG_TEST) [Thread 6526] (Msg 1/3) BIP3901I: Exception condition detected on input node '**.AggregateReply'.
Dec 22 14:03:20 abcaix1 user:err|error IIB[18677844]: IBM Integration Bus v10002 (broker1.EG_TEST) [Thread 6526] (Msg 2/3) BIP4424E: Unable to retrieve control message in AggregateReply node. Queue='SYSTEM.BROKER.AGGR.TIMEOUT'.
Dec 22 14:03:20 abcaix1 user:err|error IIB[18677844]: IBM Integration Bus v10002 (broker1.EG_TEST) [Thread 6526] (Msg 3/3) BIP4630E: An MQGET operation was unsuccessful. Completion Code 2; Reason Code 2003.
Dec 22 14:04:20 abcaix1 user:err|error IIB[18677844]: IBM Integration Bus v10002 (broker1.EG_TEST) [Thread 6526] (Msg 1/1) BIP2648E: Message backed out to a queue; message flow node 'AggregateReply'.
Dec 22 14:04:20 abcaix1 user:err|error last message repeated 6 times

You need to stop the broker and adequately clear the SYSTEM.BROKER.AGGR* queues.
You problem arises from the fact that you have cleared the control message, but not some of the reply messages, and are not handling the situation adequately on the aggregation reply node's failure terminal.
If you delete / remove a control message you should either handle the resulting error gracefully or also delete the relevant related messages.
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
shashivarungupta
PostPosted: Wed Dec 28, 2016 6:30 pm Post subject: Reply with quote

Grand Master

Joined: 24 Feb 2009
Posts: 1343
Location: Floating in space on a round rock.

@mqlover

Completely agree with fjb_saper suggestions.

Actually, you should not manually touch the messages from SYSTEM.BROKER.AGGR.* Queues. Your Aggregate Application should handle the Messages neatly that are ending up onto Timeout, Unknown Message, Failure, Catch etc. terminals of Aggregate Reply Node.

It gets worst if you have multiple Aggregate Applications in the Broker system and ALL of them would be using same SYSTEM.BROKER.AGGR.* Queues, it gets difficult to identify which one is causing issues. It is bit out of the context here, but it is good to use, following set of queues for each Broker Aggregate Application, for Broker's performance and easy administration perspectives. You may read about following in the info center.
SYSTEM.BROKER.AGGR.QueuePrefix.CONTROL
SYSTEM.BROKER.AGGR.QueuePrefix.REPLY
SYSTEM.BROKER.AGGR.QueuePrefix.REQUEST
SYSTEM.BROKER.AGGR.QueuePrefix.UNKNOWN
SYSTEM.BROKER.AGGR.QueuePrefix.TIMEOUT

Thank You.

_________________
*Life will beat you down, you need to decide to fight back or leave it.
Back to top
View user's profile Send private message Send e-mail
mqlover
PostPosted: Wed Feb 22, 2017 6:37 pm Post subject: Reply with quote

Disciple

Joined: 25 Jul 2010
Posts: 176

Hi,

This issue is still haunting around. As this is prod Qmgr we cannot delete or recreate. I restarted the EG,broker and also the Qmgr but still the issue persists.
What is it I can do, as this also occupies the log space in the file system.
Any suggestion or advise what can be done to overcome this.

Thanks in advance
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Wed Feb 22, 2017 7:52 pm Post subject: Reply with quote

Grand Poobah

Joined: 18 Nov 2003
Posts: 19923
Location: LI,NY

mqlover wrote:
Hi,

This issue is still haunting around. As this is prod Qmgr we cannot delete or recreate. I restarted the EG,broker and also the Qmgr but still the issue persists.
What is it I can do, as this also occupies the log space in the file system.
Any suggestion or advise what can be done to overcome this.

Thanks in advance

Like my esteemed colleague said, you may want to set up the configurable service so that each aggregation has its own queues.
As for the current situation, look at the queues and remove all messages that are truly overdue...
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
mqlover
PostPosted: Fri Feb 24, 2017 1:01 am Post subject: Reply with quote

Disciple

Joined: 25 Jul 2010
Posts: 176

Thanks let me try this approach.

As this is a production environment and all the applications have been deployed, its impossible to change the aggregation queues for each service/application. Hence we have to find some workarounds for the same.

Thanks
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexWebSphere Message Broker SupportSYSTEM.BROKER.AGGR Messages getting replayed
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.