| Author |
Message
|
| mqlover |
 Posted: Mon Dec 19, 2016 3:01 am Post subject: SYSTEM.BROKER.AGGR Messages getting replayed |
 |
|
Disciple
Joined: 25 Jul 2010
Posts: 176
|
Hi,
There were few messages in SYSTEM.BROKER.AGGR.TIMEOUT and SYSTEM.BROKER.AGGR.UNKNOWN Q and were getting replayed and the messages were going to Dead letter q.
Later I cleared the SYSTEM.BROKER.AGGR.* queues and 0 queue depth.
I see that still the messages are getting replayed and pushing messages to DLQ and donno from where.
Could somebody make me understand how this is happening even though i don find the messages anywhere in any of the queues.
Thanks in advance |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Mon Dec 19, 2016 3:41 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
The DLQ will contain the details of what queue does not exist
Then look at the flows that use aggregation and find out what ones use that queue.
Also look at any changes to flows that are done via barfile overrides. PErhaps the override points to a pre-prod queue and you are running in prod.
If this is production then it seems that the movement of the flow + queue definitions +++ from the test environment didn't work
Did this work in Dev? in Test? In Systest? In Pre-prod? What changes were made to enable the flow to move from environment to environment
(As an aside, this is one of the reasons I personally don't like system dependent queue names but that is a discussion for another day and time)
There is a wealth of data on your system that is just waiting for you to look at.
Have you looked at the system logs? the MQ logs?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| mqlover |
| Posted: Mon Dec 19, 2016 7:29 am Post subject: |
|
|
Disciple
Joined: 25 Jul 2010
Posts: 176
|
Well, thank you very much.
I would like to reframe my question.
There were some old messages in the system queues which couldnt be delivered to other parties for known reasons.
They were later cleared manually. Despite of that, the messages are getting repayed and I am unable to find the reason. There is nothing much in the system logs as it states that messages are getting replayed from system timeout queues but no messages are there in those system defined queues. |
|
| Back to top |
|
|
| mqlover |
| Posted: Tue Dec 20, 2016 10:46 pm Post subject: |
|
|
Disciple
Joined: 25 Jul 2010
Posts: 176
|
HI,
Can anybody help me on this?As to what has to be done? |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Dec 21, 2016 5:22 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Have you looked at the logs?
Have you looked at the DLQ reason code (Rfhutil is your friend here)?
Have you raised a PMR?
What flows use Aggregation? Could they possibly be at fault?
Have you discussed this with ther developers?
Why is the destination queue not defined? Was it defined in your test system?
Have you done any further investigation at all?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| mqlover |
| Posted: Wed Dec 21, 2016 6:09 pm Post subject: |
|
|
Disciple
Joined: 25 Jul 2010
Posts: 176
|
| Thanks will investigate all these. |
|
| Back to top |
|
|
| mqlover |
| Posted: Wed Dec 21, 2016 10:59 pm Post subject: |
|
|
Disciple
Joined: 25 Jul 2010
Posts: 176
|
Hi,
I found the following information in MQ logs and broker logs.
Will refreshing the execution group helpful?
MQ Log :
AMQ7487: Application DataFlowEngine was preventing log space from being
released.
EXPLANATION:
A long running transaction was detected, this message is intended to help
identify the application associated with this long running transaction. Message
AMQ7469 or AMQ7485 has been issued indicating if the transaction was rolled
back or rolled forward in the log to allow the log space to be released.
Message AMQ7486 has been issued identifying the transaction context of the
transaction that was rolled back or rolled forwards. The application associated
with this transaction was running with Pid 12345678, Tid 112760, under
application name DataFlowEngine, and under user identifier mqbrokerid. The
following application context may also be useful in identifying the application
causing this behaviour: <No further context>. This message can be correllated
with the previous AMQ7486 message in the queue manager error logs.
ACTION:
Identify the application responsible for the long running unit of work and
ensure this application is creating and completing transactions in a timely
manner. If the application is working as expected it may be appropriate to
increase the size of the queue manager recovery log.
Broker Log :
Dec 22 14:03:20 abcaix1 user:info IIB[18677844]: IBM Integration Bus v10002 (broker1.EG_TEST) [Thread 6526] (Msg 1/3) BIP3901I: Exception condition detected on input node '**.AggregateReply'.
Dec 22 14:03:20 abcaix1 user:err|error IIB[18677844]: IBM Integration Bus v10002 (broker1.EG_TEST) [Thread 6526] (Msg 2/3) BIP4424E: Unable to retrieve control message in AggregateReply node. Queue='SYSTEM.BROKER.AGGR.TIMEOUT'.
Dec 22 14:03:20 abcaix1 user:err|error IIB[18677844]: IBM Integration Bus v10002 (broker1.EG_TEST) [Thread 6526] (Msg 3/3) BIP4630E: An MQGET operation was unsuccessful. Completion Code 2; Reason Code 2003.
Dec 22 14:04:20 abcaix1 user:err|error IIB[18677844]: IBM Integration Bus v10002 (broker1.EG_TEST) [Thread 6526] (Msg 1/1) BIP2648E: Message backed out to a queue; message flow node 'AggregateReply'.
Dec 22 14:04:20 abcaix1 user:err|error last message repeated 6 times |
|
| Back to top |
|
|
| adubya |
| Posted: Thu Dec 22, 2016 12:33 am Post subject: |
|
|
Partisan
Joined: 25 Aug 2011
Posts: 377
Location: GU12, UK
|
Was broker still running when you cleared down the SYSTEM queues ?
I'd shutdown broker and probably MQ also, then restart MQ, clear down the aggr SYSTEM queues and then restart broker.
The error message mentions the sizing of the MQ log space, have you reviewed that and sized it appropriately ? That may well be the underlying cause of your issues.
_________________
Independent Middleware Consultant
andy@knownentity.com |
|
| Back to top |
|
|
| mqlover |
| Posted: Thu Dec 22, 2016 1:20 am Post subject: |
|
|
Disciple
Joined: 25 Jul 2010
Posts: 176
|
Thanks a ton for your reply.
Yeah the broker was running when cleared the SYSTEM queues.
Will the mqsireload be of any help? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Dec 22, 2016 2:43 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Code: |
Dec 22 14:03:20 abcaix1 user:info IIB[18677844]: IBM Integration Bus v10002 (broker1.EG_TEST) [Thread 6526] (Msg 1/3) BIP3901I: Exception condition detected on input node '**.AggregateReply'.
Dec 22 14:03:20 abcaix1 user:err|error IIB[18677844]: IBM Integration Bus v10002 (broker1.EG_TEST) [Thread 6526] (Msg 2/3) BIP4424E: Unable to retrieve control message in AggregateReply node. Queue='SYSTEM.BROKER.AGGR.TIMEOUT'.
Dec 22 14:03:20 abcaix1 user:err|error IIB[18677844]: IBM Integration Bus v10002 (broker1.EG_TEST) [Thread 6526] (Msg 3/3) BIP4630E: An MQGET operation was unsuccessful. Completion Code 2; Reason Code 2003.
Dec 22 14:04:20 abcaix1 user:err|error IIB[18677844]: IBM Integration Bus v10002 (broker1.EG_TEST) [Thread 6526] (Msg 1/1) BIP2648E: Message backed out to a queue; message flow node 'AggregateReply'.
Dec 22 14:04:20 abcaix1 user:err|error last message repeated 6 times |
You need to stop the broker and adequately clear the SYSTEM.BROKER.AGGR* queues.
You problem arises from the fact that you have cleared the control message, but not some of the reply messages, and are not handling the situation adequately on the aggregation reply node's failure terminal.
If you delete / remove a control message you should either handle the resulting error gracefully or also delete the relevant related messages. 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| shashivarungupta |
| Posted: Wed Dec 28, 2016 6:30 pm Post subject: |
|
|
Grand Master
Joined: 24 Feb 2009
Posts: 1343
Location: Floating in space on a round rock.
|
@mqlover
Completely agree with fjb_saper suggestions.
Actually, you should not manually touch the messages from SYSTEM.BROKER.AGGR.* Queues. Your Aggregate Application should handle the Messages neatly that are ending up onto Timeout, Unknown Message, Failure, Catch etc. terminals of Aggregate Reply Node.
It gets worst if you have multiple Aggregate Applications in the Broker system and ALL of them would be using same SYSTEM.BROKER.AGGR.* Queues, it gets difficult to identify which one is causing issues. It is bit out of the context here, but it is good to use, following set of queues for each Broker Aggregate Application, for Broker's performance and easy administration perspectives. You may read about following in the info center.
SYSTEM.BROKER.AGGR.QueuePrefix.CONTROL
SYSTEM.BROKER.AGGR.QueuePrefix.REPLY
SYSTEM.BROKER.AGGR.QueuePrefix.REQUEST
SYSTEM.BROKER.AGGR.QueuePrefix.UNKNOWN
SYSTEM.BROKER.AGGR.QueuePrefix.TIMEOUT
Thank You.
_________________
*Life will beat you down, you need to decide to fight back or leave it. |
|
| Back to top |
|
|
| mqlover |
| Posted: Wed Feb 22, 2017 6:37 pm Post subject: |
|
|
Disciple
Joined: 25 Jul 2010
Posts: 176
|
Hi,
This issue is still haunting around. As this is prod Qmgr we cannot delete or recreate. I restarted the EG,broker and also the Qmgr but still the issue persists.
What is it I can do, as this also occupies the log space in the file system.
Any suggestion or advise what can be done to overcome this.
Thanks in advance |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Feb 22, 2017 7:52 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| mqlover wrote: |
Hi,
This issue is still haunting around. As this is prod Qmgr we cannot delete or recreate. I restarted the EG,broker and also the Qmgr but still the issue persists.
What is it I can do, as this also occupies the log space in the file system.
Any suggestion or advise what can be done to overcome this.
Thanks in advance |
Like my esteemed colleague said, you may want to set up the configurable service so that each aggregation has its own queues.
As for the current situation, look at the queues and remove all messages that are truly overdue...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mqlover |
| Posted: Fri Feb 24, 2017 1:01 am Post subject: |
|
|
Disciple
Joined: 25 Jul 2010
Posts: 176
|
Thanks let me try this approach.
As this is a production environment and all the applications have been deployed, its impossible to change the aggregation queues for each service/application. Hence we have to find some workarounds for the same.
Thanks |
|
| Back to top |
|
|
| MohanIIB |
| Posted: Wed Oct 21, 2020 6:07 pm Post subject: |
|
|
Newbie
Joined: 21 Oct 2020
Posts: 1
|
| mqlover wrote: |
Thanks let me try this approach.
As this is a production environment and all the applications have been deployed, its impossible to change the aggregation queues for each service/application. Hence we have to find some workarounds for the same.
Thanks |
Has this approach to change the aggregation queues for each service/application is working ? I too facing the similar errors in MQLog & Broker Log. |
|
| Back to top |
|
|
|
|
