ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker Supportmqsistopmsgflow timeouts and raises coredump and High CPU

Post new topicReply to topic
mqsistopmsgflow timeouts and raises coredump and High CPU View previous topic :: View next topic
Author Message
dziku007
PostPosted: Mon Mar 09, 2020 1:14 am Post subject: mqsistopmsgflow timeouts and raises coredump and High CPU Reply with quote

Apprentice

Joined: 26 Jul 2012
Posts: 31

Hi,

Script execution of mqsistopmsgflow command ends with timeout and leads to high CPU usage warnings and later coredump file generation and excecution group restart.

This is log from mqsistopmsgflow command execution in script:
Code:

2020-03-05T21:06:16.662 Stopping message flow com.ep.ad.log.BLAdapter (inside application BLAdapterApp)
mqsistopmsgflow IB9PRD -e egGo -k BLAdapterApp -m com.ep.ad.log.BLAdapter -w 300
BIP1028I: Stopping message flow 'com.ep.ad.log.BLAdapter' on execution group 'egGo' (Application 'BLAdapterApp', Library '')...
BIP2871I: The request made by user 'mqbrkr' to 'stop' the resource '/BLAdapterApp/com.ep.ad.log.BLAdapter' of type 'MessageFlow' on parent 'egGo' of type 'ExecutionGroup' has the status of 'TIMED-OUT'.
2020-03-05T21:11:20.557 Trouble with stopping message flow com.ep.ad.log.BLAdapter


This is log from integration bus sys log:

Code:

2020-03-05T21:06:19.520511+01:00 sap-lnx-619 IIB[117593]: IBM Integration Bus v90010 (IB9PRD.egGo) [Thread 117847] (Msg 1/1) BIP2155I: About to 'stop ' the deployed resource 'com.ep.ad.log.BLAdapter' of type '.MSGFLOW'.
2020-03-05T21:06:20.140529+01:00 sap-lnx-619 IIB[117593]: IBM Integration Bus v90010 (IB9PRD.egGo) [Thread 117847] (Msg 1/1) BIP2271I: Deployed resource 'com.ep.ad.log.BLAdapter' (uuid='35681ba1-7001-0000-0080-81aaebfaa2a2',type='.MSGFLOW') successfully stopped.
2020-03-05T21:06:48.556751+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 95.849998
2020-03-05T21:07:18.557561+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 152.119995
2020-03-05T21:07:48.558986+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 185.960007
2020-03-05T21:08:18.560039+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 206.639999
2020-03-05T21:08:41.356286+01:00 sap-lnx-619 crmd[2754]:   notice: State transition S_IDLE -> S_POLICY_ENGINE
2020-03-05T21:08:41.370363+01:00 sap-lnx-619 crmd[2754]:   notice: Processing graph 1970 (ref=pe_calc-dc-1583438921-2337) derived from /var/lib/pacemaker/pengine/pe-input-446.bz2
2020-03-05T21:08:41.370462+01:00 sap-lnx-619 crmd[2754]:   notice: Transition 1970 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-446.bz2): Complete
2020-03-05T21:08:41.370574+01:00 sap-lnx-619 crmd[2754]:   notice: State transition S_TRANSITION_ENGINE -> S_IDLE
2020-03-05T21:08:48.560781+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 219.229996
2020-03-05T21:09:18.561625+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 226.899994
2020-03-05T21:09:48.562686+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 231.339996
2020-03-05T21:10:18.563277+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 234.080002
2020-03-05T21:10:48.564335+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 236.229996
2020-03-05T21:11:18.565353+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 237.449997
2020-03-05T21:11:48.566242+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 238.050003
2020-03-05T21:12:18.566876+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 238.229996
2020-03-05T21:12:24.542449+01:00 sap-lnx-619 IIB[112827]: IBM Integration Bus v90010 (IB9PRD) [Thread 112827] (Msg 1/1) BIP2066E: Broker 'IB9PRD' was unable to retrieve an internal configuration response message for execution group 'egGo' within the '360' second configuration timeout.
2020-03-05T21:12:48.567938+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 238.259995
2020-03-05T21:13:18.568949+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 238.330002
2020-03-05T21:13:48.570500+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 238.199997
2020-03-05T21:14:18.571875+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 238.669998
2020-03-05T21:14:48.572480+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 238.410004
2020-03-05T21:15:18.573632+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 238.630005
2020-03-05T21:15:48.574453+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 238.449997
2020-03-05T21:16:18.575159+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 238.550003
2020-03-05T21:16:48.575883+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 238.679993
2020-03-05T21:17:18.577489+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 239.000000
2020-03-05T21:17:48.578770+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 238.600006
2020-03-05T21:18:18.579735+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 238.490005
2020-03-05T21:18:29.580945+01:00 sap-lnx-619 IIB[112827]: IBM Integration Bus v90010 (IB9PRD) [Thread 112827] (Msg 1/1) BIP2066E: Broker 'IB9PRD' was unable to retrieve an internal configuration response message for execution group 'egGo' within the '360' second configuration timeout.
2020-03-05T21:18:33.950385+01:00 sap-lnx-619 IIB[112827]: IBM Integration Bus v90010 (IB9PRD) [Thread 112931] (Msg 1/1) BIP2060W: The broker has detected that the Execution Group egGo, process ID 117593, has shutdown.
2020-03-05T21:18:35.845234+01:00 sap-lnx-619 IIB[110730]: IBM Integration Bus v90010 (IB9PRD.egGo) [Thread 110730] (Msg 1/1) BIP2208I: Execution group (64) started: process '110730'; thread '110730'; additional information: brokerName 'IB9PRD' (operation mode 'advanced'); executionGroupUUID '512f0f1b-4a01-0000-0080-f003255d0b37'; executionGroupLabel 'egGo'; queueManagerName 'PRD_QM'; trusted 'false'; userId 'mqm'; migrationNeeded 'false'; brokerUUID '418779e0-fd9c-11e1-8797-000000000000'; filePath '/opt/ibm/mqsi/9.0.0.10'; workPath '/cluster-data/IB9PRD//mqsi'; ICU Converter Path ''; ordinality '1'.
2020-03-05T21:18:48.580297+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 187.479996
2020-03-05T21:19:04.721443+01:00 sap-lnx-619 IIB[110730]: IBM Integration Bus v90010 (IB9PRD.egGo) [Thread 110844] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '7804' for 'http' connections.
2020-03-05T21:19:13.560421+01:00 sap-lnx-619 IIB[110730]: IBM Integration Bus v90010 (IB9PRD.egGo) [Thread 110844] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '7844' for 'https' connections.
2020-03-05T21:19:18.580824+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 115.510002
2020-03-05T21:19:48.581540+01:00 sap-lnx-619 crmd[2754]:   notice: High CPU load detected: 71.709999
2020-03-05T21:20:16.482853+01:00 sap-lnx-619 IIB[110730]: IBM Integration Bus v90010 (IB9PRD.egGo) [Thread 110844] (Msg 1/1) BIP2155I: About to 'stop ' the deployed resource 'com.ep.ad.log.BLAdapter' of type '.MSGFLOW'.
2020-03-05T21:20:16.529075+01:00 sap-lnx-619 IIB[110730]: IBM Integration Bus v90010 (IB9PRD.egGo) [Thread 110844] (Msg 1/1) BIP2271I: Deployed resource 'com.ep.ad.log.BLAdapter' (uuid='35681ba1-7001-0000-0080-81aaebfaa2a2',type='.MSGFLOW') successfully stopped.


It generates coredump file:
Code:

/var/lib/systemd/coredump/core.DataFlowEngine.487.cfa035ff179e4fce9b1db09bfb4fced2.117593.1583438780000000.xz

This issue appeared after BLAdapterApp application upgrade (not major changes - few changes in transformation logic).

Many thanks in advance for any suggestions how to troubleshoot this issue or what can cause such behavior?

Environment:
Code:
mqm@sap-lnx-619:~> mqsiservice -v
BIPmsgs  en_US
  Console CCSID=1208, ICU CCSID=1208
  Default codepage=UTF-8, in ascii=UTF-8
  JAVA console codepage name=UTF-8

BIP8996I: Version:    90010
BIP8997I: Product:    IBM Integration Bus
BIP8998I: CMVC Level: S900-FP10
BIP8999I: Build Type: Production, 64 bit, amd64_linux_2

BIP8974I: Component: DFDL-C, Build ID: 20171026-1523, Version: 1.1.1.0 (1.1.1.0), Platform: linux_x86 64-bit, Type: production

BIP8071I: Successful command completion.
[/code]
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexWebSphere Message Broker Supportmqsistopmsgflow timeouts and raises coredump and High CPU
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.