| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| mqsistopmsgflow timeouts and raises coredump and High CPU |
« View previous topic :: View next topic » |
| Author |
Message
|
| dziku007 |
 Posted: Mon Mar 09, 2020 1:14 am Post subject: mqsistopmsgflow timeouts and raises coredump and High CPU |
 |
|
Apprentice
Joined: 26 Jul 2012
Posts: 32
|
Hi,
Script execution of mqsistopmsgflow command ends with timeout and leads to high CPU usage warnings and later coredump file generation and excecution group restart.
This is log from mqsistopmsgflow command execution in script:
| Code: |
2020-03-05T21:06:16.662 Stopping message flow com.ep.ad.log.BLAdapter (inside application BLAdapterApp)
mqsistopmsgflow IB9PRD -e egGo -k BLAdapterApp -m com.ep.ad.log.BLAdapter -w 300
BIP1028I: Stopping message flow 'com.ep.ad.log.BLAdapter' on execution group 'egGo' (Application 'BLAdapterApp', Library '')...
BIP2871I: The request made by user 'mqbrkr' to 'stop' the resource '/BLAdapterApp/com.ep.ad.log.BLAdapter' of type 'MessageFlow' on parent 'egGo' of type 'ExecutionGroup' has the status of 'TIMED-OUT'.
2020-03-05T21:11:20.557 Trouble with stopping message flow com.ep.ad.log.BLAdapter
|
This is log from integration bus sys log:
| Code: |
2020-03-05T21:06:19.520511+01:00 sap-lnx-619 IIB[117593]: IBM Integration Bus v90010 (IB9PRD.egGo) [Thread 117847] (Msg 1/1) BIP2155I: About to 'stop ' the deployed resource 'com.ep.ad.log.BLAdapter' of type '.MSGFLOW'.
2020-03-05T21:06:20.140529+01:00 sap-lnx-619 IIB[117593]: IBM Integration Bus v90010 (IB9PRD.egGo) [Thread 117847] (Msg 1/1) BIP2271I: Deployed resource 'com.ep.ad.log.BLAdapter' (uuid='35681ba1-7001-0000-0080-81aaebfaa2a2',type='.MSGFLOW') successfully stopped.
2020-03-05T21:06:48.556751+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 95.849998
2020-03-05T21:07:18.557561+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 152.119995
2020-03-05T21:07:48.558986+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 185.960007
2020-03-05T21:08:18.560039+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 206.639999
2020-03-05T21:08:41.356286+01:00 sap-lnx-619 crmd[2754]: notice: State transition S_IDLE -> S_POLICY_ENGINE
2020-03-05T21:08:41.370363+01:00 sap-lnx-619 crmd[2754]: notice: Processing graph 1970 (ref=pe_calc-dc-1583438921-2337) derived from /var/lib/pacemaker/pengine/pe-input-446.bz2
2020-03-05T21:08:41.370462+01:00 sap-lnx-619 crmd[2754]: notice: Transition 1970 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-446.bz2): Complete
2020-03-05T21:08:41.370574+01:00 sap-lnx-619 crmd[2754]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE
2020-03-05T21:08:48.560781+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 219.229996
2020-03-05T21:09:18.561625+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 226.899994
2020-03-05T21:09:48.562686+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 231.339996
2020-03-05T21:10:18.563277+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 234.080002
2020-03-05T21:10:48.564335+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 236.229996
2020-03-05T21:11:18.565353+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 237.449997
2020-03-05T21:11:48.566242+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 238.050003
2020-03-05T21:12:18.566876+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 238.229996
2020-03-05T21:12:24.542449+01:00 sap-lnx-619 IIB[112827]: IBM Integration Bus v90010 (IB9PRD) [Thread 112827] (Msg 1/1) BIP2066E: Broker 'IB9PRD' was unable to retrieve an internal configuration response message for execution group 'egGo' within the '360' second configuration timeout.
2020-03-05T21:12:48.567938+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 238.259995
2020-03-05T21:13:18.568949+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 238.330002
2020-03-05T21:13:48.570500+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 238.199997
2020-03-05T21:14:18.571875+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 238.669998
2020-03-05T21:14:48.572480+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 238.410004
2020-03-05T21:15:18.573632+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 238.630005
2020-03-05T21:15:48.574453+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 238.449997
2020-03-05T21:16:18.575159+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 238.550003
2020-03-05T21:16:48.575883+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 238.679993
2020-03-05T21:17:18.577489+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 239.000000
2020-03-05T21:17:48.578770+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 238.600006
2020-03-05T21:18:18.579735+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 238.490005
2020-03-05T21:18:29.580945+01:00 sap-lnx-619 IIB[112827]: IBM Integration Bus v90010 (IB9PRD) [Thread 112827] (Msg 1/1) BIP2066E: Broker 'IB9PRD' was unable to retrieve an internal configuration response message for execution group 'egGo' within the '360' second configuration timeout.
2020-03-05T21:18:33.950385+01:00 sap-lnx-619 IIB[112827]: IBM Integration Bus v90010 (IB9PRD) [Thread 112931] (Msg 1/1) BIP2060W: The broker has detected that the Execution Group egGo, process ID 117593, has shutdown.
2020-03-05T21:18:35.845234+01:00 sap-lnx-619 IIB[110730]: IBM Integration Bus v90010 (IB9PRD.egGo) [Thread 110730] (Msg 1/1) BIP2208I: Execution group (64) started: process '110730'; thread '110730'; additional information: brokerName 'IB9PRD' (operation mode 'advanced'); executionGroupUUID '512f0f1b-4a01-0000-0080-f003255d0b37'; executionGroupLabel 'egGo'; queueManagerName 'PRD_QM'; trusted 'false'; userId 'mqm'; migrationNeeded 'false'; brokerUUID '418779e0-fd9c-11e1-8797-000000000000'; filePath '/opt/ibm/mqsi/9.0.0.10'; workPath '/cluster-data/IB9PRD//mqsi'; ICU Converter Path ''; ordinality '1'.
2020-03-05T21:18:48.580297+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 187.479996
2020-03-05T21:19:04.721443+01:00 sap-lnx-619 IIB[110730]: IBM Integration Bus v90010 (IB9PRD.egGo) [Thread 110844] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '7804' for 'http' connections.
2020-03-05T21:19:13.560421+01:00 sap-lnx-619 IIB[110730]: IBM Integration Bus v90010 (IB9PRD.egGo) [Thread 110844] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '7844' for 'https' connections.
2020-03-05T21:19:18.580824+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 115.510002
2020-03-05T21:19:48.581540+01:00 sap-lnx-619 crmd[2754]: notice: High CPU load detected: 71.709999
2020-03-05T21:20:16.482853+01:00 sap-lnx-619 IIB[110730]: IBM Integration Bus v90010 (IB9PRD.egGo) [Thread 110844] (Msg 1/1) BIP2155I: About to 'stop ' the deployed resource 'com.ep.ad.log.BLAdapter' of type '.MSGFLOW'.
2020-03-05T21:20:16.529075+01:00 sap-lnx-619 IIB[110730]: IBM Integration Bus v90010 (IB9PRD.egGo) [Thread 110844] (Msg 1/1) BIP2271I: Deployed resource 'com.ep.ad.log.BLAdapter' (uuid='35681ba1-7001-0000-0080-81aaebfaa2a2',type='.MSGFLOW') successfully stopped.
|
It generates coredump file:
| Code: |
/var/lib/systemd/coredump/core.DataFlowEngine.487.cfa035ff179e4fce9b1db09bfb4fced2.117593.1583438780000000.xz
|
This issue appeared after BLAdapterApp application upgrade (not major changes - few changes in transformation logic).
Many thanks in advance for any suggestions how to troubleshoot this issue or what can cause such behavior?
Environment:
| Code: |
mqm@sap-lnx-619:~> mqsiservice -v
BIPmsgs en_US
Console CCSID=1208, ICU CCSID=1208
Default codepage=UTF-8, in ascii=UTF-8
JAVA console codepage name=UTF-8
BIP8996I: Version: 90010
BIP8997I: Product: IBM Integration Bus
BIP8998I: CMVC Level: S900-FP10
BIP8999I: Build Type: Production, 64 bit, amd64_linux_2
BIP8974I: Component: DFDL-C, Build ID: 20171026-1523, Version: 1.1.1.0 (1.1.1.0), Platform: linux_x86 64-bit, Type: production
BIP8071I: Successful command completion. |
[/code] |
|
| Back to top |
|
 |
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
