| Author |
Message
|
| pauls |
 Posted: Thu Oct 22, 2009 1:48 pm Post subject: BIP2080E: Execution group xxxxxx failed to return |
 |
|
Novice
Joined: 20 Sep 2009
Posts: 11
|
Hi, We are upgrading our dual site broker runtimes to WMQ 7.0.0.2 and WMB 6.1.0.4 on virtualised RHEL 5.3 servers. During the testing phase we are now experincing BIP2080E, BIP2804E, BIP2033I and BIP2066E messages.
The execution group vaires and the problem is intermittent. Sometimes all executiongroups start, other times between 1 to 3 execution groups fail to start.
It only happens on one of our sites and we can move a working WMB VM from the good site to the bad site and reproduce the problem so we think its data centre related.
We are about to move to the 7.0.1.0 and 6.1.0.5 fixpacs and raise a PMR.
User trace on an unresponsive execution group doesnt get processed as the messages just stack on MQ.
Has anyone else seen this type of behaviour or got any suggestions?
regards
Paul |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Thu Oct 22, 2009 4:34 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Do you have good connectivity to the brokerDB ? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| pauls |
| Posted: Thu Oct 22, 2009 4:41 pm Post subject: |
|
|
Novice
Joined: 20 Sep 2009
Posts: 11
|
The brokerDB is local on both sites and we haven't seen database connection issues. We are using V9.1 FP7.
I will get the dba to verify. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Oct 22, 2009 4:49 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| pauls wrote: |
The brokerDB is local on both sites and we haven't seen database connection issues. We are using V9.1 FP7.
I will get the dba to verify. |
Check as well the broker's startup log. You might get a clue as to the reason why there is a problem starting the eg..
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| pauls |
| Posted: Thu Oct 22, 2009 5:34 pm Post subject: |
|
|
Novice
Joined: 20 Sep 2009
Posts: 11
|
I had a look at the db2diag.log and there isnt anything unusual in there and no db2 messges on syslog. I have checked stderr and stdout for the eg's and they generally include a "ftok OS JNI: No such file or directory" message on each startup.
When you say brokers startup log, are you meaning syslog on our Linux platform. The syslog entries for an intance of the error dont show any messages before the BIP2080E that indicates its hung on startup and will never recover.
| Code: |
Oct 22 09:17:42 srv-qs-mqd2 WebSphere Broker v6104[9324]: (DVQSBR01.SIX.001)[26]BIP2152I: Configuration message received from broker. : DVQSBR01.2f34c175-2401-0000-0080-ee450250b984: /build/S610_P/src/DataFlowEngine/ImbConfigurationNode.cpp: 310: ImbConfigurationNode::evaluate: :
Oct 22 09:17:42 srv-qs-mqd2 WebSphere Broker v6104[9324]: (DVQSBR01.SIX.001)[26]BIP2153I: About to 'start' an execution group. : DVQSBR01.2f34c175-2401-0000-0080-ee450250b984: /build/S610_P/src/DataFlowEngine/ImbConfigurationNode.cpp: 313: ImbConfigurationNode::evaluate: :
Oct 22 09:17:42 srv-qs-mqd2 WebSphere Broker v6104[9324]: (DVQSBR01.SIX.001)[26]BIP2154I: Execution group finished with Configuration message. : DVQSBR01.2f34c175-2401-0000-0080-ee450250b984: /build/S610_P/src/DataFlowEngine/ImbConfigurationNode.cpp: 493: ImbConfigurationNode::evaluate: :
Oct 22 09:17:43 srv-qs-mqd2 WebSphere Broker v6104[9326]: (DVQSBR01.MER.001)[9]BIP2152I: Configuration message received from broker. : DVQSBR01.a991bf75-2401-0000-0080-ee450250b984: /build/S610_P/src/DataFlowEngine/ImbConfigurationNode.cpp: 310: ImbConfigurationNode::evaluate: :
Oct 22 09:17:43 srv-qs-mqd2 WebSphere Broker v6104[9326]: (DVQSBR01.MER.001)[9]BIP2153I: About to 'start' an execution group. : DVQSBR01.a991bf75-2401-0000-0080-ee450250b984: /build/S610_P/src/DataFlowEngine/ImbConfigurationNode.cpp: 313: ImbConfigurationNode::evaluate: :
Oct 22 09:17:43 srv-qs-mqd2 WebSphere Broker v6104[9326]: (DVQSBR01.MER.001)[9]BIP2154I: Execution group finished with Configuration message. : DVQSBR01.a991bf75-2401-0000-0080-ee450250b984: /build/S610_P/src/DataFlowEngine/ImbConfigurationNode.cpp: 493: ImbConfigurationNode::evaluate: :
Oct 22 09:18:33 srv-qs-mqd2 WebSphere Broker v6104[8173]: (DVQSBR01)[1]BIP2080E: Execution group ITS.002 failed to return an internal configuration Start message response within a 60 second time period. : DVQSBR01.agent: /build/S610_P/src/AdminAgent/ImbAdminAgent.cpp: 8263: ImbAdminAgent::getXMLResponse: :
Oct 22 09:19:19 srv-qs-mqd2 WebSphere Broker v6104[9324]: (DVQSBR01.SIX.001)[26]BIP2152I: Configuration message received from broker. : DVQSBR01.2f34c175-2401-0000-0080-ee450250b984: /build/S610_P/src/DataFlowEngine/ImbConfigurationNode.cpp: 323: ImbConfigurationNode::evaluate: :
Oct 22 09:19:19 srv-qs-mqd2 WebSphere Broker v6104[9324]: (DVQSBR01.SIX.001)[26]BIP2153I: About to 'stop' an execution group. : DVQSBR01.2f34c175-2401-0000-0080-ee450250b984: /build/S610_P/src/DataFlowEngine/ImbConfigurationNode.cpp: 326: ImbConfigurationNode::evaluate: :
Oct 22 09:19:19 srv-qs-mqd2 WebSphere Broker v6104[9332]: (DVQSBR01.PGWY.001)[7]BIP2152I: Configuration message received from broker. : DVQSBR01.30a9c175-2401-0000-0080-ee450250b984: /build/S610_P/src/DataFlowEngine/ImbConfigurationNode.cpp: 323: ImbConfigurationNode::evaluate: :
Oct 22 09:19:19 srv-qs-mqd2 WebSphere Broker v6104[9332]: (DVQSBR01.PGWY.001)[7]BIP2153I: About to 'stop' an execution group. : DVQSBR01.30a9c175-2401-0000-0080-ee450250b984: /build/S610_P/src/DataFlowEngine/ImbConfigurationNode.cpp: 326: ImbConfigurationNode::evaluate: :
Oct 22 09:19:19 srv-qs-mqd2 WebSphere Broker v6104[9326]: (DVQSBR01.MER.001)[9]BIP2152I: Configuration message received from broker. : DVQSBR01.a991bf75-2401-0000-0080-ee450250b984: /build/S610_P/src/DataFlowEngine/ImbConfigurationNode.cpp: 323: ImbConfigurationNode::evaluate: :
Oct 22 09:19:19 srv-qs-mqd2 WebSphere Broker v6104[9326]: (DVQSBR01.MER.001)[9]BIP2153I: About to 'stop' an execution group. : DVQSBR01.a991bf75-2401-0000-0080-ee450250b984: /build/S610_P/src/DataFlowEngine/ImbConfigurationNode.cpp: 326: ImbConfigurationNode::evaluate: :
Oct 22 09:19:19 srv-qs-mqd2 WebSphere Broker v6104[9223]: (DVQSBR01.ITS.001)[18]BIP2152I: Configuration message received from broker. : DVQSBR01.abf0bd75-2401-0000-0080-ee450250b984: /build/S610_P/src/DataFlowEngine/ImbConfigurationNode.cpp: 323: ImbConfigurationNode::evaluate: :
Oct 22 09:19:19 srv-qs-mqd2 WebSphere Broker v6104[9223]: (DVQSBR01.ITS.001)[18]BIP2153I: About to 'stop' an execution group. : DVQSBR01.abf0bd75-2401-0000-0080-ee450250b984: /build/S610_P/src/DataFlowEngine/ImbConfigurationNode.cpp: 326: ImbConfigurationNode::evaluate: :
Oct 22 09:19:28 srv-qs-mqd2 WebSphere Broker v6104[9223]: (DVQSBR01.ITS.001)[1]BIP2204I: Execution group using process '9223' thread '1' stopped. : DVQSBR01.abf0bd75-2401-0000-0080-ee450250b984: /build/S610_P/src/DataFlowEngine/ImbMain.cpp: 1130: ImbMain::start: :
Oct 22 09:19:29 srv-qs-mqd2 WebSphere Broker v6104[9332]: (DVQSBR01.PGWY.001)[1]BIP2204I: Execution group using process '9332' thread '1' stopped. : DVQSBR01.30a9c175-2401-0000-0080-ee450250b984: /build/S610_P/src/DataFlowEngine/ImbMain.cpp: 1130: ImbMain::start: :
Oct 22 09:19:29 srv-qs-mqd2 WebSphere Broker v6104[9326]: (DVQSBR01.MER.001)[1]BIP2204I: Execution group using process '9326' thread '1' stopped. : DVQSBR01.a991bf75-2401-0000-0080-ee450250b984: /build/S610_P/src/DataFlowEngine/ImbMain.cpp: 1130: ImbMain::start: :
Oct 22 09:19:29 srv-qs-mqd2 WebSphere Broker v6104[9324]: (DVQSBR01.SIX.001)[1]BIP2204I: Execution group using process '9324' thread '1' stopped. : DVQSBR01.2f34c175-2401-0000-0080-ee450250b984: /build/S610_P/src/DataFlowEngine/ImbMain.cpp: 1130: ImbMain::start: :
Oct 22 09:20:21 srv-qs-mqd2 WebSphere Broker v6104[8173]: (DVQSBR01)[2]BIP2804E: The broker has detected that Execution Group ITS.002, process ID 9325, has not shutdown. : DVQSBR01.agent: /build/S610_P/src/AdminAgent/ImbAdminStore.cpp: 662: ImbAdminStore::FindAndReportAllLongRunningProcesses: :
Oct 22 09:20:31 srv-qs-mqd2 WebSphere Broker v6104[11400]: (DVQSBR01)[1]BIP2033I: The broker was forced to stop an execution group process. The process ID was 9325. : DVQSBR01.mqsistop: /build/S610_P/src/CommandLine/ImbCmdmqsistop.cpp: 307: ImbCmdmqsistop::processCommand: :
Oct 22 09:20:38 srv-qs-mqd2 WebSphere Broker v6104[8167]: (DVQSBR01)[1]BIP2002W: The WebSphere Message Brokers service has stopped. : DVQSBR01.service: /build/S610_P/src/AdminAgent/ControlProcess/Linux/ImbControlService.cpp: 528: ImbControlService::CleanUp: : |
|
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Oct 22, 2009 6:03 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Open a PMR
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| pauls |
| Posted: Thu Oct 22, 2009 6:13 pm Post subject: |
|
|
Novice
Joined: 20 Sep 2009
Posts: 11
|
yes. just finished collecting all the logs and trying to describe the problem. Now all I need to do is find someone here who can raise a PMR  |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Oct 23, 2009 5:07 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
If the problem can not be replicated outside of VM, then the PMR may not get very far.
It is useful information that a VM can be moved to the bad site and start having the issue. It would be slightly more useful to know if moving a VM to the GOOD side *fixes* the issue.
Then you can run startup traces on the same image - once when it's good and once when it's bad. That will go a *very* long way towards identifying the problem area. |
|
| Back to top |
|
|
| pauls |
| Posted: Fri Oct 23, 2009 11:36 pm Post subject: |
|
|
Novice
Joined: 20 Sep 2009
Posts: 11
|
We did move a failing VM back to the good site and it didnt fail. I dont think its VM related. We tried increasing the jvm minimum and max and didnt get a problem in 4 hours of shut/start cycles wvery 10 minutes.
Will let you know if the JVM changes ran all weekend without a problem. Still no idea why one site has problems and the other doesn't. |
|
| Back to top |
|
|
| pauls |
| Posted: Mon Oct 26, 2009 4:52 pm Post subject: |
|
|
Novice
Joined: 20 Sep 2009
Posts: 11
|
Changing the default V6.1 JVM settings has made the eg startup more consistent however we still got 2 failures today in QA with our automated startup testing.
Reviewing the V6 library indicates "From WebSphere® Business Integration Message Broker Version 5.0 onwards, the JVM is created with a minimum of 128 MB of space allocated and reserved for its use." has been replaced in V6.1 with "From WebSphere® Message Broker Version 6.1 onwards, the JVM is created with a minimum of 32 MB of space, and a maximum of 256 MB, allocated and reserved for its use."
We are using HTTP nodes and it seems the new minimum default isn't sufficient for our environment. We have had one error at the good site and hundreds at the problem site. In theory both MB instances are the same.
Any suggestions on tuning JVM settings in V6.1 broker? The doco is a bit sparse. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Oct 27, 2009 2:25 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Again, if you're moving the same VM between sites and it works on one site and fails at the other, this is very telling.
Again, take a startup trace of the *same* broker at the bad site and the good site, and review the differences.
http://www-01.ibm.com/support/docview.wss?rs=849&uid=swg21177044
Even if you end up opening a PMR, you're going to have to do this. |
|
| Back to top |
|
|
|
|
