| Author |
Message
|
| sjclark1 |
 Posted: Mon Mar 07, 2005 9:43 am Post subject: New broker - Execution group falling over |
 |
|
Apprentice
Joined: 06 Jan 2003
Posts: 35
Location: Scotland
|
Hi all.
I have created a new broker on Solaris, started it. I created a reference to the broker in the Config Mgr broker domain. I was then automatically prompted to do a Delta deploy, which I did. This causes the execution group to start and fall over continually, and the config mgr reports deployment errors. More detail below.
Environment
========
ConfigMgr & Toolkit
----------------------
Windows 2000
Message Broker 5 FP4
DB2 8.1 FP7a
MQ 5.3 CSD8
1.4.1
Broker
--------
Solaris 8
Message Broker 5 FP4
Oracle 9.2.0.4
MQ 5.3 CSD8
JRE 1.4.2
When I create the reference to the broker in the config mgr, it prompts me to deploy, I then see errors in /var/adm/messages:
Every 10 seconds the following error appears twice at the same time:
Mar 7 16:54:14 sole MQSIv500[9166]: [ID 702911 user.error] (my_broker.$SYS
_mqsi)[1]BIP2203E: An execution group has encountered a problem whilst starting. : my_broker.82A6E590-12B3-11D3-81A200203506A68E: /build/S500_P/src/DataFlowEngine/ImbMain.cpp: 570: main: :
Then every few minutes I see this message (again twice at the same time):
Mar 7 16:57:32 sole MQSIv500[9160]: [ID 702911 user.error] (my_broker)[1]BIP2066E: Broker my_broker (UUID 2b25ef7d-0201-0000-0080-99c143c796d2) was unable to retrieve an internal configuration response message for execution group '$SYS_mqsi' within the 160 second Configuration Timeout. : my_broker.agent: /build/S500_P/src/AdminAgent/ImbAdminAgent.cpp: 2400: ImbAdminAgent::receiveEGResponse: :
The Config Mgr then reports the following two errors:
BIP2087E: Broker my_broker was unable to process the internal configuration message.
The entire internal configuration message failed to be processed successfully.
The internal configuration message failed to be processed, use the messages following this message to determine the reasons for the failure.
BIP2066E: Broker my_broker (UUID 2b25ef7d-0201-0000-0080-99c143c796d2) was unable to retrieve an internal configuration response message for execution group '$SYS_mqsi' within the 160 second Configuration Timeout.
The execution group did not respond within the Configuration Timeout period. A negative response is returned to the Configuration Manager for this execution group. The Configuration Timeout is the maximum length of time that an execution group is allowed to apply a deployed configuration change. By default the Configuration Timeout period is 300 seconds. You can increase (or decrease) the timeout by using the mqsichangebroker option, ConfigurationTimeout. This will not resolve any underlying problem with the deployed message, but can be used to reduce the response turnaround time or increase it to allow for large and complex deployments.
Investigate why the execution group was unable to respond before being timed out. Use the system log messages to determine if there is a problem with the execution group. You should check that your system is not running short of resources, for example, you may need to increase the WebSphere MQ log size. Reducing the complexity of the deploy by reducing the number of execution groups may also help to alleviate this problem. Correct the problem and redeploy the broker's configuration from the Message Brokers Toolkit. If there are no other failure diagnostics then you should consider increasing the Configuration Timeout, increase this value in units of 300 until this message no longer occurs. Contact your IBM support center if you are unable to resolve the problem.
and then, about 3 minutes later, I get this message in the config mgr:
BIP1536S: Configuration Manager unable to register for internal subscriptions with broker my_broker.
Although the preceding deploy operation was successful, a subsequent attempt by the Configuration Manager to register for internal subscriptions with broker my_broker was not successful. This means that the Message Brokers Toolkit may not reliably show the correct status of the resources assigned to broker my_broker. The Configuration Manager also writes this message to the system event log, accompanied by other messages giving details of the problem.
Look at the messages for this problem in the system event log. Determine and correct the cause of the problem before attempting to deploy to this broker again. If this problem persists, contact your IBM support center.
I know that the config mgr is somehow talking to the broker but something is causing it to develop a problem. Does anyone know what is going on under the covers here that might suggest what the problem could be? The error seems quite severe.
I have already uninstalled MQ & the Message Broker on Solaris and reinstalled them but it didn't make any difference.
Any help would be most appreciated.
Steven.
Last edited by sjclark1 on Mon Mar 07, 2005 10:02 am; edited 1 time in total |
|
| Back to top |
|
 |
| sjclark1 |
| Posted: Mon Mar 07, 2005 9:58 am Post subject: |
|
|
Apprentice
Joined: 06 Jan 2003
Posts: 35
Location: Scotland
|
| Sorry to followup my own post but one other thing I should mention is that after doing a Delta deploy the config mgr's Alerts window says "Broker is not running" - which I thought was strange because it is running, although obviously nothing has been deployed to it yet. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Mon Mar 07, 2005 10:19 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| sjclark1 wrote: |
| Sorry to followup my own post but one other thing I should mention is that after doing a Delta deploy the config mgr's Alerts window says "Broker is not running" - which I thought was strange because it is running, although obviously nothing has been deployed to it yet. |
This at least is normal. The broker doesn't "run", and neither do any execution groups, until at least one thing is deployed to them.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| bim_newbie |
| Posted: Mon Mar 07, 2005 1:20 pm Post subject: |
|
|
Newbie
Joined: 16 Feb 2005
Posts: 8
|
Have you verified the communication between the ConfigMgr & Broker? Please verify that
1) All the system queues were created
2) Channels are running properly
I cant think of any other thing. |
|
| Back to top |
|
|
| sjclark1 |
| Posted: Mon Mar 07, 2005 1:36 pm Post subject: |
|
|
Apprentice
Joined: 06 Jan 2003
Posts: 35
Location: Scotland
|
Yes all system queues were created and channels are up and running. I know that the config mgr can send messages to the broker because the the DataFlowEngine process is created. I think if I know what is causing this DataFlowEngine to then fall over repeatedly then that will be the crux of the problem.
I am assuming the only reason the config mgr reports errors is because the newly created execution group falls over before it can return a success message back to the config mgr... |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Mon Mar 07, 2005 2:28 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Can the broker send messages to the ConfigMgr?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| samgunddi |
| Posted: Tue Mar 08, 2005 6:17 am Post subject: |
|
|
Acolyte
Joined: 17 May 2004
Posts: 54
|
This is definitely an issue with your environment setting.
Check if the software is installed correctly (correct version of messaged broker is installed)?
pkginfo -l >xxxxx
Check the Environment variables?
Check if your have configure your .odbc.ini ?
Have you issued mqsi_setupdatabase command?
Last edited by samgunddi on Tue Mar 08, 2005 8:17 am; edited 1 time in total |
|
| Back to top |
|
|
| sjclark1 |
| Posted: Tue Mar 08, 2005 7:00 am Post subject: |
|
|
Apprentice
Joined: 06 Jan 2003
Posts: 35
Location: Scotland
|
Thanks for all your suggestions so far. The channels are all up and running and I have meticulously checked my installation and environment variables so I don't think they are the problem.
| samgunddi wrote: |
| Have you issued mqsi_setupdatabase command? |
OK I think I know what the problem is. I did run the mqsi_setupdatabase command and it created links as follows:
/var/wmqi/lib/libclntsh.so -> /opt/oracle/product/9.2.0/lib32/libclntsh.so
/var/wmqi/lib/libwtc9.so -> /opt/oracle/product/9.2.0/lib32/libwtc9.so
except the oracle 9.2.0.4 installation I was using did not have a "lib32" directory, instead it had a "lib" directory. So I re-did the links manually and thought that was sufficient. However I have just read the following on IBM's website:
| IBM wrote: |
| Oracle 9i installs both 64-bit and 32-bit libraries, with the 64-bit libraries as the default. For example, the 64-bit libraries are installed in the ${ORACLE_HOME}/lib directory and the 32-bit libraries are installed in the /lib32 directory. The customer must configure the Broker environment to use the 32-bit libraries (using mqsi_setupdatabase). |
I am not an oracle expert so I don't know why the 32-bit libraries are missing, but for now oracle is the chief suspect and I will pursue a resolution along those lines.
I assume as part of a deploy the broker tries to update the broker database, so this could be a valid explanation for the problem?
Thanks again. |
|
| Back to top |
|
|
| CodeCraft |
| Posted: Tue Mar 08, 2005 8:12 am Post subject: Some thoughts |
|
|
Disciple
Joined: 05 Sep 2001
Posts: 195
|
I may be wrong, but did "mqsicreatebroker" work successfully. I thought at that point it also tried to access the database to create tables for the broker; is there broker related tables in the database?
If that worked, I'm wondering why there would be an issue with the deploy, although if there is perceived to be an issue with the libraries that question also needs to be answered.
Is there any useful messages in the syslog on the target system over and above what is being returned to the configuration manager? |
|
| Back to top |
|
|
| CodeCraft |
| Posted: Tue Mar 08, 2005 8:19 am Post subject: "lib32" |
|
|
Disciple
Joined: 05 Sep 2001
Posts: 195
|
The absence of lib32 may mean that the Oracle 32 bit client is not installed on this system. You should check with your DBA or the installer for Oracle and see is there optional components which have not actually been installed.
Having done that, I'd be inclined to delete and recreate the broker to ensure that everything is genuinely up to scratch.
-----oOo-----
Support for 64-bit databases (DB2 and Oracle)
The Broker is a 32-bit application and interacts with 32-bit database libraries. In these Broker environments, our statement of support for databases refers to the database client.
The Broker supports connecting to a 32-bit database client which connects to a 64-bit server. The Broker does not support connecting to a 64-bit database client. This restriction is imposed by both the Broker and the DataDirect drivers.
Oracle 9i installs both 64-bit and 32-bit libraries, with the 64-bit libraries as the default. For example, the 64-bit libraries are installed in the ${ORACLE_HOME}/lib directory and the 32-bit libraries are installed in the /lib32 directory. The customer must configure the Broker environment to use the 32-bit libraries (using mqsi_setupdatabase). |
|
| Back to top |
|
|
| sjclark1 |
| Posted: Tue Mar 08, 2005 8:46 am Post subject: Re: Some thoughts |
|
|
Apprentice
Joined: 06 Jan 2003
Posts: 35
Location: Scotland
|
| CodeCraft wrote: |
I may be wrong, but did "mqsicreatebroker" work successfully. I thought at that point it also tried to access the database to create tables for the broker; is there broker related tables in the database?
If that worked, I'm wondering why there would be an issue with the deploy, although if there is perceived to be an issue with the libraries that question also needs to be answered. |
I see your point - mqsicreatebroker worked fine - the standard 29 tables are created in the database. However I don't know if mqsicreatebroker uses the two 64-bit libraries in question.
After mqsicreatebroker I don't see any rows in any of the tables I checked (think this is normal?). But after a deploy I would expect to see some of those tables populated with data. So maybe it's only the insertion of rows that causes a problem...
| CodeCraft wrote: |
| Is there any useful messages in the syslog on the target system over and above what is being returned to the configuration manager? |
All the log messages I see are see are listed at the top of the thread, including two in the target system. |
|
| Back to top |
|
|
| samgunddi |
| Posted: Tue Mar 08, 2005 8:54 am Post subject: |
|
|
Acolyte
Joined: 17 May 2004
Posts: 54
|
| Again Check if the Messages Broker ( not event broker) is installed correctly? |
|
| Back to top |
|
|
| sjclark1 |
| Posted: Tue Mar 08, 2005 8:59 am Post subject: |
|
|
Apprentice
Joined: 06 Jan 2003
Posts: 35
Location: Scotland
|
| samgunddi wrote: |
| Again Check if the Messages Broker ( not event broker) is installed correctly? |
Does this look ok to you?
Sole:->pkginfo -l mqsiIb
PKGINST: mqsiIb
NAME: IBM WebSphere Business Integration Message Broker
CATEGORY: application
ARCH: sparc
VERSION: 5.0.1.4.DSP=
BASEDIR: /opt/mqsi/
VENDOR: IBM
DESC: IBM WebSphere Business Integration Message Broker for Multiplatfo
rms
STATUS: completely installed |
|
| Back to top |
|
|
| sjclark1 |
| Posted: Tue Mar 08, 2005 9:02 am Post subject: Re: "lib32" |
|
|
Apprentice
Joined: 06 Jan 2003
Posts: 35
Location: Scotland
|
| CodeCraft wrote: |
The absence of lib32 may mean that the Oracle 32 bit client is not installed on this system. You should check with your DBA or the installer for Oracle and see is there optional components which have not actually been installed.
Having done that, I'd be inclined to delete and recreate the broker to ensure that everything is genuinely up to scratch. |
Thanks, I am checking with our DBAs, will report back tomorrow... |
|
| Back to top |
|
|
| JT |
| Posted: Tue Mar 08, 2005 9:32 am Post subject: |
|
|
Padawan
Joined: 27 Mar 2003
Posts: 1564
Location: Hartford, CT.
|
|
| Back to top |
|
|
|
|
