| Author |
Message
|
| robertr |
 Posted: Thu Oct 12, 2006 3:29 am Post subject: Execution Group Shutdown... |
 |
|
Novice
Joined: 05 Jan 2005
Posts: 19
|
Hi All,
I am experiencing some intermitten problems with my execution group and as a result of these problems, the execution group is shutting down.
Below is an exert from my syslog ...
| Code: |
Oct 10 12:49:41 airbrkdr1 WebSphere Broker v6001[78212]: (BKINT01.EG01)[6683]BIP2321E: Database error: ODBC return code '-1'. : BKINT01.bdf8d49f-0c01-0000-0080-cdece9acbcf0: /build/S600_P/src/DataFlowEngine/ImbOdbc.cpp: 227: ImbOdbcHandle::checkRcInner: :
Oct 10 12:49:41 airbrkdr1 WebSphere Broker v6001[78212]: (BKINT01.EG01)[6683]BIP2322E: Database error: SQL State '08001'; Native Error Code '-1224'; Error Text '[IBM][CLI Driver] SQL1224N A database agent could not be started to service a request, or was terminated as a result of a database system shutdown or a force command. SQLSTATE=55032 '. : BKINT01.bdf8d49f-0c01-0000-0080-cdece9acbcf0: /build/S600_P/src/DataFlowEngine/ImbOdbc.cpp: 355: ImbOdbcHandle::checkRcInner: :
Oct 10 12:50:18 airbrkdr1 WebSphere Broker v6001[93108]: (BKINT01)[2057]BIP2060W: The broker has detected that the Execution Group EG01, process ID 78212, has shutdown. : BKINT01.agent: /build/S600_P/src/AdminAgent/ImbAdminAgent.cpp: 4070: ImbAdminAgent::startAndMonitorADataFlowEngine: : |
The db2 log contains...
| Code: |
2006-10-11-17.44.34.828803+060 I686719C522 LEVEL: Severe (OS)
PID : 125328 TID : 6683 PROC : DataFlowEngine
INSTANCE: siint01 NODE : 000
FUNCTION: DB2 UDB, SQO Memory Management, sqlocshr2, probe:200
CALLED : OS, -, shmat
OSERR : EMFILE (24) "The process file table is full."
DATA #1 : Memory set handle, PD_TYPE_OSS_MEM_SET_HDL, 20 bytes
0x3C4572E0 : FFFF FFFF FFFF FFFF 0000 0000 079C 00D4 ................
0x3C4572F0 : 0004 0000 .... |
I believe this maybe a memory issue with the execution group EG01 or with DB2 but as of now i am unable to find hard evidence of this.
If anyone has seen anything similar or could point me in the right direction I would be very appreciative.
Rob |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Thu Oct 12, 2006 3:43 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
|
| Back to top |
|
|
| sieijish |
| Posted: Thu Oct 12, 2006 3:44 am Post subject: |
|
|
Acolyte
Joined: 29 Nov 2004
Posts: 67
Location: London
|
What is the MAXFILOP of ur DB?
Also, what is the ulimit for the 'nofiles' ? |
|
| Back to top |
|
|
| robertr |
| Posted: Thu Oct 12, 2006 5:36 am Post subject: |
|
|
Novice
Joined: 05 Jan 2005
Posts: 19
|
Hi Sieijish,
The MAXFILOP of the db is 64.
The ulimit for the nofiles is 2000.
Ulimit values included below.
| Code: |
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 245760
stack(kbytes) 16384
memory(kbytes) unlimited
coredump(blocks) unlimited
nofiles(descriptors) 2000 |
Thanks in advance for your help.
Rob |
|
| Back to top |
|
|
| dsriksha |
| Posted: Thu Oct 12, 2006 6:14 am Post subject: |
|
|
Voyager
Joined: 27 Feb 2005
Posts: 95
|
Hi Rob:
Did you get a chance to look for SQL1224N from below Jeff link. 
|
|
| Back to top |
|
|
| robertr |
| Posted: Thu Oct 12, 2006 6:31 am Post subject: |
|
|
Novice
Joined: 05 Jan 2005
Posts: 19
|
Hi,
I looked at the link and found it very helpful - Thanks Jeff.
However, I was hoping that the fix would not be as drastic as dropping my broker - i have multiple environments with multiple brokers and thus it would have widespread implications.
I will explore all other avenues before dropping the broker.
Do you know if just stopping the broker would be sufficient?
Rob |
|
| Back to top |
|
|
| sieijish |
| Posted: Thu Oct 12, 2006 6:36 am Post subject: |
|
|
Acolyte
Joined: 29 Nov 2004
Posts: 67
Location: London
|
Put this in your db2user & broker user profile. That will solve the issue of 10 db2 connections limit without using the TCP/IP loop back solution given in the URL |
|
| Back to top |
|
|
| sieijish |
| Posted: Thu Oct 12, 2006 6:55 am Post subject: |
|
|
Acolyte
Joined: 29 Nov 2004
Posts: 67
Location: London
|
After adding export EXTSHM=ON in the db2 instance owner profile, you should login as your db2 instance owner (with the new profile) and execute the following command
| Code: |
db2set DB2ENVLIST=EXTSHM |
Make sure that you have EXTSHM set while executing this above command
After than restart db2 for this to take effect. |
|
| Back to top |
|
|
| dsriksha |
| Posted: Thu Oct 12, 2006 6:59 am Post subject: |
|
|
Voyager
Joined: 27 Feb 2005
Posts: 95
|
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Oct 12, 2006 7:01 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
I would open a PMR on the necessity to recreate the broker.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| sieijish |
| Posted: Thu Oct 12, 2006 7:08 am Post subject: |
|
|
Acolyte
Joined: 29 Nov 2004
Posts: 67
Location: London
|
I do not think you have to recreate the broker. You can change the DataSource name by using
| Code: |
| mqsiservice <brokerName> -r DataSourceName=NEWNAME |
|
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Oct 12, 2006 7:08 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
You can also recatalog the database locally under a different name, and then catalog it remotely using the same name...
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| robertr |
| Posted: Thu Oct 12, 2006 7:09 am Post subject: |
|
|
Novice
Joined: 05 Jan 2005
Posts: 19
|
Thank you very much Sieijish.
I have implemented the changes you suggested and will monitor the situation for the next couple of days.
Rob |
|
| Back to top |
|
|
| robertr |
| Posted: Tue Oct 24, 2006 8:38 am Post subject: |
|
|
Novice
Joined: 05 Jan 2005
Posts: 19
|
Hi All,
I implemented the changes suggested and unfortunately they have caused an error. I startDB2 and the broker but they are shutdown or hang within minutes when removing a flow from the execution group. The error received in the syslog is enclosed below.
| Code: |
Oct 24 17:13:50 airbrkdr1 WebSphere Broker v6001[102234]: (BKSYSTEM01)[1]BIP2048E: An Exception was caught while issuing database SQL command connect. : BKSYSTEM01.agent: /build/S600_P/src/AdminAgent/ImbAdminStore.cpp: 304: ImbAdminStore::Connect: :
Oct 24 17:13:50 airbrkdr1 WebSphere Broker v6001[102234]: (BKSYSTEM01)[1]BIP2321E: Database error: ODBC return code '-1'. : BKSYSTEM01.agent: /build/S600_P/src/DataFlowEngine/ImbOdbc.cpp: 227: ImbOdbcHandle::checkRcInner: :
Oct 24 17:13:50 airbrkdr1 WebSphere Broker v6001[102234]: (BKSYSTEM01)[1]BIP2322E: Database error: SQL State '08001'; Native Error Code '-1032'; Error Text '[IBM][CLI Driver] SQL1032N No start database manager command was issued. SQLSTATE=57019 '. : BKSYSTEM01.agent: /build/S600_P/src/DataFlowEngine/ImbOdbc.cpp: 355: ImbOdbcHandle::checkRcInner: |
I also receive an error message in the event log in the toolkit. This occurs when trying to remove a flow from the execution group after implementing the "export EXTSHM=ON" suggesstion in the users profile.
| Code: |
BIP2035E: The internal configuration message was rejected due to broker persistent store problem.
The broker's persistent store is not available, the internal configuration message was rejected because of this.
Ensure that the broker's database is restarted. |
I have searched high and low for an explanation so if anyone has any advice/suggesstions - please let me know!
Regards,
Rob |
|
| Back to top |
|
|
|
|
