| Author |
Message
|
| brianb |
 Posted: Thu Dec 30, 2010 4:01 pm Post subject: mqsirestoreconfigmgr |
 |
|
Voyager
Joined: 12 May 2010
Posts: 85
|
Hi
What are the implications of running a mqsirestoreconfigmgr ?
We are having an issue where we can no longer do deploys after a full system restart
WMB6.1.0.7
AIX 6.1
I have a configuration manager backup from a week prior to the issue. What would restoring this actuall do ?
error
Dec 30 16:33:54 servername user:info WebSphere Broker v6107[9240634]: (CManager01)[4627]BIP1205E: Unexpected exception accessing configuration repository; exception text: 'No current connection. error code: 40000'. : CMmanager.agent: /
build/S610_P/src/DataFlowEngine/NativeTrace/ImbNativeTrace.cpp: 739: com.ibm.bro
ker.config.RegistrationHandler.run: :
this gets gets repeated 1000's of time and puts to SYSTEM.BROKER.ADMIN.REPLY
There are also messages in SYSTEM.BROKER.CONFIG.QUEUE should these be removed prior to restore ?
I have seen references to the above error that refer to FP 6.1.0.9 |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Thu Dec 30, 2010 6:56 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I don't think FP 6.1.0.9 is released yet.
So if you've found APARs that are targeted for that release, you may still have to wait.
What happens when you mqsistop the configmgr?
Do you see that all associated configmgr processes (bipservice and bipconfigmgr) have ended?
Do you then see any errors logged when you mqsistart the configmgr?
Do you have anything else that might be attempting - harmfully and falsely- to inquire or connect to the configmgr's private repository? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Dec 31, 2010 10:14 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
 with mqjeff on this one.
It looks more likely from what you posted so far that you are suffering from some connection problem to the config mgr's repository.
Likely somebody trying to browse it in SQL? Or the mqsistop having left config mgr processes running? If you see processes still running against the config mgr after you stopped it, you will need to use kill -9 against them.
Do not restore yet. Try just recycling as Jeff suggested.
If that does not work, open a PMR. 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| brianb |
| Posted: Fri Dec 31, 2010 11:04 am Post subject: mqsirestoreconfigmgr |
|
|
Voyager
Joined: 12 May 2010
Posts: 85
|
Hi
I found FP 9 is not released untill first quarter when I went to download it  so maybe FP8
anyway there is no problem stopping config manager the bip process's stop and I do not see anyything hanging around with ps -ef |grep "cfmanname"
When It starts back up the 4000 error begins immediatly as above.
Not sure how to check if something is accessing the repository ?
I wanted to be at latest FP before opening PMR
I did try the mqsirestore but it did not help
the error began right after the server was rebooted for maint task
I am looking at deleteing the cfgmgr and recreating it |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Dec 31, 2010 11:30 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
If you delete the configmgr you will have to adopt all of your brokers or delete and recreate them as well...
If the errors occurred right after a server reboot, then perhaps the configmgr file system is mounted read only. |
|
| Back to top |
|
|
| brianb |
| Posted: Fri Dec 31, 2010 11:47 am Post subject: |
|
|
Voyager
Joined: 12 May 2010
Posts: 85
|
Hi Jeff
thanks for the help
Yes that is sort of the plan...testing it out in test environment
Of course I would like to find the actual cause before doing this but am running out of time
As for filesystem would that be where the work path var/mqsi is mounted ?
also - I can connect to the cfgmgr between the errors for a few seconds just not long enough to do a deploy
bipconfig
Jan 1 06:44:31 server1 user:info WebSphere Broker v6107[15204438]: (CM_xxxxx)[1]BIP1003I: The Configuration Manager is available for use. : CM_xxxxx.agent: /build/S610_P/src/Config/ImbConfigMgrAgent.cpp: 714: ImbConfigMgrAgent::employJavaConfigManager: :
Jan 1 06:44:33 server1 user:info syslog: ifconfig -au
Jan 1 06:44:35 server1 user:info WebSphere Broker v6107[15204438]: (CM_xxxxx)[4627]BIP1205E: Unexpected exception accessing configuration repository; exception text: 'No current connection. error code: 40000'. : CM_xxxx.agent: /build/S610_P/src/DataFlowEngine/NativeTrace/ImbNativeTrace.cpp: 739: com.ibm.broker.config.RegistrationHandler.run: : |
|
| Back to top |
|
|
| brianb |
| Posted: Fri Dec 31, 2010 12:03 pm Post subject: |
|
|
Voyager
Joined: 12 May 2010
Posts: 85
|
Hi
just found this
Jan 1 06:52:28 server1 user:info WebSphere Broker v6107[20316370]: (CM_xxxx)[4370]BIP1107S: Unexpected exception in Configuration Manager MQProcessor class No current connection. error code: 40000 method; exception text: 'com.ibm.broker.config.ConfigManagerSevereException: handleSQLException'. : CM_xxxx.agent: /build/S610_P/src/DataFlowEngine/NativeTrace/ImbNativeTrace.cpp: 739: processSimple: :
[/b] |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Dec 31, 2010 12:25 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
It's time for a PMR.
Even if FP 9 isn't available, support could give you an APAR. Or advice.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| brianb |
| Posted: Fri Dec 31, 2010 12:39 pm Post subject: |
|
|
Voyager
Joined: 12 May 2010
Posts: 85
|
agreed
radical surgery not yet called for  |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Dec 31, 2010 12:50 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Stop the configmgr entirely.
Make sure there is no bipconfigmgr or bipservice process running.
confirm that /var/mqsi/components/<ConfigMgr name>/ is mounted read/write. Review the derby.log file in that location for more information and hints.
confirm that there are FDC files in /var/mqsi/common/errors or abend files at least. Review them for hints.
Sit your sysadmins in a conference room and deny them coffee until they admit to what they changed during the last reboot.. |
|
| Back to top |
|
|
| brianb |
| Posted: Fri Dec 31, 2010 1:16 pm Post subject: |
|
|
Voyager
Joined: 12 May 2010
Posts: 85
|
Stop the configmgr entirely. - Done
Make sure there is no bipconfigmgr or bipservice process running - done
confirm that /var/mqsi/components/<ConfigMgr name>/ is mounted read/write. Review the derby.log file in that location for more information and hints. I am able to create a file in /var/mqsi/components/<ConfigMgr name>/
only 1 message in derby copied below generated last night
confirm that there are FDC files in /var/mqsi/common/errors or abend files at least. Review them for hints.
No FDC's
Sit your sysadmins in a conference room and deny them coffee until they admit to what they changed during the last reboot.. or call em in on New Years day 
Derby.log
2010-12-31 20:40:34.725 GMT:
Booting Derby version The Apache Software Foundation - Apache Derby - 10.2.2.1
- (815839): instance c013800d-012d-3e29-0fb5-00004c95941e
on database directory /var/mqsi/components/CMxxxx/BERNARD
Database Class Loader started - derby.database.classpath=''
2010-12-31 20:40:43.857 GMT Thread[BrokerManager,5,main] (XID = 56820505), (SESS
IONID = 4), (DATABASE = /var/mqsi/components/CMxxxx/BERNARD), (DRDAID = nu
ll), Cleanup action starting
2010-12-31 20:40:43.858 GMT Thread[BrokerManager,5,main] (XID = 56820505), (SESS
IONID = 4), (DATABASE = /var/mqsi/components/CMxxxx/BERNARD), (DRDAID = nu
ll), Failed Statement is: UPDATE CBROKER SET CSTATUS = ? , CQMANAGER = ? , CSUBS
CRIBED = ? , CNAME = ? , CLASTUPDATE = ? , CXMLDATA = ? WHERE CUUID = '583048a9
-2101-0000-0080-fa0dddaa1540' AND CSECTION='DEPLED' with 6 parameters begin para
meter #1: 00000100 :end parameter begin parameter #2: QMxxxx :end paramete
r begin parameter #3: Y :end parameter begin parameter #4: BK_xxxx :end par
ameter begin parameter #5: 2010-12-17 10:03:22.469 :end parameter begin paramete
r #6: CLOB(704768) :end parameter
ERROR XSDG1: Page Page(32767,Container(0, 960)) could not be written to disk, pl
ease check if disk is full.
Lots of disk space
|
|
| Back to top |
|
|
| brianb |
| Posted: Fri Dec 31, 2010 2:30 pm Post subject: |
|
|
Voyager
Joined: 12 May 2010
Posts: 85
|
This may be more helpful
mqsistop CMxxxx
export MQSI_CONSOLEMODE=1
export MQSIJVERBOSE=-DMQSI_CONSOLEMODE=1
bipconfigmgr CMxxxx
(1/1/11 9:24:39 AM) BIP1202S: Unexpected exception in the Configuration Manager DBManager class handleSQLException method; exception text: 'Initial Exception; SQLState: '08006' errorCode: '45000' Msg: 'Database '/var/mqsi/components/CMxxxx/BERNARD' shutdown.'', 'ERROR 08006: Database '/var/mqsi/components/CMxxxx/BERNARD' shutdown. at org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.shutdownDatabaseException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source) at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedResultSet.closeOnTransactionError(Unknown Source) at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(Unknown Source) at org.apache.derby.impl.jdbc.EmbedResultSet.next(Unknown Source) at com.ibm.broker.config.DBManager.getEntry(DBManager.java:5507) at com.ibm.broker.config.RegistrationHandler.getRequiredObjectData(RegistrationHandler.java:1179) at com.ibm.broker.config.RegistrationHandler.notifyProxies(RegistrationHandler.java:629) at com.ibm.broker.config.RegistrationHandler.notifyAllProxies(RegistrationHandler.java:1978) at com.ibm.broker.config.RegistrationHandler.run(RegistrationHandler.java:1330) at java.lang.Thread.run(Thread.java:811) '.
(1/1/11 9:24:39 AM) BIP1202S: Unexpected exception in the Configuration Manager DBManager class handleSQLException method; exception text: 'Initial Exception; SQLState: 'XSDG1' errorCode: '45000' Msg: 'Page Page(32767,Container(0, 960)) could not be written to disk, please check if disk is full.'', 'ERROR XSDG1: Page Page(32767,Container(0, 960)) could not be written to disk, please check if disk is full. at org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at org.apache.derby.impl.store.raw.data.CachedPage.writePage(Unknown Source) at org.apache.derby.impl.store.raw.data.CachedPage.createIdentity(Unknown Source) at org.apache.derby.impl.services.cache.CachedItem.takeOnIdentity(Unknown Source) at org.apache.derby.impl.services.cache.Clock.addEntry(Unknown Source) at org.apache.derby.impl.services.cache.Clock.create(Unknown Source) at org.apache.derby.impl.store.raw.data.FileContainer.initPage(Unknown Source) at org.apache.derby.impl.store.raw.data.FileContainer.newPage(Unknown Source) at org.apache.derby.impl.store.raw.data.BaseContainer.addPage(Unknown Source) at org.apache.derby.impl.store.raw.data.StoredPage.getNewOverflowPage(Unknown Source) at org.apache.derby.impl.store.raw.data.BasePage.insertLongColumn(Unknown Source) at org.apache.derby.impl.store.raw.data.StoredPage.doUpdateAtSlot(Unknown Source) at org.apache.derby.impl.store.raw.data.BasePage.updateAtSlot(Unknown Source) at org.apache.derby.impl.store.access.conglomerate.GenericConglomerateController.replace(Unknown Source) at org.apache.derby.impl.sql.execute.RowChangerImpl.updateRow(Unknown Source) at org.apache.derby.impl.sql.execute.UpdateResultSet.collectAffectedRows(Unknown Source) at org.apache.derby.impl.sql.execute.UpdateResultSet.open(Unknown Source) at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source) at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown Source) at com.ibm.broker.config.DBManager.putEntry(DBManager.java:6313) at com.ibm.broker.config.DBManager.putEntry(DBManager.java:5819) at com.ibm.broker.config.BrokerManager.resolveSync(BrokerManager.java:14138) at com.ibm.broker.config.BrokerManager.listen(BrokerManager.java:4308) at com.ibm.broker.config.BrokerManager.run(BrokerManager.java:912) at java.lang.Thread.run(Thread.java:811) '.
Next Exception; SQLState: 'XJ001' errorCode: '0' Msg: 'Java exception: 'A file cannot be larger than the value set by ulimit.: java.io.IOException'.'
java.io.IOException: A file cannot be larger than the value set by ulimit.
at java.io.RandomAccessFile.write(RandomAccessFile.java:493)
at org.apache.derby.impl.store.raw.data.RAFContainer.writePage(Unknown Source)
at org.apache.derby.impl.store.raw.data.CachedPage.writePage(Unknown Source)
at org.apache.derby.impl.store.raw.data.CachedPage.createIdentity(Unknown Source)
at org.apache.derby.impl.services.cache.CachedItem.takeOnIdentity(Unknown Source)
at org.apache.derby.impl.services.cache.Clock.addEntry(Unknown Source)
at org.apache.derby.impl.services.cache.Clock.create(Unknown Source)
at org.apache.derby.impl.store.raw.data.FileContainer.initPage(Unknown Source)
at org.apache.derby.impl.store.raw.data.FileContainer.newPage(Unknown Source)
at org.apache.derby.impl.store.raw.data.BaseContainer.addPage(Unknown Source)
at org.apache.derby.impl.store.raw.data.StoredPage.getNewOverflowPage(Unknown Source)
at org.apache.derby.impl.store.raw.data.BasePage.insertLongColumn(Unknown Source)
at org.apache.derby.impl.store.raw.data.StoredPage.doUpdateAtSlot(Unknown Source)
at org.apache.derby.impl.store.raw.data.BasePage.updateAtSlot(Unknown Source)
at org.apache.derby.impl.store.access.conglomerate.GenericConglomerateController.replace(Unknown Source)
at org.apache.derby.impl.sql.execute.RowChangerImpl.updateRow(Unknown Source)
at org.apache.derby.impl.sql.execute.UpdateResultSet.collectAffectedRows(Unknown Source)
at org.apache.derby.impl.sql.execute.UpdateResultSet.open(Unknown Source)
at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown Source)
at com.ibm.broker.config.DBManager.putEntry(DBManager.java:6313)
at com.ibm.broker.config.DBManager.putEntry(DBManager.java:5819)
at com.ibm.broker.config.BrokerManager.resolveSync(BrokerManager.java:14138)
at com.ibm.broker.config.BrokerManager.listen(BrokerManager.java:4308)
at com.ibm.broker.config.BrokerManager.run(BrokerManager.java:912)
at java.lang.Thread.run(Thread.java:811)
(1/1/11 9:24:39 AM) BIP1202S: Unexpected exception in the Configuration Manager DBManager class handleSQLException method; exception text: 'Next Exception; SQLState: 'XJ001' errorCode: '0' Msg: 'Java exception: 'A file cannot be larger than the value set by ulimit.: java.io.IOException'.'', 'java.io.IOException: A file cannot be larger than the value set by ulimit. at java.io.RandomAccessFile.write(RandomAccessFile.java:493) at org.apache.derby.impl.store.raw.data.RAFContainer.writePage(Unknown Source) at org.apache.derby.impl.store.raw.data.CachedPage.writePage(Unknown Source) at org.apache.derby.impl.store.raw.data.CachedPage.createIdentity(Unknown Source) at org.apache.derby.impl.services.cache.CachedItem.takeOnIdentity(Unknown Source) at org.apache.derby.impl.services.cache.Clock.addEntry(Unknown Source) at org.apache.derby.impl.services.cache.Clock.create(Unknown Source) at org.apache.derby.impl.store.raw.data.FileContainer.initPage(Unknown Source) at org.apache.derby.impl.store.raw.data.FileContainer.newPage(Unknown Source) at org.apache.derby.impl.store.raw.data.BaseContainer.addPage(Unknown Source) at org.apache.derby.impl.store.raw.data.StoredPage.getNewOverflowPage(Unknown Source) at org.apache.derby.impl.store.raw.data.BasePage.insertLongColumn(Unknown Source) at org.apache.derby.impl.store.raw.data.StoredPage.doUpdateAtSlot(Unknown Source) at org.apache.derby.impl.store.raw.data.BasePage.updateAtSlot(Unknown Source) at org.apache.derby.impl.store.access.conglomerate.GenericConglomerateController.replace(Unknown Source) at org.apache.derby.impl.sql.execute.RowChangerImpl.updateRow(Unknown Source) at org.apache.derby.impl.sql.execute.UpdateResultSet.collectAffectedRows(Unknown Source) at org.apache.derby.impl.sql.execute.UpdateResultSet.open(Unknown Source) at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source) at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown Source) at com.ibm.broker.config.DBManager.putEntry(DBManager.java:6313) at com.ibm.broker.config.DBManager.putEntry(DBManager.java:5819) at com.ibm.broker.config.BrokerManager.resolveSync(BrokerManager.java:14138) at com.ibm.broker.config.BrokerManager.listen(BrokerManager.java:4308) at com.ibm.broker.config.BrokerManager.run(BrokerManager.java:912) at java.lang.Thread.run(Thread.java:811) '.
com.ibm.broker.config.ConfigManagerLoggedException: handleSQLException<<Page Page(32767,Container(0, 960)) could not be written to disk, please check if disk is full.
at com.ibm.broker.config.DBManager.handleSQLException(DBManager.java:1757)
at com.ibm.broker.config.DBManager.putEntry(DBManager.java:6349)
at com.ibm.broker.config.DBManager.putEntry(DBManager.java:5819)
at com.ibm.broker.config.BrokerManager.resolveSync(BrokerManager.java:14138)
at com.ibm.broker.config.BrokerManager.listen(BrokerManager.java:4308)
at com.ibm.broker.config.BrokerManager.run(BrokerManager.java:912)
at java.lang.Thread.run(Thread.java:811)
(1/1/11 9:25:03 AM) BIP1083I: The Configuration Manager 'CMxxxx' received a CMP request from user 'QMxxxx\mqbrk' (Proxy session ID=61b9883e-2d01-0000-0080-a9501a3c36c9, type=register).
(1/1/11 9:25:03 AM) BIP1084I: The Configuration Manager 'CMxxxx' responded to the CMP request from user 'QMxxxx\mqbrk' (Proxy session ID=61b9883e-2d01-0000-0080-a9501a3c36c9, 0/1 request elements successful).
mqbrk@QMxxxx:/# |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Dec 31, 2010 3:27 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
I think this is the crutial part for your PMR:
| Quote: |
at java.lang.Thread.run(Thread.java:811)
(1/1/11 9:24:39 AM) BIP1202S: Unexpected exception in the Configuration Manager DBManager class handleSQLException method; exception text: 'Next Exception; SQLState: 'XJ001' errorCode: '0' Msg: 'Java exception: 'A file cannot be larger than the value set by ulimit.: java.io.IOException'.'', 'java.io.IOException: A file cannot be larger than the value set by ulimit. at java.io.RandomAccessFile.write(RandomAccessFile.java:493) at |
Are you using the service user to start the config mgr or some other user in the mqbrokers group?
| mqjeff wrote: |
| Sit your sysadmins in a conference room and deny them coffee until they admit to what they changed during the last reboot.. |
Like changing ulimit on you??
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Dec 31, 2010 4:19 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| fjb_saper wrote: |
| mqjeff wrote: |
| Sit your sysadmins in a conference room and deny them coffee until they admit to what they changed during the last reboot.. |
Like changing ulimit on you?? |
In this case, it may simply be that they weren't informed that they should have set ulimit to "unlimited" during the initial build of the box, and the Broker Domain has simply grown larger than the ulimit allowed.
It could also be that there needs to be some compression done on the configmgr repository... Which the PMR will provide scripts for... |
|
| Back to top |
|
|
| brianb |
| Posted: Fri Dec 31, 2010 7:49 pm Post subject: |
|
|
Voyager
Joined: 12 May 2010
Posts: 85
|
yes it turned out to be ulimit
thanks for all the help guys |
|
| Back to top |
|
|
|
|
