| Author |
Message
|
| dexter |
 Posted: Tue Feb 22, 2011 8:04 am Post subject: Broker not starting up |
 |
|
Apprentice
Joined: 07 Jan 2011
Posts: 32
Location: Michigan, USA
|
WMB v7.0, MQ v7.0, zLinux v11
This is the DEV broker. I started getting complaints since yesterday that the broker was down. It shut down twice and I had to restart it.
Now it won't even start back up.
Here's what the error log says.
Feb 22 10:02:37 ediwtxdev2 WebSphere Broker v7001[33102]: (BRKAD)[1]BIP8873I: Starting the component verification for component 'BRKAD'. : BRKAD.mqsistart: /build/S700_P/src/CommandLine/ImbCmdLib/ImbComponentVerification.cpp: 81: ImbComponentVerification::verify: :
Feb 22 10:02:37 ediwtxdev2 WebSphere Broker v7001[33102]: (BRKAD)[1]BIP8896E: Verification failed. Unable to access the registry with user ID 'ediwmb'. Additional information for IBM support: create exception. : BRKAD.mqsistart: /build/S700_P/src/CommandLine/ImbCmdLib/ImbEnvironmentVerification.cpp: 239: ImbEnvironmentVerification::verify: :
Feb 22 10:02:37 ediwtxdev2 WebSphere Broker v7001[33102]: (BRKAD)[1]BIP8893E: Unable to access file '/var/mqsi/BRKAD_bipcvp_temp.tmp' with user ID 'ediwmb'. Additional information for IBM support: close 28. : BRKAD.mqsistart: /build/S700_P/src/CommandLine/ImbCmdLib/ImbEnvironmentVerification.cpp: 967: ImbEnvironmentVerification::fileIO: :
Feb 22 10:02:37 ediwtxdev2 WebSphere Broker v7001[33102]: (BRKAD)[1]BIP8895E: Verification failed. Environment variable 'MQSI_REGISTRY' is incorrect or missing. : BRKAD.mqsistart: /build/S700_P/src/CommandLine/ImbCmdLib/ImbEnvironmentVerification.cpp: 507: ImbEnvironmentVerification::verify: :
Feb 22 10:02:37 ediwtxdev2 WebSphere Broker v7001[33102]: (BRKAD)[1]BIP8907E: Verification failed. Unable to verify Java level. : BRKAD.mqsistart: /build/S700_P/src/CommandLine/ImbCmdLib/ImbEnvironmentVerification.cpp: 509: ImbEnvironmentVerification::verify: :
Feb 22 10:02:37 ediwtxdev2 WebSphere Broker v7001[33102]: (BRKAD)[1]BIP8875W: The component verification for 'BRKAD' has finished, but one or more checks failed. : BRKAD.mqsistart: /build/S700_P/src/CommandLine/ImbCmdLib/ImbComponentVerification.cpp: 207: ImbComponentVerification::verify: :
Some abend files were generated and I've raised a PMR. But in the meanwhile, I wanted to know if you guys had any input on this. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Tue Feb 22, 2011 8:24 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Quote: |
| Environment variable 'MQSI_REGISTRY' is incorrect or missing. |
This is pretty clear... Did you mqsistart from a shell that does not have mqsiprofile sourced?
Did you modify mqsiprofile somehow? |
|
| Back to top |
|
|
| dexter |
| Posted: Tue Feb 22, 2011 8:31 am Post subject: |
|
|
Apprentice
Joined: 07 Jan 2011
Posts: 32
Location: Michigan, USA
|
I am using the same shell as always.
The mqsiprofile has been sourced. It is done every time I login to the admin id because i added the source command to the .profile file. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Feb 22, 2011 8:54 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Did someone change permissions on the /var/mqsi filesystem or otherwise mount it in a read-only mode? |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Feb 22, 2011 9:11 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| mqjeff wrote: |
| Did someone change permissions on the /var/mqsi filesystem or otherwise mount it in a read-only mode? |
Yeah, I've seen that before. They mistook 'var' for 'usr'.
These Unix sysadmins? It is so hard to get the skills these days.
Only joking.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| dexter |
| Posted: Tue Feb 22, 2011 12:32 pm Post subject: |
|
|
Apprentice
Joined: 07 Jan 2011
Posts: 32
Location: Michigan, USA
|
I finally got a hold of my UNIX admin.
No changes were made to the file directory structure.
No memory full issues with the mounted file directories. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Feb 22, 2011 12:35 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| dexter wrote: |
I finally got a hold of my UNIX admin.
No changes were made to the file directory structure.
No memory full issues with the mounted file directories. |
Then something else has hosed your mqsiprofile, which sets the missing environment variable.
Presumably you can see that variable defined correctly in the shell you're using?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| dexter |
| Posted: Tue Feb 22, 2011 12:58 pm Post subject: |
|
|
Apprentice
Joined: 07 Jan 2011
Posts: 32
Location: Michigan, USA
|
MQSI_REGISTRY=/var/mqsi
This is part of the 'env' output for my wmb admin id. |
|
| Back to top |
|
|
| dexter |
| Posted: Tue Feb 22, 2011 1:04 pm Post subject: |
|
|
Apprentice
Joined: 07 Jan 2011
Posts: 32
Location: Michigan, USA
|
PATH=/app/mqsi/7.0/jre16/bin:/app/mqsi/7.0/bin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/usr/lib/mit/bin:/usr/lib/mit/sbin:/home/db2inst1/sqllib/bin:/home/db2inst1/sqllib/adm:/home/db2inst1/sqllib/misc
I don't see anything missing here either. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Feb 22, 2011 1:42 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Presumably the real cause of the outage is buried in the rest of the data that you have hopefully already supplied to the PMR.
Things like the full environment output, the abend files, the rest of the system log, a startup trace... |
|
| Back to top |
|
|
| bsiggers |
| Posted: Tue Feb 22, 2011 4:02 pm Post subject: |
|
|
Acolyte
Joined: 09 Dec 2010
Posts: 53
Location: Vancouver, BC
|
Have you checked that you are, in fact, able to access and/or create the file below as the user 'ediwmb' as indicated in the error message?
'/var/mqsi/BRKAD_bipcvp_temp.tmp'
Have you tried doing a 'touch' in that directory to verify that /var/mqsi is writable?
Does that temp file above actually exist, and is maybe owned by another user and you're not able to write to it?
FYI - 'close 28', the error number in the message, indicates 'no space left on device' - you might want to re-check your disk space with your unix admins, as you may be out of inodes or something.  |
|
| Back to top |
|
|
| dexter |
| Posted: Wed Feb 23, 2011 9:15 am Post subject: |
|
|
Apprentice
Joined: 07 Jan 2011
Posts: 32
Location: Michigan, USA
|
My /var/mqsi file system was full.
The culprit was a larger than normal core file.
I also got the UNIX Admin to increase the file system size to 5GB from 2GB.
Thanks everyone. |
|
| Back to top |
|
|
| bsiggers |
| Posted: Wed Feb 23, 2011 11:29 am Post subject: |
|
|
Acolyte
Joined: 09 Dec 2010
Posts: 53
Location: Vancouver, BC
|
Thanks for posting the resolution!
Just one more response to this thread - 5GB for /var is probably way too small for a production server. Please have a look at how you're utilizing this space, as core dumps can happen at any time, and they can each be as big as the largest process size.
In addition, if your MQ data for Broker is also located on /var on this same machine, which it probably is, you run the risk of running into *real* problems. Be careful!
Some sites I've worked at have turned off core dumps for this very reason - but this is somewhat of an iffy solution which I wouldn't necessarily recommend, as by doing this you lose a valuable debugging and support tool. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Wed Feb 23, 2011 11:38 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| dexter wrote: |
| The culprit was a larger than normal core file. |
Hi Dex,
What process created the core file? "larger than normal"? Do you often get core files?
Just checking cause in my code, I always try to avoid cores. They are a pain to clean up.
Lance
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| dexter |
| Posted: Wed Feb 23, 2011 1:07 pm Post subject: |
|
|
Apprentice
Joined: 07 Jan 2011
Posts: 32
Location: Michigan, USA
|
Sorry Lance, I was in such a hurry to get my broker up that I deleted the core file without first taking a backup of it on my local system.
It was a DEV system, so, I had to get it back up before the 50 plus developers started chasing me down the road.
And no, this is the first core file to be generated. All the more reason I should have saved it.
I also learned a valuable lesson that there is no way to restore a deleted file on *NIX systems. |
|
| Back to top |
|
|
|
|
