ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » Broker not starting up

Post new topic  Reply to topic Goto page 1, 2  Next
 Broker not starting up « View previous topic :: View next topic » 
Author Message
dexter
PostPosted: Tue Feb 22, 2011 8:04 am    Post subject: Broker not starting up Reply with quote

Apprentice

Joined: 07 Jan 2011
Posts: 32
Location: Michigan, USA

WMB v7.0, MQ v7.0, zLinux v11

This is the DEV broker. I started getting complaints since yesterday that the broker was down. It shut down twice and I had to restart it.
Now it won't even start back up.

Here's what the error log says.
Feb 22 10:02:37 ediwtxdev2 WebSphere Broker v7001[33102]: (BRKAD)[1]BIP8873I: Starting the component verification for component 'BRKAD'. : BRKAD.mqsistart: /build/S700_P/src/CommandLine/ImbCmdLib/ImbComponentVerification.cpp: 81: ImbComponentVerification::verify: :
Feb 22 10:02:37 ediwtxdev2 WebSphere Broker v7001[33102]: (BRKAD)[1]BIP8896E: Verification failed. Unable to access the registry with user ID 'ediwmb'. Additional information for IBM support: create exception. : BRKAD.mqsistart: /build/S700_P/src/CommandLine/ImbCmdLib/ImbEnvironmentVerification.cpp: 239: ImbEnvironmentVerification::verify: :
Feb 22 10:02:37 ediwtxdev2 WebSphere Broker v7001[33102]: (BRKAD)[1]BIP8893E: Unable to access file '/var/mqsi/BRKAD_bipcvp_temp.tmp' with user ID 'ediwmb'. Additional information for IBM support: close 28. : BRKAD.mqsistart: /build/S700_P/src/CommandLine/ImbCmdLib/ImbEnvironmentVerification.cpp: 967: ImbEnvironmentVerification::fileIO: :
Feb 22 10:02:37 ediwtxdev2 WebSphere Broker v7001[33102]: (BRKAD)[1]BIP8895E: Verification failed. Environment variable 'MQSI_REGISTRY' is incorrect or missing. : BRKAD.mqsistart: /build/S700_P/src/CommandLine/ImbCmdLib/ImbEnvironmentVerification.cpp: 507: ImbEnvironmentVerification::verify: :
Feb 22 10:02:37 ediwtxdev2 WebSphere Broker v7001[33102]: (BRKAD)[1]BIP8907E: Verification failed. Unable to verify Java level. : BRKAD.mqsistart: /build/S700_P/src/CommandLine/ImbCmdLib/ImbEnvironmentVerification.cpp: 509: ImbEnvironmentVerification::verify: :
Feb 22 10:02:37 ediwtxdev2 WebSphere Broker v7001[33102]: (BRKAD)[1]BIP8875W: The component verification for 'BRKAD' has finished, but one or more checks failed. : BRKAD.mqsistart: /build/S700_P/src/CommandLine/ImbCmdLib/ImbComponentVerification.cpp: 207: ImbComponentVerification::verify: :

Some abend files were generated and I've raised a PMR. But in the meanwhile, I wanted to know if you guys had any input on this.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Tue Feb 22, 2011 8:24 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

Quote:
Environment variable 'MQSI_REGISTRY' is incorrect or missing.


This is pretty clear... Did you mqsistart from a shell that does not have mqsiprofile sourced?

Did you modify mqsiprofile somehow?
Back to top
View user's profile Send private message
dexter
PostPosted: Tue Feb 22, 2011 8:31 am    Post subject: Reply with quote

Apprentice

Joined: 07 Jan 2011
Posts: 32
Location: Michigan, USA

I am using the same shell as always.
The mqsiprofile has been sourced. It is done every time I login to the admin id because i added the source command to the .profile file.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Tue Feb 22, 2011 8:54 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

Did someone change permissions on the /var/mqsi filesystem or otherwise mount it in a read-only mode?
Back to top
View user's profile Send private message
smdavies99
PostPosted: Tue Feb 22, 2011 9:11 am    Post subject: Reply with quote

Jedi Council

Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.

mqjeff wrote:
Did someone change permissions on the /var/mqsi filesystem or otherwise mount it in a read-only mode?


Yeah, I've seen that before. They mistook 'var' for 'usr'.
These Unix sysadmins? It is so hard to get the skills these days.

Only joking.

_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995

Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions.
Back to top
View user's profile Send private message
dexter
PostPosted: Tue Feb 22, 2011 12:32 pm    Post subject: Reply with quote

Apprentice

Joined: 07 Jan 2011
Posts: 32
Location: Michigan, USA

I finally got a hold of my UNIX admin.
No changes were made to the file directory structure.
No memory full issues with the mounted file directories.
Back to top
View user's profile Send private message
Vitor
PostPosted: Tue Feb 22, 2011 12:35 pm    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

dexter wrote:
I finally got a hold of my UNIX admin.
No changes were made to the file directory structure.
No memory full issues with the mounted file directories.


Then something else has hosed your mqsiprofile, which sets the missing environment variable.

Presumably you can see that variable defined correctly in the shell you're using?
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
dexter
PostPosted: Tue Feb 22, 2011 12:58 pm    Post subject: Reply with quote

Apprentice

Joined: 07 Jan 2011
Posts: 32
Location: Michigan, USA

MQSI_REGISTRY=/var/mqsi

This is part of the 'env' output for my wmb admin id.
Back to top
View user's profile Send private message
dexter
PostPosted: Tue Feb 22, 2011 1:04 pm    Post subject: Reply with quote

Apprentice

Joined: 07 Jan 2011
Posts: 32
Location: Michigan, USA

PATH=/app/mqsi/7.0/jre16/bin:/app/mqsi/7.0/bin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/usr/lib/mit/bin:/usr/lib/mit/sbin:/home/db2inst1/sqllib/bin:/home/db2inst1/sqllib/adm:/home/db2inst1/sqllib/misc

I don't see anything missing here either.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Tue Feb 22, 2011 1:42 pm    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

Presumably the real cause of the outage is buried in the rest of the data that you have hopefully already supplied to the PMR.

Things like the full environment output, the abend files, the rest of the system log, a startup trace...
Back to top
View user's profile Send private message
bsiggers
PostPosted: Tue Feb 22, 2011 4:02 pm    Post subject: Reply with quote

Acolyte

Joined: 09 Dec 2010
Posts: 53
Location: Vancouver, BC

Have you checked that you are, in fact, able to access and/or create the file below as the user 'ediwmb' as indicated in the error message?

'/var/mqsi/BRKAD_bipcvp_temp.tmp'

Have you tried doing a 'touch' in that directory to verify that /var/mqsi is writable?

Does that temp file above actually exist, and is maybe owned by another user and you're not able to write to it?

FYI - 'close 28', the error number in the message, indicates 'no space left on device' - you might want to re-check your disk space with your unix admins, as you may be out of inodes or something.
Back to top
View user's profile Send private message
dexter
PostPosted: Wed Feb 23, 2011 9:15 am    Post subject: Reply with quote

Apprentice

Joined: 07 Jan 2011
Posts: 32
Location: Michigan, USA

My /var/mqsi file system was full.
The culprit was a larger than normal core file.
I also got the UNIX Admin to increase the file system size to 5GB from 2GB.

Thanks everyone.
Back to top
View user's profile Send private message
bsiggers
PostPosted: Wed Feb 23, 2011 11:29 am    Post subject: Reply with quote

Acolyte

Joined: 09 Dec 2010
Posts: 53
Location: Vancouver, BC

Thanks for posting the resolution!

Just one more response to this thread - 5GB for /var is probably way too small for a production server. Please have a look at how you're utilizing this space, as core dumps can happen at any time, and they can each be as big as the largest process size.

In addition, if your MQ data for Broker is also located on /var on this same machine, which it probably is, you run the risk of running into *real* problems. Be careful!

Some sites I've worked at have turned off core dumps for this very reason - but this is somewhat of an iffy solution which I wouldn't necessarily recommend, as by doing this you lose a valuable debugging and support tool.
Back to top
View user's profile Send private message
lancelotlinc
PostPosted: Wed Feb 23, 2011 11:38 am    Post subject: Reply with quote

Jedi Knight

Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA

dexter wrote:
The culprit was a larger than normal core file.


Hi Dex,

What process created the core file? "larger than normal"? Do you often get core files?

Just checking cause in my code, I always try to avoid cores. They are a pain to clean up.

Lance
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER
Back to top
View user's profile Send private message Send e-mail
dexter
PostPosted: Wed Feb 23, 2011 1:07 pm    Post subject: Reply with quote

Apprentice

Joined: 07 Jan 2011
Posts: 32
Location: Michigan, USA

Sorry Lance, I was in such a hurry to get my broker up that I deleted the core file without first taking a backup of it on my local system.
It was a DEV system, so, I had to get it back up before the 50 plus developers started chasing me down the road.
And no, this is the first core file to be generated. All the more reason I should have saved it.
I also learned a valuable lesson that there is no way to restore a deleted file on *NIX systems.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » Broker not starting up
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.