| Author |
Message
|
| bsiggers |
 Posted: Wed Aug 01, 2012 8:58 pm Post subject: MB 7.0.0.3 - Unable to delete execution group |
 |
|
Acolyte
Joined: 09 Dec 2010
Posts: 53
Location: Vancouver, BC
|
Hello Broker Experts,
We've run into an issue on one of our Linux MB 7.0.0.3 broker standalone instances. It's similar to the good old phantom execution group issue, but I didn't know this could occur in MB 7 as well.
The start of the issue cause was a manual mistake in setting up the JVM arguments for an execution group, which caused the execution group to not be able to start up correctly as the JVM parameters weren't valid. It also left it in a state where the issue could not be corrected via the MB explorer or command line, and worst of all, it also can't be deleted, seemingly because the execution group can't be started. Starting and stopping the broker instance does not help. The DataFlowEngine for the process very briefly starts and stops repeatedly as the broker tries to restart the process, but there's no way of letting it get far enough to actually respond to the deletion request.
This is quite a problem, as it looks like we'll have to recreate the broker, which is a pain. I've read a lot about the 'phantom execution groups' on these forums, which seem to be able to be solved by manipluating the broker repository database, but since this is MB 7, this isn't an option any more.
We're raising a PMR for this - it's quite a bad issue - It's easy enough to reproduce (one typo in the JVM parameters, or even setting the debug port to something < 1024, etc) and it looks like it will always force a Broker instance rebuild.
Our main concern is how to avoid this in the future - we had thought that a mqsibackupbroker would be enough. But examining the ZIP file that this command creates - I can't see the actual broker command lines anywhere in the file (at least in ASCII, using grep throughout the file contents). My question is, is it really contained in the mqsibackupbroker output... or is this information stored somewhere else in MB 7? Being a skeptic in general, if I don't see it with my own eyes, I have a tendency to doubt that it's really there.  Is there anyone who can confirm that the JVM parameters for the EGs are really stored in the backup file?
In any case, will post the results of the PMR as the recommendations come in from IBM support. In any case we will soon find out if our backup/restore is working
Best Regards,
Ben Siggers. |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Wed Aug 01, 2012 10:33 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Quote: |
| We're raising a PMR for this - it's quite a bad issue - It's easy enough to reproduce (one typo in the JVM parameters, or even setting the debug port to something < 1024, etc) and it looks like it will always force a Broker instance rebuild. |
For the case where the debug port is < 1024, or any port lower than 1024, try the following:
- add root to the mqbrkrs group
- start the broker with root
- change the port to a number > 1024 using root to do it
- bounce the broker twice (using root)
- stop the broker
- remove root from mqbrkrs group
- start broker with normal (service) id
Let us know if this works.
Typically ports under 1024 require root authority or they won't work... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| bsiggers |
| Posted: Thu Aug 02, 2012 8:01 am Post subject: |
|
|
Acolyte
Joined: 09 Dec 2010
Posts: 53
Location: Vancouver, BC
|
Hello, thanks for the response - unforutnately the error we actually had was leaving out a '-' in one of the '-X' JVM arguments. That's good advice about the ports < 1024 though, as that would solve that particular issue.
We've even considered temporarily replacing the java executable JVM packaged with MQSI with a script file that would corect our typo and then run the real java executable with the corrected arguments. Was a brave idea which didn't work
Anyone have any thoughts about whether mqsibackupbroker includes the JVM arguments for the execution groups - or is this somewhere else? Am somewhat resigned to rebuilding the broker instance this time, but ideally if I can say with certainty that mqsibackupbroker includes this information, we can update our change process to include this step before touching any execution group parameters, and then move on. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Aug 02, 2012 8:05 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
There is usually no good reason to change the JVM defaults. WMB 7.0.0.3 runs exceptionally well out-of-the-box like a well-oiled machine. If you tweak the JVM parameters with a shotgun, your likely to shoot yourself in the foot. Hence, the problem you just ran into.
Let WMB manage its own JVM. I was in the USAF, we had signs posted on the flightline: "Keep out. Deadly force authorized." The Generals didn't want people tweaking their airplanes. Neither would you want people tweaking your WMB JVM.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Aug 02, 2012 8:08 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
What I would expect is that mqsibackupbroker's output contains the runtime configuration of the EG, which would include it's jvm options.
You may be able to find the offending parameter in the .dat files in the repository folder in the EGUUID folder in the broker's workpath, and manually edit this to fix or remove the bad change. You should then be able to start the EG long enough to delete it and recreate it with an EG that does not have a manually edited config file.
Also, you should be working a PMR. |
|
| Back to top |
|
|
| bsiggers |
| Posted: Thu Aug 02, 2012 10:28 am Post subject: |
|
|
Acolyte
Joined: 09 Dec 2010
Posts: 53
Location: Vancouver, BC
|
Thanks for the feedback - we've managed to recover from the issue by restoring the repository folder from backup. So it *is* in there somewhere (even if not directly visible in ASCII format via tools like grep), which would imply that it's contained in the mqsibackupbroker contents as well.
Would suggest to anyone out there with the same issue that you should be running mqsibackupbroker on a regular basis (probably daily), and before you making any changes to any execution group properties, JVM or otherwise. Restoring the repository at the filesystem level (or even better, using the backup created by mqsibackupbroker) will resolve the issue.
Regarding avoiding changes the guts of the JVM settings, I agree as well, there's lots of options in there to tweak. The actual JVM settings that we were changing though were to enable a third-party monitoring tool to peek into the internals as a proof of concept, rather than trying to be too clever and trying to optimize garbage collection routines or something.
That being said, we also understand the risks of adding these kinds of third-party plugins to the JVM - if anything goes wrong at all this would be the first thing we'd pull out, as we wouldn't expect IBM support to be able to help much with issues in an environment where we're adding low-level plugins to the core of the JVM.
Thanks very much for your help, much appreciated. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Aug 02, 2012 10:35 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| bsiggers wrote: |
| Thanks for the feedback - we've managed to recover from the issue by restoring the repository folder from backup. So it *is* in there somewhere (even if not directly visible in ASCII format via tools like grep), which would imply that it's contained in the mqsibackupbroker contents as well. |
the dat files are xml files that contain a fair amount of base64 encoded data.
So, yeah, a bit of archaeology necessary. And only attempted if *strictly* necessary. And, again, only long enough to get you to the point of deleting and recreating the EG. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Aug 02, 2012 11:05 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| bsiggers wrote: |
| Regarding avoiding changes the guts of the JVM settings, I agree as well, there's lots of options in there to tweak. The actual JVM settings that we were changing though were to enable a third-party monitoring tool to peek into the internals as a proof of concept, rather than trying to be too clever and trying to optimize garbage collection routines or something. |
You don't need to change the command line JVM properties as these can be changed on EG startup by a JCN.
I would bet your third-party tool is using JMX - Google JMX lancelotlinc /site:mqseries.net and you will get source code that you can use to do the same thing in a JCN that you would have done on the command line.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| bsiggers |
| Posted: Fri Aug 03, 2012 7:47 am Post subject: |
|
|
Acolyte
Joined: 09 Dec 2010
Posts: 53
Location: Vancouver, BC
|
Thanks, I'll have a look at that as I wasn't aware that was possible, that would probably be possible for some of the arguments. But if I recall correctly, I think part of the JVM arg changes involved modifying the bootclasspath as well, which I'm guessing you couldn't change via a JCN as the train would have already left the station
Thanks again for your help. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Aug 03, 2012 8:02 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| bsiggers wrote: |
Thanks, I'll have a look at that as I wasn't aware that was possible, that would probably be possible for some of the arguments. But if I recall correctly, I think part of the JVM arg changes involved modifying the bootclasspath as well, which I'm guessing you couldn't change via a JCN as the train would have already left the station
Thanks again for your help. |
Don't change the boot classpath.
That runs the risk of, as you have seen, interfering with the Broker's internal classes!
Use a JavaClassloader Configurable Service. It will load suitably sooner than your JCN code such that you can rely on it to be monitored by whatever you're looking at.
As lancelotlinc suggests, you can alter the behavior of the JVM by including code in the constructor for your JCN. Unlike him, I'm not going to strongly recommend this option. You don't have any control over the relative order of constructors for different JCNs, so you run the risk of a random set of JCNs having the changes NOT applied during their startup. I.e. anything that has a constructor called before yours won't be affected.
And since the JVM is not affected by classloaders, the changes you make affect everything. So it's more reliable to change the EG properties as you did.
Provided you take due diligence in executing your commands to change the EG properties - i.e. putting them in a script, testing the script against a new EG in the dev environment, putting the script under change control, promoting the script to UAT ....
 |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Fri Aug 03, 2012 10:30 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| bsiggers wrote: |
Thanks, I'll have a look at that as I wasn't aware that was possible, that would probably be possible for some of the arguments. But if I recall correctly, I think part of the JVM arg changes involved modifying the bootclasspath as well, which I'm guessing you couldn't change via a JCN as the train would have already left the station
Thanks again for your help. |
You dont need any additional proprietary Jars to turn on JMX. Therefore, you don't need to modify the classpath. There may be a few Sun jars you want to add in your normal classpath locations such as $MQSI_WORKPATH/shared-classes.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Fri Aug 03, 2012 10:44 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| bsiggers wrote: |
| you couldn't change via a JCN as the train would have already left the station ... |
WMB runs exceptionally well out-of-the-box. There is no need to change any of the default JVM startup parms.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Aug 03, 2012 11:07 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| lancelotlinc wrote: |
WMB runs exceptionally well out-of-the-box. There is no need to change any of the default JVM startup parms. |
WMB runs exceptionally well out-of-the-box. In most cases there is no need to change any of the default JVM startup parms.
There fixed it for you...
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Fri Aug 03, 2012 11:09 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Aug 03, 2012 11:17 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| lancelotlinc wrote: |
| I can always count on you davies... I'll buy you a pint and some fish & chips. |
It will have for be a pint of Harveys (Lewes), OTT(Hog's Back) or 6x out of the Wood though. I'm afraid it won't be cooled to within a degree of 0C.
(My local Chippy is one of the top 10 in the UK so you are welcome anytime)
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
|
|
