| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| Any successes with automated builds using mqsicreatebar? |
« View previous topic :: View next topic » |
| Author |
Message
|
| skol |
 Posted: Wed Apr 12, 2006 11:38 am Post subject: Any successes with automated builds using mqsicreatebar? |
 |
|
Apprentice
Joined: 05 Apr 2006
Posts: 38
|
Has anyone been consistently successful building bars outside of the toolkit with a script that uses mqsicreatebar? We have 5 different deployment environments we use (3 kinds of tests, one QA, one prod), and want to guarantee that a BAR is built properly for a given environment, matches what is in source control, etc., all with no manual intervention by a developer. The way we do that is with an ANT script that checks projects out of source control, runs mqsicreatebar to build a bar, then unzips the bar, applies an XSLT to tailor deploy.xml in the bar with the property overrides for each flow (queue names, dataset names, etc) to the target environment, and finally zips it back up with a name specific to the target environment.
This all works fine on some projects, but does not work in others, and the failures are always in the mqsicreatebar step. We've had some issues with mqsicreatebar that we have found workarounds for, but have hit one that we can't work around. The current failure centers around ESQL, with an error like this one from the ANT exec task that is running mqsicreatebar:
[exec] ERROR: The reference to ESQL module 'esql://routine/#AssetPricing2_C
ompute.Main' with function 'Main' cannot be resolved. Ensure the ESQL module exi
sts in the correct schema, and that it contains function 'Main', and try again.
[exec] BIP0988W Error occurred while completing mqsicreatebar command.
In this case, if I point the toolkit at the same source, it has no trouble resolving the ESQL and building a bar. I only get this error when using mqsicreatebar separate from the toolkit in our automated builds.
All of our projects have flows, and just about all the flows have ESQL. Most of them build fine with mqsicreatebar, others show an error like the one above. If anyone has more info about how mqsicreatebar locates ESQL and what could be causing this error, I'd sure appreciate it.
I'd also be interested in knowing if many others have a desire for this type of automated build processing, and if others are already successfully doing it.
Thanks in advance for any info...
_________________
Steve |
|
| Back to top |
|
 |
| coding_monkey |
| Posted: Thu Apr 13, 2006 4:36 am Post subject: |
|
|
Newbie
Joined: 29 Mar 2006
Posts: 6
|
Hi skol,
We use just this kind of build process and successfully build BARS using mqsicreatebar. We are using MB v6.
The main problem we found was due to an assumption made by the mqsicreatebar tool, that all message flow projects are physically located within the workspace folder in the file system. In our environment our physical workspace is separate from the actual projects:
| Code: |
c:\Workspace
|_ .metadata
c:\Projects
|_ project1
|_ .project
|_ project1.msgflow
|_ etc
|_ project2
|_ .project
|_ project2.msgflow
|_ etc
|_ etc
|
This is so that we can check projects into CVS and not workspaces; effectively allowing people to set their workspace up how they wish.
Now, while Toolkit can cope with this and builds the projects correctly, the mqsicreatebar tool does not. I seems to expect a structure like:
| Code: |
c:\Workspace
|_ .metadata
|_ etc
|_ project1
|_ .project
|_ project1.msgflow
|_ etc
|_ project2
|_ .project
|_ project2.msgflow
|_ etc
|_ etc
|
In order to accomodate this strange requirement I have configured my ANT scripts to perform the CVS checkout and then copy the message flow projects into a BuildWorkspace directory as shown above. Then it runs the mqsicreatebar tool on the complete structure.
Now this introduced some additional problems. In order for the mqsicreatebar tool to run, the project metadata in the workspace has to be correct and show that all the projects are referenced properly and can compile etc. Well, if the messageflows are dynamically copied into the BuildWorkspace directory the metadata needs to somehow be refreshed in order to be correct. I am not aware of a command tool to refresh the workspace so this has to be done through toolkit (effectively adding a manual step to the build process!). So, the ANT script copies the projects into a BuildWorkspace and then stops. I then fire up toolkit and refresh/rebuild the projects.
Once the copied projects are showing without the ubiquitous red X the mqsicreatebar tool can be run and the build process can continue as normal.
While this is really bad I find it is only a problem while projects are changing a lot. As soon as you are into regular builds and minor flow changes it is less of an issues because the project metadata tends to stay the same during that phase of a project.
Now, this all may be completely different from what you are seeing but it is certainly the only issue we have had with the mqsicreatebar tool.
For reference the ANT target we use is as follows:
| Code: |
<target name="parameterised.buildbar">
<echo message="Building BAR file - ${wbimb.project.name}.bar for ${ant.project.name}"/>
<exec executable="${toolkit.home}\eclipse\mqsicreatebar.exe" spawn="false">
<arg value="-data"/>
<arg value="${workspaces.dir}"/>
<arg value="-b"/>
<arg value="${workspaces.dir}/bars/${wbimb.project.name}.bar"/>
<arg value="-o"/>
<arg value="${param.msgset.or.flow.01}"/>
<arg value="${param.msgset.or.flow.02}"/>
</exec>
<echo message="Completed building BAR file - ${wbimb.project.name}.bar for ${ant.project.name}"/>
</target>
|
Cheers
_________________
- scott |
|
| Back to top |
|
|
| coding_monkey |
| Posted: Thu Apr 13, 2006 4:47 am Post subject: |
|
|
Newbie
Joined: 29 Mar 2006
Posts: 6
|
Hi skol,
In terms of the error you are seeing:
| Quote: |
[exec] ERROR: The reference to ESQL module 'esql://routine/#AssetPricing2_C
ompute.Main' with function 'Main' cannot be resolved. Ensure the ESQL module exi
sts in the correct schema, and that it contains function 'Main', and try again.
[exec] BIP0988W Error occurred while completing mqsicreatebar command. |
In MB v6 we see this all the time both in Toolkit and our ANT build process. Often it is fixed by simply clean all/rebuilding the projects in Toolkit which points to Toolkit getting it's bits in a twist! It is very frustrating as clearly the function does exist in the correct schema.
Does your ANT script work if you first use Toolkit to compile the flows then run the script? If so, in your example it could just be that when you run the ANT script the workspace metadata is a bit screwed so using Toolkit to refresh it then running the script with the good metadata works? If not, then I am not sure, sorry.
Cheers
Scott
_________________
- scott |
|
| Back to top |
|
|
| skol |
| Posted: Thu Apr 13, 2006 8:05 am Post subject: |
|
|
Apprentice
Joined: 05 Apr 2006
Posts: 38
|
Thanks, this is very interesting info. I haven't tried breaking the flows up into the project structure you show in your first reply. I will definitely try this and see if it helps my ESQL problem. Thanks you for sharing your experience here.
Whatever is happening to me only happens with mqsicreatebar. The toolkit always works. What I see is consistent with your point of there being something different about how mqsicreatebar is dealing with the project(s) compared to the toolkit. I have done this test, which to me proves I'm battling some nuance of mqsicreatebar:
1) Do the checkout from source control into a fresh directory
2) Run the ant script, which does some prep steps then runs mqsicreatebar
3) mqsicreatebar fails with an error on the ESQL reference
4) Open up the toolkit, make a fresh empty workspace, import the above projects, and make a bar - it all works fine in the toolkit, proving that the source and the .project files are not bad.
5) Attempt to rerun mqsicreatebar against the same source, and it still fails with the same error.
To your point of requiring a manual step here, I completely understand. We were trying to avoid any manual steps in this process, just as it sounds like you were. Based on your info, I may punt on full automation after trying your suggestions, if I still can't get something working that I can use on all our projects.
The wierd thing is I have mutiple projects that I use the same process on, and they work fine! It's only certain projects that get this error, and I don't understand enough about how mqsicreatebar works to diagnose what's different about these.
In case anyone from IBM is reading this, I would really like a good automated build process to be available in the future with broker deployments. For other deployments (java, .Net, C++, VB6 etc) automated builds are the only way we have found that can we can reliably guarantee that what is in source control exactly matches what is deployed. This is true because in all those environments there are no dependencies on the contents of somebody's toolkit workspace that is outside of source control. I would be more than happy to offer input on why this is important to us, and details on the current issues that are making this sort of automation difficult, to anyone interested.
Thanks again for your posts, I appreciate the time it took to reply in such detail.
_________________
Steve |
|
| Back to top |
|
|
| x061294 |
| Posted: Thu Apr 13, 2006 8:19 am Post subject: |
|
|
Acolyte
Joined: 05 Apr 2005
Posts: 62
|
Have either of you, or any others, encountered a problem with the mqsicreatebar in the amount of time it takes to run? If so, did you resolve it?
I can take a very simple flow/project and create a bar through the toolkit in just a second or two. However, if I try and create the bar through mqsicreatebar the process seems to hang for 10 - 30 seconds, then executes. In a previous post someone mentioned that the mqsicreatebar seems to need something from the eclipse framework, and that's almost what it appears to be, that the command is searching the directory structure, finally finds whatever it was looking for and then executes the command. While through the toolkit the mystery file/setting/whatever is already available so it executes right away.
So, in your ant processing did you find anything like this and have to work around it? |
|
| Back to top |
|
|
| skol |
| Posted: Thu Apr 13, 2006 8:44 am Post subject: |
|
|
Apprentice
Joined: 05 Apr 2006
Posts: 38
|
Hmm, interesting question. mqsicreatebar always is CPU intensive when we run it, but I haven't seen hangs. It typically just goes to 100% CPU quickly, stays there for 10 or 15 seconds, and then finishes. I'd be real interested in hearing more about the sort of searching it might be doing - if my runs of mqsicreatebar are looking elsewhere for stuff other than what I've given it on the command line, it would possibly explain some of the difficulties I've been having with it.
I always do our builds with locally-stored flows and message sets. When you have the hangs, is it possible mqsicreatebar is accessing files on network-based resources and having trouble?
I wish mqsicreatebar wasn't this much of a "black box" where we have to deduce what it needs to run successfully by prodding it different ways. 
_________________
Steve |
|
| Back to top |
|
|
| aseelemann |
| Posted: Mon Apr 17, 2006 9:15 am Post subject: |
|
|
Newbie
Joined: 17 Apr 2006
Posts: 2
|
Mqsicreatebar is actually Eclipse running the bar editor in "headless" mode. One reason Mqsicreatebar takes lots of CPU is because it needs to start a largish portion of Eclipse when it starts up.
Since Mqsicreatebar is another way of running Eclipse, the usual rule about one Eclipse per workspace applies. So you can't run Mqsicreatebar and the Eclipse GUI on the same workspace at the same time.
When it starts, Mqsicreatebar tries to open or add the listed projects to the workspace. A side effect of this is that Mqsicreatebar can create a new workspace with projects, if the project directories are in the workspace directory.
If the project directories are not in the Eclipse workspace directory, Mqsicreatebar will not be able to find them. In this scenario, I would suggest using the GUI to add the projects to the workspace once, and then shutdown the GUI and use Mqsicreatebar until the projects change. You can repeatedly delete and overwrite the projects from your repository. If you need to add or delete projects from the workspace, please use the GUI. If this is too limiting, I suggest opening a PMR to request that Mqsicreatebar be enabled to support adding project directories that are not in the workspace directory. In addition to (workspace relative) project names, the command should accept absolute paths.
Mqsicreatebar will perform an Eclipse workspace "incremental" build when it starts to ensure the most recent workspace content is included, including edits outside of Eclipse. This is the same build that happens when "automatic" project builds are enabled. Unless there are many changes, this will complete in a few seconds. "Coding_monkey" suggested starting the GUI to do this, however starting the GUI shouldn't be necessary and in fact runs the exact same code. I appreciate the suggestion may be to work around the "Missing ESQL" issue. I don't have any recommendation on how to work-around this as I have never observed it myself.
The brokers toolkit caches a great deal of information to speed validation and bar creation. All ESQL (hand-coded and Mapper generated) is cached by the toolkit in a format that speeds bar creation. This includes the "static linker" information that enables the bar tools to only include ESQL that is actually used.
If, after the build, there are any Brokers Toolkit specific errors in the Eclipse "Problems" list, Mqsicreatebar will abort with a non zero result code (usually -1), and print the problems messages to the console. The ESQL error message is an example of this behaviour.
Otherwise, Mqsicreatebar terminates with a result code of 0, indicating success.
Sometimes, other Eclipse components print error messages to the console. Oftentimes these aren't part of the brokers toolkit, for example messages may be printed by Eclipse itself while updating internal metadata, or by Rational Application Developer or (if installed) by WebSphere Integration developer. While alarming, these messages can be ignored -- rely on the result code returned by Mqsicreatebar to determine whether the command completed successfully.
Finally, a question: I was pointed to this forum append by SHanson, but have any of you followed up with IBM Service? I don't recall seeing this "Missing ESQL" issue on our support queues, but the problem might require a code change to correct. The only way to get that is through service. |
|
| Back to top |
|
|
| skol |
| Posted: Mon Apr 17, 2006 9:53 am Post subject: |
|
|
Apprentice
Joined: 05 Apr 2006
Posts: 38
|
Thank you, this is very useful information, and explains alot of what I have been seeing as behavior. My intent is to develop an automated build and deploy script that does not share a workspace with any manual developer work. It should be able to extract source from source control, run the build, and create a bar with no manual intervention. In the test scripts I've been running, it checks out source from source control, without any of the .metada from a workspace. Just the .project files, and the flows and message set projects, all in the proper directory structure. When mqsicreatebar runs on this source, I have seen it generate an entire .metadata directory tree, as if it is trying to make a workspace out of the downloaded source, in the directory that holds the source. It then finishes, and on many projects finishes with no errors, and a valid BAR is built. On some projects, I get this ESQL failure instead, mqsicreatebar fails with an error, and an incomplete bar is built. I have ended up relying on this capability of mqsicreatebar to "fake up" a workspace as part of the automated build, but am now wondering if that's valid.
With this automated build goal in mind, builds of various bars would be done on the same build machine. The build machine will not typically be used by any developers. Developers will have their own machines and workspaces, with their own checked-out source. Based on your very informative response, I have some questions.
1) Is it reasonable for me to expect mqsicreatebar to create a workspace on the fly when pointed to source that does not have a workspace, but does have the .project files from the various projects (flow projects and message set projects)? Or have I just been lucky to get as far as I have with this?
2) If mqsicreatebar is using any other workspace data, how does it decide where to get it from? Is it using the eclipse startup config that tells the GUI which workspace to use at startup?
3) If the answer to #1 is "no", and I need to have a full workspace to get successful automated builds, is it sufficient to check into source control the contents of the entire .metadata directory, or is there more stuff that makes up a "workspace"? I'm really hoping there's some way to avoid source controlling the workspace .metadata, as that raises multiple problems.
As far as opening a service request goes, I've been suspicious up to now that I've just been doing something wrong while getting this to work, as I'm new to the broker. But I think I'm almost far enough in my understanding now that opening one sounds like the thing to do, depending on the answers to the above questions.
I appreciate your info as that has filled in some big gaps in my understanding of mqsicreatebar. What I'm not sure is what to ask for in a service request with support. The goal is a fully automated build, using only what comes from source control, run on a machine that is not normally used by a developer.
If the answer to #1 is yes, and I can prevent mqsicreatebar from using any other workspace data, then the only remaining problem with an automated, unattended build is this ESQL reference issue that happens on some source. I can easily package up a sample build, with our ANT script, for IBM to see and test with. That would be a straightforward service request. If the answer to #1 is no, then the proposed service request is less straightforward.
Given all this, I'd be very interested both in your answers, and in your opinion on how I should proceed (service request or whatever). Thanks again...
_________________
Steve |
|
| Back to top |
|
|
| aseelemann |
| Posted: Mon Apr 17, 2006 10:17 am Post subject: |
|
|
Newbie
Joined: 17 Apr 2006
Posts: 2
|
Hi Skol, I'll try to answer your questions. One thing I realise is I don't know which version of brokers you are using, 5.0, 5.1, or 6.0, and what service level you are at.
1) Its 100% reasonable to expect Mqsicreatebar to create a workspace on the fly. So long as the "project" parameters all have a .project file this is fully supported. "Supported" in this context means if it doesn't work IBM will help you get it working.
2) Hmmm version 5 and 6 are different for this one. Version 5 Mqsicreatebar will always use the "default" workspace, unless you specify -data. Version 6 requires you to always specify the workspace location using the -data parameter -- there is no longer a "default" workspace.
In either case, all workspace data comes from the workspace .
3) The answer was "yes". What you describe as an alternative is not supported. The rule is all workspaces must be created by the Eclipse toolkit "in place"; workspaces may not be copied or moved around. Copying and moving may work (sometimes) but is not a supported scenario. In this context, "not supported" means IBM will decline a service request as its not a supported scenario. You milage may vary 
I agree with your conclusion that you need to open a service request for the ESQL problem. |
|
| Back to top |
|
|
| skol |
| Posted: Mon Apr 17, 2006 11:52 am Post subject: |
|
|
Apprentice
Joined: 05 Apr 2006
Posts: 38
|
We are 6.0.1 on the toolkit. Part of the ANT build script after mqsicreatebar depends on the ability to XSLT the meta-inf/broker.xml file in the built bar to change the promoted property values to match the environment we're building for, so 6.0.1 was a prereq.
I'll open an incident and see if we can work through the ESQL error, since it sounds like I'm hopefully not abusing mqsicreatebar.
Thanks again!
_________________
Steve |
|
| Back to top |
|
|
| Michael Dag |
| Posted: Tue Dec 19, 2006 2:17 pm Post subject: |
|
|
Jedi Knight
Joined: 13 Jun 2002
Posts: 2607
Location: The Netherlands (Amsterdam)
|
| skol wrote: |
| Part of the ANT build script after mqsicreatebar depends on the ability to XSLT the meta-inf/broker.xml file in the built bar to change the promoted property values to match the environment we're building for |
Hi Steve,
do you have a sample of the scripts and xslt for the meta-inf/broker.xml to change the promoted properties?
Thanks.
_________________
Michael
MQSystems Facebook page |
|
| Back to top |
|
|
| rajasri |
| Posted: Wed Jan 24, 2007 6:47 am Post subject: in need of XSLT |
|
|
Centurion
Joined: 11 Jun 2006
Posts: 114
|
| Hi Skol, i have built ant scripts for cheking out files and creating bar files. iam need of XSLT through which we can change the promoted properties. would you please share them. expectiong a positive response. thank you |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Jan 24, 2007 7:05 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
With the 6.0.2.0 Release of the Toolkit, there is an mqsiapplybaroverrides command.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Fred |
| Posted: Tue Feb 20, 2007 12:41 pm Post subject: |
|
|
Newbie
Joined: 14 Jul 2004
Posts: 9
|
Is there a projected release date for the 6.0.2.0 Toolkit?
And do you have any more details regarding the mqsiapplybaroverrides command. All I really need to do at this point is change the additional instances property for a msgflow. I would much prefer to do this via a supported method rather than performing XSLT transforms outside of the product. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Feb 20, 2007 3:01 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
We managed to handle that nightmare a little bit differently:
It is the responsability of the developer to tag his code. It is the responsability of the developer to tag the corresponding bar file.
So now we have source and deployables tagged.
The deployable is build from within the Toolkit.
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
|
  |
Goto page 1, 2, 3 Next |
Page 1 of 3 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
