| Author |
Message
|
| Amitha |
 Posted: Wed Mar 02, 2011 3:09 pm Post subject: Automating Builds using mqsicreatebar |
 |
|
Voyager
Joined: 20 Nov 2009
Posts: 80
Location: Newyork
|
Hi,
I have an requirement to automate our build process, we plan to use ANT. We are using PVCS for code repository. The build machine is Windows XP.
Currently, I am able to build the complete bar file using Ant.This is good for initial deployment.
Now,the question is how to handle change request. In the past when we built the bar file using toolkit, if there are 5 flows in the bar and only 1 flow(mf1) changed,we just unchecked the other 4 flows from the bar, and choose not to "Remove contents of Broker Archive Before building" option on toolkit, hence this created new bar with 5 flows, where 4 flows were untouched and 1 flow(mf1) was modified.
But to do this using mqsicreatebar is not possible because this doesn't provide option to modify existing bar, it always builds the bar from scratch.
Experts, Please let me know how you handle change requests.Do you build complete bar with all latest flows coming from repository? Or you just modify existing bar file, modifying only flows which changed? or build new temporary bar with only the artifacts which change? |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Wed Mar 02, 2011 3:40 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
What is the end result of these two scenarios: - Deploy a bar file that has all the flows in it, but only one has been updated
- Deploy a bar file with just the changed flow in it
|
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Mar 03, 2011 4:56 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
The end state of your build automation should be a Continuous Integration process. This CI process should build from scratch every time, no skipping any flows.
Look at CruiseControl or Hudson to implement the CI process.
Automating a mqsicreatbar batch file is not very useful unless you pair it with automated source code refresh and automated deployment through the CI process.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| Amitha |
| Posted: Thu Mar 03, 2011 6:53 am Post subject: |
|
|
Voyager
Joined: 20 Nov 2009
Posts: 80
Location: Newyork
|
| Quote: |
What is the end result of these two scenarios:
1. Deploy a bar file that has all the flows in it, but only one has been updated
|
With this, it is easier to track what was deployed previously to production, and in case of fail back to older code, just take the complete bar and deploy it.
Also, it is easier for build automation, in my scenario I use properties file with ant script which says what items go into this bar and just builds the complete bar.
| Quote: |
2. Deploy a bar file with just the changed flow in it
|
Problem with this is, it is difficult to trace back what are the latest flows deployed. Since, the latest flows deployed are sprayed across multiple bar files.
Also, in my scenario since I am using properties, I will have to keep editing the properties file to add just the changed item.
| Quote: |
The end state of your build automation should be a Continuous Integration process. This CI process should build from scratch every time, no skipping any flows.
Look at CruiseControl or Hudson to implement the CI process.
|
why and what do you think are the limitations of ANT.
| Quote: |
| Automating a mqsicreatbar batch file is not very useful unless you pair it with automated source code refresh and automated deployment through the CI process. |
Sorry, I didn't understand how to do this, can you please explain. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Mar 03, 2011 7:16 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| Amitha wrote: |
| Quote: |
| The end state of your build automation should be a Continuous Integration process. This CI process should build from scratch every time, no skipping any flows. Look at CruiseControl or Hudson to implement the CI process. |
why and what do you think are the limitations of ANT. |
ANT is used by Hudson or CruiseControl or other CI products. Ant is very useful. Hudson drives the build process usually through Ant.
| Amitha wrote: |
| Quote: |
| Automating a mqsicreatbar batch file is not very useful unless you pair it with automated source code refresh and automated deployment through the CI process. |
Sorry, I didn't understand how to do this, can you please explain. |
The purpose of automation in the build and change control process is to insulate the target deployment environments from human influence unless that human touch is well documented in a change control system.
Simply wrapping an MS-DOS batch file with Ant, or driving mqsicreatebar directly with Ant, is not sufficient to accomplish the objective of build automation.
Continuous Integration (CI) provides real-time proving of your controlled configuration. Through automation, CI builds, deploys, and tests your BAR files without human input.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| Amitha |
| Posted: Thu Mar 03, 2011 9:13 am Post subject: |
|
|
Voyager
Joined: 20 Nov 2009
Posts: 80
Location: Newyork
|
lancelotlinc. thanks for your inputs. Now i understand Hudson or Cruise control can be used on top of Ant.
But, to achieve build bar, I still need to use Ant. so, you suggest to to build complete bar from scratch.
How are other folks handling their build process? Build bar from scratch or just update the bar with only the flow which changes? |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Mar 03, 2011 9:33 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| Amitha wrote: |
lancelotlinc. thanks for your inputs. Now i understand Hudson or Cruise control can be used on top of Ant.
But, to achieve build bar, I still need to use Ant. so, you suggest to to build complete bar from scratch.
How are other folks handling their build process? Build bar from scratch or just update the bar with only the flow which changes? |
Yes, because trying to incremental build fails most of the time, even if you clear out .metadata directory.
Here is the process:
1. Create a totally new directory.
2. Through your source code control client, get the relevant files to that new directory.
3. Run your build script.
Never try to reuse an old directory because the binary files there can be out of date and the resulting BAR file may bloat because it includes duplicate dependency files.
I would create an MS-DOS batch file that does the three steps above. I would not try to accomplish these steps with a human.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Thu Mar 03, 2011 10:27 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
But...
I've never understood why mqsicreatebar (or your ant or maven or whatever) operation takes so long to build the bar file.
at 15-20 minutes(Average) per bar files and with a large system you quickly get to the stage where you can't do a complete build in 24 hours.
So we've given up trying to implement Continuious Integration.
Now the developers build the .bar file with one object per file. This is timestamped and put into source control. These .bar files are then deployed to all test/uat & prod environments.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Mar 03, 2011 10:48 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| smdavies99 wrote: |
But...
I've never understood why mqsicreatebar (or your ant or maven or whatever) operation takes so long to build the bar file.
at 15-20 minutes(Average) per bar files and with a large system you quickly get to the stage where you can't do a complete build in 24 hours.
So we've given up trying to implement Continuious Integration.
Now the developers build the .bar file with one object per file. This is timestamped and put into source control. These .bar files are then deployed to all test/uat & prod environments. |
They would not take 24 hours if you didn't sequentialize them one behind the other. One of the great features of our modern PC world is the ability to parallelize tasks.
As for the mqsicreatebar utility taking too long, might be good for you to raise a PMR. I know that there is an APAR for WTX regarding this.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Thu Mar 03, 2011 11:39 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Running more than one build at a time?
Been there. done that. got the 'T' shirt.
Result
Even quicker JVM crap out. Due to inability to get more ram. The system has 4GB of RAM.
The slow mqsicreatebar has always been slow. It was slow in V5 and has IMHO not improved a lot since.
I have to admit that I groaned when we started on the CI project.
We are going to try it again on an i7-970 system with 8GB...
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Mar 03, 2011 11:46 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
|
| Back to top |
|
|
| Amitha |
| Posted: Thu Mar 03, 2011 11:56 am Post subject: |
|
|
Voyager
Joined: 20 Nov 2009
Posts: 80
Location: Newyork
|
| Quote: |
So we've given up trying to implement Continuious Integration.
Now the developers build the .bar file with one object per file. This is timestamped and put into source control. These .bar files are then deployed to all test/uat & prod environments. |
It is interesting to know that bar file contains one object. Doesn't this create overhead of having too many bar files?I kinda like this approach but I am sure if its gonna create overhead in the long run with too many bar files?
I totally agree smdavies99 mqsicreatebar is very slow. But, how do you parallelize this, because mqsicreatebar creates .metadata directory during build,isn't this specific to a build?Can you over write this file? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Mar 03, 2011 12:00 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
While I am willing to concede that mqsicreatebar may be slower than it should be, there is another point of view.
You need to make sure you know exactly what you are comparing it's speed to.
mqsibar starts up the Toolkit, creates (potentially) a new workspace, imports the relevant projects into that workspace, creates a new bar, and THEN does the same that you do when you ask the Toolkit to build a bar file.
So, yes, that does take longer than just building a bar file in the Toolkit manually.
And yes, that does take longer than just creating a brand new bar and then building that.
And as lancelotlinc says, you should keep your mqsicreatebar system patched to the appropriate levels and open PMRS if performance seems unduly slow (taking the above into account). |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Mar 03, 2011 12:01 pm Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| Amitha wrote: |
| Quote: |
So we've given up trying to implement Continuious Integration.
Now the developers build the .bar file with one object per file. This is timestamped and put into source control. These .bar files are then deployed to all test/uat & prod environments. |
It is interesting to know that bar file contains one object. Doesn't this create overhead of having too many bar files?I kinda like this approach but I am sure if its gonna create overhead in the long run with too many bar files?
I totally agree smdavies99 mqsicreatebar is very slow. But, how do you parallelize this, because mqsicreatebar creates .metadata directory during build,isn't this specific to a build?Can you over write this file? |
You have a separate .metadata directory for each build process. You don't use the same one.
BAR = JAR. JAR = ZIP. Therefore, BAR = ZIP. I've never heard of someone having too many JAR files or ZIP files. There is no such condition.
There is an art to segmenting your distribution of message flows into Execution Groups and BARs.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| j.f.sorge |
| Posted: Fri Mar 04, 2011 12:39 am Post subject: |
|
|
Master
Joined: 27 Feb 2008
Posts: 218
|
| Amitha wrote: |
| How are other folks handling their build process? Build bar from scratch or just update the bar with only the flow which changes? |
We are building our bar files (containing one flow each) from scratch. Source gets checked out from CVS and bar will be built by mqsicreatebar and checked into CVS. This bar will be deployed onto the broker after some environment-specific-changes has been done.
Build-time for one bar is around one (1) minute and can be run in four instances on my quattro-core WinXP workstation with 4GB RAM.
Automation will be done by using batch-script, Java-Code and ANT-Scripts (which do most of the tasks to do).
_________________
IBM Certified Solution Designer - WebSphere MQ V6.0
IBM Certified Solution Developer - WebSphere Message Broker V6.0
IBM Certified Solution Developer - WebSphere Message Broker V6.1
IBM Certified Solution Developer - WebSphere Message Broker V7.0 |
|
| Back to top |
|
|
|
|
