| Author |
Message
|
| lancelotlinc |
 Posted: Fri Jun 08, 2012 1:14 pm Post subject: Steps for Automating your Toolkit environment |
 |
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
Steps for Automating your Toolkit environment
1. Setup your directory structure thusly:
C:\apps\ccviews -- your main source code control directory
C:\apps\ws -- your main Eclipse workspace directory
C:\apps\apache -- your collection of Apache and other open source tools
2. In C:\apps\ccviews, you should obtain source code streams from your source code control system in separate directories for each code stream, for example C:\apps\ccviews\MessageFlows_12_02_Maint\Services\MessageFlows_src (second release of 2012), C:\apps\ccviews\MessageFlows_12_03_Maint\Services\MessageFlows_src (third release of 2012), C:\apps\ccviews\MessageFlows_12_04_Maint\Services\MessageFlows_src (fourth release of 2012). This is your source code working directory. Source files you edit from within Toolkit will be edited here.
3. In C:\apps\ws, use your toolkit to create separate workspaces corresponding to each code stream. C:\apps\ws\MessageFlows_12_02_Maint, C:\apps\ws\MessageFlows_12_03_Maint, C:\apps\ws\MessageFlows_12_04_Maint
4. Open each workspace in toolkit, and import the projects from the corresponding source code stream.
5. Download the needed Apache projects into your Apache directory: C:\apps\apache\apache-ant-1.8.3, C:\apps\apache\axis2-1.6.1, C:\apps\apache\db-derby-10.8.2.2-bin, C:\apps\apache\jenkins, C:\apps\apache\jre6, C:\apps\apache\junit
6. Create directories to receive builds from your toolkit. C:\apps\Auto_Build\MessageFlows_12_02_Maint\CurrentBuild, C:\apps\Auto_Build\MessageFlows_12_03_Maint\CurrentBuild, C:\apps\Auto_Build\MessageFlows_12_04_Maint\CurrentBuild
7. Create a ccviews directory in jenkins. C:\apps\apache\jenkins\ccviews
8. In C:\apps\apache\jenkins\ccviews, you should obtain source code streams from your source code control system in separate directories for each code stream, for example C:\apps\apache\jenkins\ccviews\MessageFlows_12_02_Maint\Services\MessageFlows_src, C:\apps\apache\jenkins\ccviews\MessageFlows_12_03_Maint\Services\MessageFlows_src, C:\apps\apache\jenkins\ccviews\MessageFlows_12_04_Maint\Services\MessageFlows_src. This is your Jenkins_Build source code area. BAR files get automatically built and deployed from here.
9. In your message flow project, create a 'devbuilder' directory to hold your Ant scripts and properties files: C:\apps\ccviews\MessageFlows_12_03_Maint\Services\MessageFlows_src\MyMessageFlowProjectName\devbuilder
10. In 'devbuilder', create a properties-driven Ant file. The Ant file should have separate targets for building your base Bar and overriding that Bar for a particular DEV machine.
11. In 'devbuilder', create a directory called jenkins. Then jenkins/jobs. C:\apps\ccviews\MessageFlows_12_03_Maint\Services\MessageFlows_src\MyMessageFlowProjectName\devbuilder\jenkins\jobs
12. Create directories to receive builds from jenkins. C:\apps\apache\jenkins\Auto_Build\MessageFlows_12_02_Maint\CurrentBuild, C:\apps\apache\jenkins\Auto_Build\MessageFlows_12_03_Maint\CurrentBuild, C:\apps\apache\jenkins\Auto_Build\MessageFlows_12_04_Maint\CurrentBuild
13. In devbuilder/jenkins/jobs, put one jenkins script for each Bar. Each script should have four steps: [1] remove the old bar from the jenkins/Auto_Build, [2] invoke the command line from your source code control system that updates your source code file in your jenkins source code stream directory, [3] invokes the Ant script that builds and overrides your Bar, and [4] invokes mqsideploy to move the newly created Bar to the Message Broker instance of your choice. Each Bar should be built on one hour intervals spaced no closer to the previous Bar by 5 minutes.
14. Create jenkins workspace and import projects into those workspaces. C:\apps\apache\jenkins\ws\MessageFlows_12_02_Maint
Whaa-laa. You now have a license-fee-free Continuous Integration DEV environment. Why do we want one?
Continuous Integration implements a continuous process of applying quality control, small pieces of effort, applied frequently. Continuous integration improves the quality of software, increases team veloicy and reduces the time taken to deliver the finished product. It is important that developers shorten the time that source code files are checked out because the longer code remains checked out, the greater the risk of multiple integration conflicts and failures becomes when the changed code is reintegrated into the main code line. The idea is to check out source code files, make some changes, then check them in within a short period of time. The automatic build process will build and deploy your bar files to DEV environment, then run some basic regression tests. Any conflicts or breakage will immediately be known and reported to the development team by email. When unit tests fail or a bug emerges, developers might revert the codebase to a bug-free state, without debugging in the shared DEV environment. Developers detect and fix integration problems continuously, avoiding last-minute chaos at release dates, when everyone tries to check in their slightly incompatible versions. Early warning of broken/incompatible code, conflicting changes and immediate unit testing of all changes improves team velocity. Quality is improved by immediate feedback to developers on the functionality and system-wide impact of code. Frequent code check-in encourages developers to create modular, less complex code.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Fri Jun 08, 2012 9:33 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
sorry to disagree with you.
Unless you do what you have said AND then take the time (if your PHB will let you) to learn 'ant' and jekins nee hudson you might have a continuious integration env in principle but in practice? No way. 
Half the story is at least better then none... 
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Mon Jun 11, 2012 5:29 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| smdavies99 wrote: |
sorry to disagree with you.
Unless you do what you have said AND then take the time (if your PHB will let you) to learn 'ant' and jekins nee hudson you might have a continuious integration env in principle but in practice? No way.
Half the story is at least better then none... |
Not sure I understand why you think it is time consuming or difficult for either Ant or Jenkins.
http://www.ibm.com/developerworks/websphere/library/techarticles/0706_spriet/0706_spriet.html
Might take a good developer half a day to learn how to use Ant. As for Jenkins, from ground zero, I had four bars being built within an hour. Is there something about these two technology examples that is hard to understand?
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Jun 11, 2012 6:29 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Well yes there is.
After spending 30+ hours trying to get a complete system to build (100+ bars) and the build times for each bar getting longer an longer I gave up with ant and mqsicreatebar.
It might be that these problems are now resolved but in my current positon we have at most 20 bar files per system and we are not operating in a CI environment. A little difficult when the target is up to 6000 miles away and you only have very limited access via RDP.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Mon Jun 11, 2012 7:09 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| smdavies99 wrote: |
Well yes there is.
After spending 30+ hours trying to get a complete system to build (100+ bars) and the build times for each bar getting longer an longer I gave up with ant and mqsicreatebar.
It might be that these problems are now resolved but in my current positon we have at most 20 bar files per system and we are not operating in a CI environment. A little difficult when the target is up to 6000 miles away and you only have very limited access via RDP. |
All the more reason why you need a standard, repeatable way to build bars. A defined build process ensures that the software in your development project is built in the exact same manner each time a build is executed. If you don't like Ant, you could try other make programs.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Mon Jun 11, 2012 7:19 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| smdavies99 wrote: |
| each bar getting longer an longer |
When you have a Continuous Integration program like Jenkins, you don't care how long a build takes. The build could take hours, and you would never know it because the build happens behind the scenes. If a build fails, Jenkins will send you an email.
With WMB 7.0.0.3, our build times were cut in half. Our bars now build in two minutes rather than five. We segment our architecture so we don't build huge monolithic bars. Maybe you could look into modularizing your message flows?
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Jun 11, 2012 7:20 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Most of the difficulties people have with automating mqsicreatebar are related to what and how they check things out from source control into a workspace. It doesn't entirely help that mqsicreatebar got a lot more strict about workspaces as of v7.
The other subtlety that tends to crop up is knowing when to use
and when to use
.
There's a reasonably good devworks article on how to use Ant to automate bulid and deploy of Broker stuff.
All that said, having a CI system that automatically builds and deploys every hour is of only limited use if the rest of the process around development is lacking in similar infrastructure. For example, no rules on when to check things in, no automated TEST procedures, no rules on naming standards, or worst of all no accountability for what IS checked in by whom and when.
CI is great if the corporate culture supports it. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Mon Jun 11, 2012 7:30 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
Here is an example Ant exec command that drives mqsicreatebar. Note all parms are sourced from a properties file.
| Code: |
<exec executable="${toolkit.home}/mqsicreatebar.exe" spawn="false" >
<arg value="-data" />
<arg value="${workspaces.dir}" />
<arg value="-b" />
<arg value="${bar2.name}" />
<arg value="-p" />
<arg value="MyProjectName" />
<arg value="${_mset_project_f}" />
<arg value="${_mset_project_g}" />
<arg value="${_mset_project_h}" />
<arg value="${_mset_project_l}" />
<arg value="-o" />
<arg value="${target_mflow_path}/${target_mflow_2_a}" />
<arg value="${target_mflow_path}/${target_mflow_2_b}" />
<arg value="${target_mset_path_f}/${target_mset_2_f}" />
<arg value="${target_mset_path_g}/${target_mset_2_g}" />
<arg value="${target_mset_path_h}/${target_mset_2_h}" />
<arg value="${target_mset_path_l}/${target_mset_2_l}" />
</exec> |
All the pseudonames (ie. target_mset_2_h) are standardized, so only certain ones are included in a particular bar.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Jun 11, 2012 1:03 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
IMHO build times are important in a CI setup.
We gave up when the builds stated to take in excess of 3 hours per bar and the system stated to hang. Admittedly this was with V6.1.0.3 so I know that things have changed since then BUT the project interdependencies were just crazy which clearly didn't help.
CI isn't for everyone or everysite. In a perfect world it might be but it isn't.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Jun 11, 2012 1:09 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| lancelotlinc wrote: |
| Code: |
<arg value="${_mset_project_f}" />
<arg value="${_mset_project_g}" />
<arg value="${_mset_project_h}" />
<arg value="${_mset_project_l}" /> |
|
If you used
then you would be able to use a single
| Code: |
| <arg line=${_mset_project_list}" /> |
and put all of the projects into a single property, rather than each in their own properties that then requires a modified script for each build.
Last edited by mqjeff on Mon Jun 11, 2012 1:28 pm; edited 1 time in total |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Mon Jun 11, 2012 1:12 pm Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| smdavies99 wrote: |
| in excess of 3 hours per bar |
I consider this a product defect. If it were my bar, I would open a PMR. Build times should be measured in minutes not hours. Per the steps above, my Bar#1 has 22 project dependencies (18 message sets, four Java projects), ClearCase source code updates in 23 seconds, bar builds in 77 seconds, and mqsideploy in 49 seconds, for a total task time of 149 seconds or 2.49 minutes.
| smdavies99 wrote: |
| CI isn't for everyone or everysite. In a perfect world it might be but it isn't. |
A defined build process ensures that the software in your development project is built in the exact same manner each time a build is executed. If you go on vacation, who builds your bars?
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Jun 11, 2012 10:41 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| lancelotlinc wrote: |
| If you go on vacation, who builds your bars? |
If a change is needed to any of my code whilst I'm on holiday any of my team can checkout the projects from SVN and use the TK to build the bars for testing.
Typically our release/refresh cycle is 1-2 years. Answers on a postcard as to how much CI will save us here.
In our list of priorities, CI is almost at the bottom. Getting or generating relevant test data is far more critical to us espcially as one set of data in not transferrable between systems.
So we have a choice... Better and more accurate test data or a CI System?
I know which one I'd prefer.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Tue Jun 12, 2012 12:45 am Post subject: Re: Steps for Automating your Toolkit environment |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| lancelotlinc wrote: |
Steps for Automating your Toolkit environment
...
You now have a license-fee-free Continuous Integration DEV environment. |
Automating builds is not the same thing as Continuous Integration.
These seem like very site-specific instructions. Different sites may require a completely different set of instructions to accomplish similar goals, for too many reasons to list here.
| lancelotlinc wrote: |
| Whaa-laa. |
Really? |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Tue Jun 12, 2012 5:01 am Post subject: Re: Steps for Automating your Toolkit environment |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| rekarm01 wrote: |
| lancelotlinc wrote: |
Steps for Automating your Toolkit environment
...
You now have a license-fee-free Continuous Integration DEV environment. |
Automating builds is not the same thing as Continuous Integration.
|
No doubt. Continuous Integration takes changes from many developers and builds the deployable artifacts often and regularly to see if any changes broke the build.
| rekarm01 wrote: |
| lancelotlinc wrote: |
| Whaa-laa. |
Really? |
The point of the post was to demonstrate how simple and easy it is. In my previous editorials on similar lines, many people complain that its too complex, too time consuming, and requires an act of Congress or Parliament. In less than a day, with license-fee-free tooling, I was able to fully automate the build and implement an elementary CI platform. Fourteen steps to freedom.
YMMV.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Tue Jun 12, 2012 5:42 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| smdavies99 wrote: |
| lancelotlinc wrote: |
| If you go on vacation, who builds your bars? |
If a change is needed to any of my code ... Answers on a postcard.
|
Good idea. If I send you my address, please send me a postcard from your vaca spot. I'll return the favor from Mactan Island, Cebu, Philippines.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
|
|
