| Author |
Message
|
| craigp |
 Posted: Mon Mar 28, 2011 1:53 pm Post subject: |
 |
|
Newbie
Joined: 28 Mar 2011
Posts: 8
|
Our builds have always been automated and are now linked in with a Jenkins build server...
Under 7.0.0 there was no workspace validation and our build scripts created workspaces with no more than what the BAR needed.
Eg: Flow project A depends on project B.flows (for ESQL) and B.flows depends on B.model we didn't need to inlcude B.model into the build for a BAR containing "A".
This means that we have build times around 5 mins / BAR.
With 7.0.2 came workspace validation on mqsicreatebar ....
This means that we need to include b.model because something in B.flows
Now B.flows might also depend on C.flows and we have to include both B.model and C.model and anything else C depends on (yes we probably need to clean up our dependencies)... but in the mean times... to create exactly the same BAR we're now looking at a build time around 45 mins.
7.0.0 didn't care about transititive dependancies. 7.0.2 does. the result is a huge increase in build times. Considering that mqsicreate bar is nothing but a glorified zip task it's insane. (The real compile happens during deployment).
Is there a way to turn off the validation?
We have a CI server that builds the BARs, deploys them automatically and runs some test XML over the top to get any regression problems.
This used to run in about 20 mins / build (of about 10 BARs) now its 20-60mins per BAR.
Some have increased from 2 min build times to 50+min.
This is running with 8GB RAM, i7 Processor (8Core) and a solid state HDD (and the virus scanner is turned off). Its CPU bound and I had look in IBM support assistant (health center) and its spending ~80% of the time in an .equals() method). |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Mon Mar 28, 2011 3:01 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I'm not sure if there's a way to disable the validation.
You may have better luck if you can control the resources you pass to mqsicreatebar at a more granular level than just the project level - which is to say that rather than adding the entirety of project B, you just pull in the exact resources that FlowA requires.
The reason the validation was added is that the results of mqsicreatebar without the validation were not actually as reliable as they appeared to be. That is, if a particular dependency was not actually available, there was a chance that the mqsicreatebar would succeed without mentioning the missing dependency or otherwise appear to have succeeded to a casual observation.
So that's what disabling the validation would re-enable... the ability to have mqsicreatebar return successfully after having produced an unreliable bar file. |
|
| Back to top |
|
|
| craigp |
| Posted: Mon Mar 28, 2011 3:52 pm Post subject: |
|
|
Newbie
Joined: 28 Mar 2011
Posts: 8
|
We actually do filter what gets passed to it...
<include name="test.flows/**/mf*.msgflow"/>
<include name="polisy.claim.cobol.model/**/messageSet.mset"/>
<include name="polisy.party.cobol.model/**/messageSet.mset"/>
<include name="polisy.policy.cobol.model/**/messageSet.mset"/>
<include name="polisy.risks.model/**/messageSet.mset"/>
<include name="polisy.upload.model/**/messageSet.mset"/>
Eg: Just include the test flows, and the MRM message sets.
This takes about 1-2 mins to build under 7.0.0.0 and fails under 7.0.0.2
This gets resolved through ANT as:
D:\bin\IBM\MQSI\7.0\bin>mqsiprofile & mqsicreatebar -data d:\jenkins\workspace\IntegrationUnit\build.common\build/../../out/workspace.tmp -b d:\jenkins\workspace\IntegrationUnit\build.common\build/../../out/bar/TEST-7T.02.3669-base.bar -p test.flows cm.common.flows cm.services.model common.version.flows common.support.flows acms.common.flows acms.claim.flows eclips.common.flows eclips.policy.flows esb.common esb.common.flows esb.claim.flows esb.policy.flows evolution.policy.flows harmony.party.flows polisy.claim.cobol.model polisy.common.flows polisy.claim.flows polisy.risks.model polisy.party.flows polisy.policy.cobol.model polisy.policy.flows polisy.upload.flows polisy.upload.model polisy.util.flows seec.event.flows seec.claim.flows seec.common.flows seec.policy.flows -o polisy.claim.cobol.model\polisy.claim.cobol.model\messageSet.mset polisy.policy.cobol.model\polisy.policy.cobol.model\messageSet.mset polisy.risks.model\polisy.risks.model\messageSet.mset polisy.upload.model\polisy.upload\messageSet.mset test.flows\zz\com\foo\claim\lodgement\mfTEST_ClaimLodgementACMSMapping.msgflow test.flows\zz\com\foo\claim\lodgement\mfTEST_ClaimLodgementPOLISYMapping.msgflow test.flows\zz\com\foo\claim\lodgement\mfTEST_POLISYClaimRegistration.msgflow test.flows\zz\com\foo\claim\search\mfTEST_ClaimSearchMapping.msgflow test.flows\zz\com\foo\event\seec\mfTEST_CreateContactNote.msgflow test.flows\zz\com\foo\event\seec\mfTEST_TestEventCreateSEEC.msgflow test.flows\zz\com\foo\party\validate\mfTEST_PartyValidate.msgflow test.flows\zz\com\foo\policy\customer\mfTEST_CustomerSearchMapping.msgflow test.flows\zz\com\foo\policy\inquiry\mfTEST_PolicyInquiryResponseMapping.msgflow test.flows\zz\com\foo\policy\rating\mfTEST_PolicyRatingMapping.msgflow test.flows\zz\com\foo\policy\search\mfTEST_PolicySearchPolisyResponse.msgflow test.flows\zz\com\foo\policy\vur\mfTEST_PolicyVUREvoMapping.msgflow test.flows\zz\com\foo\polisy\util\event\mfTEST_PolisyTestEventCreate.msgflow test.flows\zz\com\foo\test\policy\upload\mfTEST_UploadEvoMapping.msgflow test.flows\zz\com\foo\test\policy\upload\mfTEST_UploadFullMapping.msgflow test.flows\zz\com\foo\test\test\mfTEST_TestSimpleTest.msgflow
There are no errors in the flows we include ... there are no errors in the main project we're wanting to work with ... however the transitive dependancies are what kill it.
The test flows reference ESQL in other Projects (you can see there's quite a few projects listed in the -p section). These Projects then contain flows which require even more other projects (eg models ... some of which are extremely large)... and these transitive dependancies is what causes the built times to go up from 1-2 mins to around 1 hr.
Eg:
Project test.flows contains TestFlow1.msgflow which depends on ESQL in Project policy.flows. Project policy.flows is now required in the workspace. Project policy.flows also contains a flow which depends on Project policy.model [which we don't care about when we're building project test.flows BAR file]. Project policy.model is a model which contains 60000 elements.
One solution would be to re-structure our repo... seperate all ESQL and Flows and then if you were dependant on ESQL you wouldn't be including a project which included any flows that depended on even more projects. Such a refactoring would not be a small task.
Another possible solution might be... as the build script is creating a temporary workspace... only copy ESQL files from dependant projects.. ie: only copy *.esql and sf*.msgflow. (However, this gets complicated when you depend on common sub flows).
Our CI build server takes all the BARs and deploys them locally. Then runs some test messages over the top... Only if this all passes does our build server publish the build as successful. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Mar 28, 2011 6:14 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I think my response stands?
I can't tell from here whether or not everything you are including on -p is actually required or not.
If it is, then I am confused as to the difficulty. Are you really willing to trade an absolute result for an indeterminate result without validation? Are you *sure* you are?
If it is not, then... why are you including them on the -p ? |
|
| Back to top |
|
|
| craigp |
| Posted: Mon Mar 28, 2011 6:46 pm Post subject: |
|
|
Newbie
Joined: 28 Mar 2011
Posts: 8
|
| The validation that we rely on comes afterwards because we take the BAR, deploy it and then run tests over it. Only if all this passes is the build successful. This validation takes about 1 minute to run on the CI server. |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Jul 27, 2011 6:35 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
|
| Back to top |
|
|
| ghoshly |
| Posted: Wed Aug 03, 2011 9:35 am Post subject: Parallel execution |
|
|
Partisan
Joined: 10 Jan 2008
Posts: 333
|
Hi..
Could you please put some light on how mqsicreatebar script can be paralled for better , rather quicker outcome? We need to build and deploy around 80 bar files and deploy them into 8 execution groups. No complain with mqsideploy though even for remote machine.
Thanks in advance. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Wed Aug 03, 2011 10:10 am Post subject: Re: Parallel execution |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| ghoshly wrote: |
Hi..
Could you please put some light on how mqsicreatebar script can be paralled for better , rather quicker outcome? We need to build and deploy around 80 bar files and deploy them into 8 execution groups. No complain with mqsideploy though even for remote machine.
Thanks in advance. |
There are five steps:
1. Create the bar.
2. Override the bar.
3. Package the bar (into tar or zip).
4. Move package to remote machine.
5. Deploy the package (bars & jars) locally.
Each of these steps is a separate Ant task.
Using a Continuous Integration server, like Hudson, CruiseControl, BuildForge, these Ant tasks can be run in parallel.
Avoid deploying the bar remotely through mqsideploy. Only deploy the bar locally after moving the package (tar or zip file) to the target machine.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Aug 03, 2011 11:09 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Yeah but....
many people fall at the first step. The create bar process takes far too long.
Then there are barfile overrides. In my shop they are banned. We have removed them. One bar file deployed to all environments.
We know the reason why the mqsicreatebar takes an ever increasing time. We you have to totally redesign our project infrstructure. That is not an easy thing to do.
So we live with the developer creating the bar file that is deployed to all environments from source control. All environment specific params are read from a DB table under timer control.
We have end had our resident ant/hudson expert look at the build system. He ended up scratching his head.
Yes we can do it but we will need to sort out 150+ projects into perhaps 200+ and then resolve the dependencies. IT is possible but it will take a lot of time. Time that we don't have.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Wed Aug 03, 2011 11:12 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| smdavies99 wrote: |
Yeah but....
many people fall at the first step. The create bar process takes far too long.
Then there are barfile overrides. In my shop they are banned. We have removed them. One bar file deployed to all environments.
We know the reason why the mqsicreatebar takes an ever increasing time. We you have to totally redesign our project infrstructure. That is not an easy thing to do.
So we live with the developer creating the bar file that is deployed to all environments from source control. All environment specific params are read from a DB table under timer control.
We have end had our resident ant/hudson expert look at the build system. He ended up scratching his head.
Yes we can do it but we will need to sort out 150+ projects into perhaps 200+ and then resolve the dependencies. IT is possible but it will take a lot of time. Time that we don't have. |
Your shop needs to apply Build Engineering principles. Without them, your on the spiral downwards.
What I mean by this is, you need professional Build Engineers. Sounds like a dozen of them, from the size you specify.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Aug 03, 2011 12:01 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| lancelotlinc wrote: |
What I mean by this is, you need professional Build Engineers. Sounds like a dozen of them, from the size you specify. |
Excuse me while I fall on the floor laughing. We have one build engineer who knows what they are doing.
That said, we have made a concious decision to operate in the way we do. It has been working without a problem for the past 10 months.
What we do and the way we do it works.
Ironically the teams (Java, js, JBoss etc) that do use continuious integration have far longer and more tortuous release cycles than the Broker, Oracle or SAP teams.
But every site is different. Ours is very different from yours. Plus ca Change!
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Wed Aug 03, 2011 12:07 pm Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| smdavies99 wrote: |
| lancelotlinc wrote: |
| What I mean by this is, you need professional Build Engineers. Sounds like a dozen of them, from the size you specify. |
Excuse me while I fall on the floor laughing. We have one build engineer who knows what they are doing.
That said, we have made a concious decision to operate in the way we do. It has been working without a problem for the past 10 months.
What we do and the way we do it works.
Ironically the teams (Java, js, JBoss etc) that do use continuious integration have far longer and more tortuous release cycles than the Broker, Oracle or SAP teams.
But every site is different. Ours is very different from yours. Plus ca Change! |
It works for you, but you are cutting corners. You omit the bar override step, which is an important part of Build Engineering practices. For example, you wouldn't want your trace nodes to fire in production. Or, you want QA env paths to be different from Prod paths, etc. If it works for you, great. But I never did like cutting corners.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Aug 03, 2011 10:45 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| lancelotlinc wrote: |
For example, you wouldn't want your trace nodes to fire in production. Or, you want QA env paths to be different from Prod paths, etc. If it works for you, great. But I never did like cutting corners. |
1 - The trace nodes are controlled by a DB param. Stop flow, change param, start flow. Job done.
2 - The path's/URL's/Usernames etc are held in the DB. A different DB it used by P-Prod, Test, Dev and Live
The system was designed this way. Specifically to remove the need for barfile overrides AND to give us the flexibility to change things like passwords etc without having to redeploy anything.
How is that cutting corners? Answers on a pin-head please.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Aug 04, 2011 4:42 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| smdavies99 wrote: |
| How is that cutting corners? Answers on a pin-head please. |
Cutting corners, or less effective, because your shifting functions away from off-the-shelf solution and into a custom coded solution that consumes horsepower. The idea of a bar file override is that you can leverage the off-the-shelf tool to make changes to your bar file in the development time rather than in the production run time.
Because you are making calls to a database to do what the bar file override would do, you introduce increased cost in the form of increased maintenance of custom code where you could have simply used a BFO to accomplish the same thing. If your database solution is not caching these data values (via Singleton et. al. or other mechanism), seems you would be even more wasteful of the horsepower of your production system since you would have to look up these data elements every invocation.
I'm not sure why people are not wanting to use the BFO portion of the product. The BFO file is simply a list of <string>=<value>. Where <string> equals something, change <string> to the <value> in the BFO.
I especially like IBM's product design here. It is very efficient in that you do not need to spend production system horespower in order to account for environmental changes between Sandbox, Dev-Int, QA, Perf Test, Prod or Hotfix. Alas, it seems some people don't have the luxury of more than one environment. Reminds me of my Gateway 2000 days when we had one AS/400 G95 for all, including development and production. The day when I consumed 99.7 percent of the CPU trying to get an SQL query to work and left 0.3 percent of the CPU for the other 5,000 employees trying to enter orders. But I digress....
Its vastly more cost and time efficient to leverage off-the-shelf tooling rather than custom code. Would you agree?
Your database approach reminds me of the central processing concepts espoused by mainframe-oriented developers. Developing distributed systems is an entirely different animal. Trying to shoehorn mainframe centralized processing concepts into a world of distributed systems processing is square in round.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Thu Aug 04, 2011 5:44 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
As we are way off topic, I'll make this my last answer
1) The DB reads are cached. Many of my posts in this area will show that I'm a supporter of that technique. Oh, and it does not use a singleton.
2) We have a common subflow that is used for inputs of various types. This subflow holds the code to read the DB and to refresh its cache. I don't need to expand on the benefits of doing it that way.
Regarding BFO's.
Because you are making a change to the deployed artifact (ie the one that was released) you will have to go through a full set of tests once it is deployed into production. This may very well not be possible in a fully live system. If this is the case then you are on somewhat of a wing and a prayer that the BFO's were not only correctly formulated but also applied correctly.
In our solution, exactly the same bar file is deployed to QA, UAT & Live. I can see nowt wrong in that.
Now I will shut up.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
|
|
