Author |
Message
|
lancelotlinc |
Posted: Tue Jul 02, 2013 5:43 am Post subject: mqsipackage bar bloat |
|
|
 Jedi Knight
Joined: 22 Mar 2010 Posts: 4941 Location: Bloomington, IL USA
|
WMBT 8.0.0.2
We had some Ant build scripts written to include specified files into a bar. These scripts have been working for a few months now, no problems. We explicitly did not create an 'Application' from our project. We simply built the bar by including all the needed files, file-by-file, and used command line like:
Code: |
mqsipackagebar -w xyz -a zyx -o file1 file2 ... file49 file50 file51 |
Someone down the line converted the Message Flows into an Application. Now, the mqsipackagebar fails with
Quote: |
BIP1868W: Ignoring 'file1.esql', which is in application '<projectname>'. Use the -k option to include the application. |
We have thousands of xml files that make up test case payloads located in a subdirectory of the project. We never want to include these xml files in the bar files.
When putting the -k option in the mqsipackagebar command, the mqsipackagebar scans the project and includes all files specified by
http://publib.boulder.ibm.com/infocenter/wmbhelp/v8r0m0/index.jsp?topic=%2Fcom.ibm.etools.mft.doc%2Fbc31720_.htm
Quote: |
By default files with only the following extensions are included in the BAR file: .xsdzip, .tblxmi, .xsd, .wsdl, .dictionary, .xsl, .xslt, .xml, .jar, .inadapter, .outadapter, .insca, .outsca, .descriptor, .php, .idl, .map, .esql, .msgflow, and .subflow. You can specify the -i option to include files with any extension. |
which is incorrectly including also all the thousands of xml payload files.
How to de-Applicationify the project?
We don't want to use the Application construct.
Or else how to change the default so that xml files are not included, or specified subdirectories are ignored? _________________ http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
Back to top |
|
 |
smdavies99 |
Posted: Tue Jul 02, 2013 5:48 am Post subject: |
|
|
 Jedi Council
Joined: 10 Feb 2003 Posts: 6076 Location: Somewhere over the Rainbow this side of Never-never land.
|
I think that a directory named 'Docs&Testing' gets ignored.
Worth a try... _________________ WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
Back to top |
|
 |
lancelotlinc |
Posted: Tue Jul 02, 2013 5:56 am Post subject: |
|
|
 Jedi Knight
Joined: 22 Mar 2010 Posts: 4941 Location: Bloomington, IL USA
|
smdavies99 wrote: |
I think that a directory named 'Docs&Testing' gets ignored.
Worth a try... |
Good suggestion, however, - by default - all subdirectories are scanned for files within the filename extension specified in the command reference, which includes 'Docs", 'docs', 'Testing', etc.
The InfoCentre leads the reader to believe that the default can be changed. The question is: how to change the default behavior of mqsipackagebar. _________________ http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
Back to top |
|
 |
mqjeff |
Posted: Tue Jul 02, 2013 6:37 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 17447
|
You can change an Application project back into a Message Flow project - but you have to manually edit the .project file
Code: |
<natures>
<nature>com.ibm.etools.msgbroker.tooling.applicationNature</nature>
<nature>com.ibm.etools.msgbroker.tooling.messageBrokerProjectNature</nature>
<nature>com.ibm.etools.mft.bar.ext.barnature</nature>
</natures> |
Otherwise, I'd look into modifying your source code procedures to either not check out certain folders or to delete certain folders after checkout but before build.
The way to "modify" the default behavior of mqsipackagebar is not what you want - you use the -i to include "non-deployable" artifacts.
Last edited by mqjeff on Tue Jul 02, 2013 6:43 am; edited 2 times in total |
|
Back to top |
|
 |
lancelotlinc |
Posted: Tue Jul 02, 2013 6:40 am Post subject: |
|
|
 Jedi Knight
Joined: 22 Mar 2010 Posts: 4941 Location: Bloomington, IL USA
|
mqjeff wrote: |
You can change an Application project back into a Message Flow project - but you have to manually edit the .project file
Code: |
<natures>
<nature>com.ibm.etools.msgbroker.tooling.applicationNature</nature>
<nature>com.ibm.etools.msgbroker.tooling.messageBrokerProjectNature</nature>
<nature>com.ibm.etools.mft.bar.ext.barnature</nature>
</natures> |
|
Ok, thats one solution, and I appreciate your comment.
Is there a way to set mqsipackagebar -k option to exclude certain subdirectories? The InfoCentre leads one to believe that changing this option is possible since it says "By default..." _________________ http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
Back to top |
|
 |
mqjeff |
Posted: Tue Jul 02, 2013 6:44 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 17447
|
You might be able to use both -o and -k together to only include named files from the application. |
|
Back to top |
|
 |
lancelotlinc |
Posted: Tue Jul 02, 2013 6:49 am Post subject: |
|
|
 Jedi Knight
Joined: 22 Mar 2010 Posts: 4941 Location: Bloomington, IL USA
|
mqjeff wrote: |
You might be able to use both -o and -k together to only include named files from the application. |
When doing this, mqsipackagebar rejects the -o args with the error above, and then includes those files, along with all the other unwanted files, as part of the application.
We need a way to scope the file inclusion somehow. Simply including all files inside the project filesystem is not desirable. There are files we want to be included in the project filesystem so that they get recorded into the version control on that project but we don't want to deploy those thousands of XML files to the Broker. _________________ http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
Back to top |
|
 |
mqjeff |
Posted: Tue Jul 02, 2013 6:52 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 17447
|
Yeah. There doesn't seem to be an option you need.
The -i option only lets you expand the list, not shorten the list of things that are included.
But it's worth a quick check to see if it changes the behavior of -o and -k together.
Otherwise, again, look at modifying your build process to delete known subfolders before build. |
|
Back to top |
|
 |
lancelotlinc |
Posted: Tue Jul 02, 2013 6:58 am Post subject: |
|
|
 Jedi Knight
Joined: 22 Mar 2010 Posts: 4941 Location: Bloomington, IL USA
|
Thanks for your comments.
-i has no impact on the unwanted files on any combination of -o or -k.
Deleting the directories and files is not viable because that impacts the source code control system (Jazz - RTC).
How long would it take to create an APAR on mqsipackagebar to exclude various directories, or check each subdirectory folder for an .exclude file :RFE: ?
We're scheduled to go to Production in 30 days. For now, we're building the bar with all the unwanted files. We cannot delete the XML files from the broker once deployed because they get included in the appzip file.
I suppose we'll open a PMR and see where it takes us, quickly or not. _________________ http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
Back to top |
|
 |
mqjeff |
Posted: Tue Jul 02, 2013 7:04 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 17447
|
I'm not talking about changes that would affect source control.
You should be able to check out the whole shebang and then either move or delete files and then package the bar file without affecting the source control repository's understanding.
At a bare minimum, something like - check out to directory A
- copy directory A to directory B
- delete files from dir B
- package from directory B
- copy bar file back to directory A
- sync/commit/etc
Alternately, mqsipackagebar produces a bar file which is a zip file which can be unzipped. The appzip inside it can be unzipped. Any libzips inside that can be unzipped. Or java based tools, perhaps invoked from Ant, can be used to edit the zip file contents "in place". |
|
Back to top |
|
 |
goffinf |
Posted: Tue Jul 02, 2013 11:56 am Post subject: Re: mqsipackage bar bloat |
|
|
Chevalier
Joined: 05 Nov 2005 Posts: 401
|
lancelotlinc wrote: |
WMBT 8.0.0.2
Someone down the line converted the Message Flows into an Application.
....
We have thousands of xml files that make up test case payloads located in a subdirectory of the project. We never want to include these xml files in the bar files.
|
Yeah we noted this when we were working thru our migration strategy from v6.1 to v8. If you stick with MBPs then removing what you want from a BAR created in the toolkit or (as in your case) specifying exactly what gets included is straight-forward.
What we decided on was to refactor our existing projects and split them into separate projects, one (the v8 Application/Library or MBP) for the integration solution we want to deploy to production, and another (an MBP) to contain all of the test resources (following a standard naming convention). In our case the 'test' project includes and flows and ESQL that are just used for the purpose of unit/e2e testing. We also have a further Java project in where we keep the JUnit test classes (that's what we use to test all our flows) and all of the associated resources, including request and expected response messages, schemas and such like.
Ther are a few issues with referencing Application projects from an MBP but they are resolvable.
It is a bit of a nuisance and it would probably be better to be able to selectively choose the contents of the BAR, but I guess the key concept is that an Application is intended to be an indivisible unit of deployment.
In the end it didn't turn out to be much of a problem.
HTHs
Fraser. |
|
Back to top |
|
 |
lancelotlinc |
Posted: Tue Jul 02, 2013 12:13 pm Post subject: |
|
|
 Jedi Knight
Joined: 22 Mar 2010 Posts: 4941 Location: Bloomington, IL USA
|
Submitted RFE 36740 in case anyone wants to vote on it.
The RFE asks for an APAR that allows proper scoping of files. We don't want to maintain our test payloads outside of our project. We want our test payloads to be recorded in the Version Control system of choice. We want our test payloads to be within the project filesystem hierarchy. We do not want our thousands of XML file test payloads to be deployed to a Broker runtime. _________________ http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
Back to top |
|
 |
smdavies99 |
Posted: Tue Jul 02, 2013 10:16 pm Post subject: |
|
|
 Jedi Council
Joined: 10 Feb 2003 Posts: 6076 Location: Somewhere over the Rainbow this side of Never-never land.
|
lancelotlinc wrote: |
We do not want our thousands of XML file test payloads. |
Perhaps now would be a good time to 'zip' them all up?
I do that with many of my non time dependent test messages. _________________ WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
Back to top |
|
 |
lancelotlinc |
Posted: Wed Jul 03, 2013 3:14 am Post subject: |
|
|
 Jedi Knight
Joined: 22 Mar 2010 Posts: 4941 Location: Bloomington, IL USA
|
smdavies99 wrote: |
lancelotlinc wrote: |
We do not want our thousands of XML file test payloads. |
Perhaps now would be a good time to 'zip' them all up?
I do that with many of my non time dependent test messages. |
Hi smdavies - thanks for your comment. Our automated test tool which regression tests the message flows uses the payloads often. Zipping them up would be a workaround, not a solution. For a long-term solution, our IBM Accelerated Value Leader (AVL) representative is in discussions with IBM support for an APAR to mqsipackagebar. _________________ http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
Back to top |
|
 |
|