| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Deploying bar files to distinct environments |
« View previous topic :: View next topic » |
| Author |
Message
|
| ruimadaleno |
 Posted: Thu Apr 09, 2015 2:16 am Post subject: Deploying bar files to distinct environments |
 |
|
Master
Joined: 08 May 2014
Posts: 274
|
Hi all,
we have 3 broker environments - Production (PROD), Quality (QUA) and Development (DEV).
Bar files are deployed in DEV, validated, them deployed in QUA , validated by business users and at last deployed in PROD.
The whole process was done using message broker explorer (MBX). We are perfecting the process to achieve automation.
Right now we use scripts and mqsideploy command to deploy the bar files to the broker environment.
The next challenge is to automate the user defined properties.
Each bar file has several message flows and each mf has several user defined properties- udp. The values stored in these udp change as the environment changes . Right now we use MBX to change the udp values accordingly to the environment, we want to improve this step.
From your experience, what is the best practice to improve ? The goal is to script the whole deployment process so it can be automated (and this is the first step to achieve continuous integration in the future).
My ideia:
For each bar file there should be three properties files. Each properties files should contain all user defined properties for every mf the bar files contains for a certain environment.
Example:
PrismaXPTO.Bar (bar file with a set of message flows)
PrismaXPTO_DEV.properties (message flows udp with values set to DEV environment)
PrismaXPTO_QUA.properties (message flows udp with values set to QUA environment)
PrismaXPTO_PROD.properties (message flows udp with values set to PROD environment)
so the deployment process should be:
1) Apply udp values for DEV environment -> mqsiapplybaroverride <barfile> <barfile>_DEV.properties
2) Deploy bar file in DEV (script with mqsideploy)
3) validate
4) Apply udp values for QUA environment -> mqsiapplybaroverride <barfile> <barfile>_QUA.properties
5) Deploy bar file in QUA (script with mqsideploy)
6) business users validation
7) Apply udp values for PROD environment -> mqsiapplybaroverride <barfile> <barfile>_PROD.properties
8 ) Deploy bar file in PROD (script with mqsideploy)
from your experience what are the pros/cons of this approach ? what can/should be change ?
my environment
Broker 8.0.0.4 running on windows server 2008 R2
_________________
Best regards
Rui Madaleno |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Thu Apr 09, 2015 3:22 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
You really need to think about what happens when a UAT configured bar file is deployed to PROD.
don't think it can happen?
Think again. It WILL happen at some point in time.
{as some here will know, this is a pet gripe of mine.}
There are ways to use the same bar file for all three environments but that is outside the scope of your question.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Apr 09, 2015 4:51 am Post subject: Re: Deploying bar files to distinct environments |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ruimadaleno wrote: |
My ideia:
For each bar file there should be three properties files. Each properties files should contain all user defined properties for every mf the bar files contains for a certain environment.
Example:
PrismaXPTO.Bar (bar file with a set of message flows)
PrismaXPTO_DEV.properties (message flows udp with values set to DEV environment)
PrismaXPTO_QUA.properties (message flows udp with values set to QUA environment)
PrismaXPTO_PROD.properties (message flows udp with values set to PROD environment)
so the deployment process should be:
1) Apply udp values for DEV environment -> mqsiapplybaroverride <barfile> <barfile>_DEV.properties
2) Deploy bar file in DEV (script with mqsideploy)
3) validate
4) Apply udp values for QUA environment -> mqsiapplybaroverride <barfile> <barfile>_QUA.properties
5) Deploy bar file in QUA (script with mqsideploy)
6) business users validation
7) Apply udp values for PROD environment -> mqsiapplybaroverride <barfile> <barfile>_PROD.properties
8 ) Deploy bar file in PROD (script with mqsideploy)
from your experience what are the pros/cons of this approach ? what can/should be change ? |
This site uses exactly this principle; one properties file for each possible target environment. Things you should consider:
I'll assume that by UDP you also mean promoted properties, and if you don't you should. The properties should control everything that changes between environments (web service URLs are the obvious example) so that you use the same bar file in all cases.
Note that whatever you have in your properties files, the PROD version is never applied and tested until you get to PROD. So some kind of stage gate / desk check / manual verification is crucial. Our experience is that about 5% of the PROD properties here are created by copying the QA file and renaming it. Of these 5%, 0% make it through our stage gate and I'm proud that the 5% are typcially new developers who've not encountered the sanctions I've built into the stage gate.
We don't get repeat offenders.
In answer to the pet gripe of @smdavies99, have the script that applies the overrides and does the deploy calculate which environment it's deploying to. We do this by examining the server name (one character of the name describes the environment) but you could do this with a list in the script, dbase look up, etc, etc. I can hear my associate going "someone will use a server with the wrong name in the wrong environment. It WILL happen at some point in time".
He may be right. I'm confident in the checks & balances we have in place (like the servers are themselves built with a script). I'm also confident that a server with a name that doesn't match it's environmental use will cause bigger problems than the wrong code version in IIB. Such as alerting, monitoring, chargeback, security controls, audit and support functions not working correctly.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| akil |
| Posted: Sun Apr 12, 2015 6:53 am Post subject: |
|
|
Partisan
Joined: 27 May 2014
Posts: 338
Location: Mumbai
|
Bar file overrides, rely on a configuration file, the management of the configuration files comes with its own set of challenges, some folks say commit these files to the version control, some say do not..
Certain configurations can be applicable across applications/libraries/flows ( like a proxy, databases, back-end services ), and they tend to get repeated across these file ..
UserDefined configurable services are a bit closer to the 12 factor app (http://12factor.net/) .. But then this requires some bit of ESQL to set the read the properties and set the LE, and then there are certain properties that dont have an LE override.
It would be good to have a built in method for the broker to read the Env/UserDefined properties to use this UserDefined.
_________________
Regards |
|
| Back to top |
|
|
| ruimadaleno |
| Posted: Mon Apr 13, 2015 3:03 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
First of all thank you Smdavies99, Vitor and Akil for you knowledge sharing.
Keeping the configuration files in source control (in my environment subversion -SVN) it's very logical to me, i can think about a SVN subtree to store the configuration files like:
| Code: |
SVN
|__BrokerConfigs
|_____Production
|___prismaXPTO_PROD.properties
|___......
|_____Staging
|___prismaXPTO_QUA.properties
|___......
|_____Development
|___prismaXPTO_DEV.properties
|___......
|
I can see the PROS of keeping the configuration files in SVN:
- we can track changes
- It is a single point of "truth"  (keeping this in folders can lead to several folders: somewhere, somehow, by some unknown reason will make a copy of the folder and after some months nobody knows what are the "real" or the copied configuration files)
- Simple for deployment process: just take the last revision of properties and apply to bar files
How about the CONS ? what are the dangers and pitfalls to avoid in this approach ?
CONS
- someone makes change in a property file, override bar, deploy bar in environment and forgets to commit the changed property file to SVN.
_________________
Best regards
Rui Madaleno |
|
| Back to top |
|
|
| ruimadaleno |
| Posted: Mon Apr 13, 2015 3:07 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
hi Akil,
thank you for you response. I need some help to understand the following:
| akil wrote: |
.. But then this requires some bit of ESQL to set the read the properties and set the LE, and then there are certain properties that dont have an LE override.
It would be good to have a built in method for the broker to read the Env/UserDefined properties to use this UserDefined. |
I need some clarification here. What do you mean by "LE" ?
| Quote: |
UserDefined configurable services are a bit closer to the 12 factor app (http://12factor.net/)
|
After a quick read it seems to me this is a very nice resource !  i'll read through it !
_________________
Best regards
Rui Madaleno |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Apr 13, 2015 4:21 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ruimadaleno wrote: |
How about the CONS ? what are the dangers and pitfalls to avoid in this approach ?
CONS
- someone makes change in a property file, override bar, deploy bar in environment and forgets to commit the changed property file to SVN. |
Not if the deploy process (script) always references SVN for bar & override file. Then the "con" is that they forget to commit the change and the environment is redeployed with it's old value. Which is a problem, but less of a problem than your scenario where the SVN property file (single point of truth) does not accurately reflect what's deployed.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Apr 13, 2015 4:46 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
As a final validation, you should (IMHO) put the modified .BAR file into SVN as a unique timestamped entity. You have modified it so it make sense to save it.
This way you know what was deployed to what environment and when.
Yes you could merge it all but when the system is not working as designed and a totalt re-deploy is needed having the deployed version readily identifiable is a godsend.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| akil |
| Posted: Mon Apr 13, 2015 10:27 am Post subject: |
|
|
Partisan
Joined: 27 May 2014
Posts: 338
Location: Mumbai
|
By LE I meant the LocalEnvironment..
The UserDefined configurable service comes pretty close to the ENV as suggested by the 12 factor app.
What one does in such a case is ,
# writes ESQL to read the values from the UserDefined CS, at this point, if certain properties are not defined, you throw an error so that you don't by mistake run with an incorrect default ( note that override is just an override, if not specified, it will run with the default, which can have its own consequences ). In case of a UserDefined, there are no defaults, you have to set the values of all the properties in the service.
# the ESQL sets the Local Environment (LE) properties, for example instead of using promoted properties & bar-override for the proxy connect headers, use the UserDefined CS for specifying the user/password
# Passwords should not be committed to SVN, if certain properties are passwords or access keys, which you cannot commit to the SVN, then how do you manage them with the override ? You can't , you use the UserDefined.
The catch is that there are certain properties which have a bar-override, but do not have a Local Environment override. For such properties, you'r left with using the bar-overrides ..
Also, I suppose properties of the Input nodes , the first node in your flow, also need a bar-override, and can't be via a LE..
_________________
Regards |
|
| Back to top |
|
|
| ruimadaleno |
| Posted: Fri Apr 17, 2015 7:45 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
from you point of view, what should be the bar file repository ? a shared/local folder ? source control (svn) ? other ? what are the options you are using and what problems can we expect from each one of those ?
_________________
Best regards
Rui Madaleno |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Apr 17, 2015 12:06 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
In our system we store the build bar file in the project folder. The bar file is built with the option to include the sources checked.
This is all in SVN (older projects) or in a GIT repo.
Once deployed the bar file is stored again in SVN or GIT inder a folder with the deploy date in the name. This way we can see the historic deployments (including re-deployments of things like messagesets/libraries) at a glance.
Problems? What sort of problems are you expecting?
If your bar file has been deploted to the different environments succesfully AND you are confident that the setup of the different enviroments is correct then what can go wrong?
Don't answer that!
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| ruimadaleno |
| Posted: Fri Apr 24, 2015 6:50 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
| akil wrote: |
By LE I meant the LocalEnvironment..
The UserDefined configurable service comes pretty close to the ENV as suggested by the 12 factor app.
What one does in such a case is ,
# writes ESQL to read the values from the UserDefined CS, at this point, if certain properties are not defined, you throw an error so that you don't by mistake run with an incorrect default ( note that override is just an override, if not specified, it will run with the default, which can have its own consequences ). In case of a UserDefined, there are no defaults, you have to set the values of all the properties in the service.
# the ESQL sets the Local Environment (LE) properties, for example instead of using promoted properties & bar-override for the proxy connect headers, use the UserDefined CS for specifying the user/password
# Passwords should not be committed to SVN, if certain properties are passwords or access keys, which you cannot commit to the SVN, then how do you manage them with the override ? You can't , you use the UserDefined.
The catch is that there are certain properties which have a bar-override, but do not have a Local Environment override. For such properties, you'r left with using the bar-overrides ..
Also, I suppose properties of the Input nodes , the first node in your flow, also need a bar-override, and can't be via a LE.. |
Hi Akil,
we don't use the UserDefined configurable service. We have a policy in place: all values/properties/endpoint/url/whatever that can change between environment must be declared as a user defined property (UDP). At message flow startup the first node should be a compute node that reads the UDP and sets localenvironment. If one of the UDP is not present (empty udp value) then the message flows exits with an error.
regarding service endpoints (in case a message flows needs to call a web service) our policy is to use endpoint lookup nodes , getting the endpoint url from WSRR (in this cases we have a UDP that defines the environment the message flow is running on, and with this value the message flow can search WSRR for the correct endpoint)
So, my ideia is to put all the UDP for a certain message flows in properties file. There should be one properties file for each environment for each message flow. This properties files are kept under SVN.
The deployment process, should get the latest properties file for the target environment from SVN, apply to bar (mqsiapplyoverride) and then deploy the resulting bar file to target environment.
_________________
Best regards
Rui Madaleno |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Apr 24, 2015 7:01 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| oddly enough, mqsiapplybarovererides allows you to use a properties file. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
