| Author |
Message
|
| madi |
 Posted: Fri Jan 14, 2011 7:58 am Post subject: WMB Scaling |
 |
|
Chevalier
Joined: 17 Jan 2006
Posts: 475
|
We were two teams serving separate divisions with two different approaches.
1. Have multiple message types in each flow using label nodes. Similarly multiple message types (grouped by application) in each message set.
Our environment had 4 brokers with 4 or 5 execution groups each.
We have about 400 interfaces(using 100+ message flows) (end to end) deployed on this environment supporting close to a million if not more transactions per day.
Team had 4 developers/analysts/testers/admins along with at least 1 contractor.
Interfaces are mostly MQ to MQ with some web service calls.
We focus on reusablity and inteligent interface design.
Strictly deal with MQ, Message Broker.
2. Each message type gets its own flow. Usually each message goes through 2 unique flows. Around 400 interfaces (end to end) using 700+ flows
Environment has 3 brokers on 3 servers. Environment supports close to a million messages per day.
Flows are a mix of web service calls, web services and mq to mq transactions.
Team has 2 leads, 1 estimator, 1 build guy, 2 admins, 4 designers and then usualyy abt 5 off shore developers.
Focus on adding more and more interfaces in a short amount of time, hiring resources as when needed.
Access to BPM products, WPS, Datapower, Various Adaptors etc.
Now, both the teams merged a couple months ago. Apart from the culture change and ideology change, the question now is which is the way to go forward.
1. Pros: Smaller setup, more efficient, less cost, less people.
Cons: Coordination of check-in check-outs from source control is a bottleneck if you have big flows with multiple interfaces in them.
Any major change to that flow means all other interfaces also need to be tested.
2. Pros: Scalability in terms of developers is very good, each can work on their flow not having to worry abt checkout time etc.
No need to test any other interface for chances to the current interface.
Cons: The number of message broker objects grows exponentially. System resoures for so many deployed objects is very high, hence the need of more (powerful)servers.
Expensive to manage servers, more people etc
Nightmare to manage and keep track of the thousands of flows and msgsets that will be created.
What Im looking for is your opinion on both these strategies.
Also helpful if you can describe the setup you have at your shop and pros and cons.
Appreciate any thoughts.
Thanks
_________________
IBM Certified Solutions Developer - WMB 6.0 |
|
| Back to top |
|
 |
| fatherjack |
| Posted: Fri Jan 14, 2011 8:15 am Post subject: |
|
|
Knight
Joined: 14 Apr 2010
Posts: 522
Location: Craggy Island
|
Have you looked at SupportPac IP04. It might give some useful pointers for some of your issues.
_________________
Never let the facts get in the way of a good theory. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jan 14, 2011 8:26 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I have some difficulty with Approach 1 if the distinction between the types of messages supported by a single flow is very large.
That is, if you are essentially implementing multiple full applications in a single message flow - one that processes orders and one that reconciles accounting changes for example....
If, instead, your approach 1 is more like "This flow process all new order messages, regardless of what system or format the order comes from", and thus you have multiple distinct and widely varied formats of messages coming in - but they are all logically "new order" messages, then I'm okay with approach 1.
I guess what I'd say is that I'd tend to start implementing under approach 2, and then consolidate under reasonable practices towards approach 1 as a given set of functions matures. |
|
| Back to top |
|
|
| paintpot |
| Posted: Fri Jan 14, 2011 8:46 am Post subject: |
|
|
Centurion
Joined: 19 Sep 2005
Posts: 112
Location: UK
|
You are missing an architect and a refactoring budget 
You could try mixing the teams up via (50:50) transfer, then have the debate internally once both sides have experienced the other's points of view
Beyond that, I would also suggest asking the business what they need (agility, cost reduction, etc) and what they can afford. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jan 14, 2011 8:48 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| paintpot wrote: |
| You are missing an architect and a refactoring budget |
There probably is a refactoring budget,but it's a) not been mentioned, b) probably insufficient for the actual work that needs to be done.
As for the architect, they probably now have TWO. I think a cage match is the best way to settle that. |
|
| Back to top |
|
|
| madi |
| Posted: Fri Jan 14, 2011 9:17 am Post subject: |
|
|
Chevalier
Joined: 17 Jan 2006
Posts: 475
|
| mqjeff wrote: |
I have some difficulty with Approach 1 if the distinction between the types of messages supported by a single flow is very large.
That is, if you are essentially implementing multiple full applications in a single message flow - one that processes orders and one that reconciles accounting changes for example....
If, instead, your approach 1 is more like "This flow process all new order messages, regardless of what system or format the order comes from", and thus you have multiple distinct and widely varied formats of messages coming in - but they are all logically "new order" messages, then I'm okay with approach 1.
I guess what I'd say is that I'd tend to start implementing under approach 2, and then consolidate under reasonable practices towards approach 1 as a given set of functions matures. |
Usually we have 3 message flows per source application, regular transactions, batch transactions and Synchronous transactions.
Each flow typically can have anywhere from 3 to 15 types of messages coming in.
_________________
IBM Certified Solutions Developer - WMB 6.0 |
|
| Back to top |
|
|
| madi |
| Posted: Fri Jan 14, 2011 9:20 am Post subject: |
|
|
Chevalier
Joined: 17 Jan 2006
Posts: 475
|
| paintpot wrote: |
You are missing an architect and a refactoring budget
You could try mixing the teams up via (50:50) transfer, then have the debate internally once both sides have experienced the other's points of view
Beyond that, I would also suggest asking the business what they need (agility, cost reduction, etc) and what they can afford. |
We do have an architect who is from the Option 2 team.
The architecture is for things going forward, no plans to change the existing interfaces in the near future
My concern is if we keep adding flows and message sets, where is the limit? If we do that, i can see us having a total of 2000+ flows and may be 500 msg set if not more.
I want to know if others have such numbers and how they go about supporting such environment.
_________________
IBM Certified Solutions Developer - WMB 6.0 |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Fri Jan 14, 2011 9:23 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
Lance LIKES the "1 build guy". Would we also call him a Build Engineer? Has he managed to implement automated build process? What tools did he choose?
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| paintpot |
| Posted: Fri Jan 14, 2011 9:55 am Post subject: |
|
|
Centurion
Joined: 19 Sep 2005
Posts: 112
Location: UK
|
ok - now for some real advice, based on what I like to do (which may not be what others like to do!)
Divide your domain in ways that make sense (e.g. business unit, or service-based, or quality of service based, or transport type etc - or any mix of)
Decide if the above matches your deployment strategy (i.e. the divided parts could be candidates for EGs)
Use the right tool for the job - DataPower on the outside of the organisation
Make sure you can drive change rapidly by using test automation (and having test packs).
Standardise and use patterns (not necessarily WMB code generation though) to do the right things in the right order for future builds.
Promote reuse of code / subflows etc - harvest the best from both camps
Create a common object model (canonical form), if appropriate.
Mediation into and out of this using WMB.
Spend lots of time talking together to keep eevryone involved and on side.
Good luck - you just need the right attitudes from all the team members  |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Fri Jan 14, 2011 9:57 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| paintpot wrote: |
ok - now for some real advice, based on what I like to do (which may not be what others like to do!)
Divide your domain in ways that make sense (e.g. business unit, or service-based, or quality of service based, or transport type etc - or any mix of)
Decide if the above matches your deployment strategy (i.e. the divided parts could be candidates for EGs)
Use the right tool for the job - DataPower on the outside of the organisation
Make sure you can drive change rapidly by using test automation (and having test packs).
Standardise and use patterns (not necessarily WMB code generation though) to do the right things in the right order for future builds.
Promote reuse of code / subflows etc - harvest the best from both camps
Create a common object model (canonical form), if appropriate.
Mediation into and out of this using WMB.
Spend lots of time talking together to keep eevryone involved and on side.
Good luck - you just need the right attitudes from all the team members |
Where have you been all my life? Are you single female?
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| madi |
| Posted: Fri Jan 14, 2011 10:06 am Post subject: |
|
|
Chevalier
Joined: 17 Jan 2006
Posts: 475
|
| paintpot wrote: |
Good luck - you just need the right attitudes from all the team members |
there in lies the problem, there is no discussion anymore
I agree there must be a happy medium between our approach and theirs.
In the end, we support just as much traffic and interfaces as they do but with less people, servers and hence cost!!
more importantly i want to know how other shops work.
paintpot: can u give me that for your setup? i.e. number of interfaces end to end, number of flows, egs, brokers etc
_________________
IBM Certified Solutions Developer - WMB 6.0 |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Jan 14, 2011 10:40 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| madi wrote: |
| I agree there must be a happy medium between our approach and theirs. |
Well you've got one good thing at least then; an understanding that neither method is "right"!
| madi wrote: |
| In the end, we support just as much traffic and interfaces as they do but with less people, servers and hence cost!! |
Which ties into the age old debate of where "cost" needs to be, and how do you calculate it? Do you go with method #2 because you think hardware is cheap and there's always room for 1 more server? Or do you go with method #1 because a large team "costs" more? I've worked for both viewpoints and in many cases it varies depending on who's budget what money is coming out of...
| madi wrote: |
more importantly i want to know how other shops work.
|
I've seen it done both ways, and with most shades in between. Where IMHO it's worked best is where there's been either a business or technical grouping, with like-purpose or like-structure messages going into a single flow & being split out by label / Filter / Compute node logic. With a little effort you can keep the retesting to a minimum when you make a change, especially if you ensure common code is truely common, and specific code is segrigated & properly marked.
This of course needs to be balanced against flow size & resource usage. One flow here has a huge amount of logic in the Compute nodes (much of which I think is unnecessary and/or in the wrong product) but we've honoured the spirit of the law by having a single flow which takes all of those message types and splits out into separate flows which are not used external to broker.
It's all about finding the balance that works for your situation.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fatherjack |
| Posted: Fri Jan 14, 2011 3:25 pm Post subject: |
|
|
Knight
Joined: 14 Apr 2010
Posts: 522
Location: Craggy Island
|
| madi wrote: |
My concern is if we keep adding flows and message sets, where is the limit? If we do that, i can see us having a total of 2000+ flows and may be 500 msg set if not more.
I want to know if others have such numbers and how they go about supporting such environment. |
Why are these numbers of concern? 30+ years ago on the mainframe we regularly managed thousands of programs and copybooks and we didn't even have decent source control tools in those days.
_________________
Never let the facts get in the way of a good theory. |
|
| Back to top |
|
|
| jlaisbett |
| Posted: Fri Jan 14, 2011 3:27 pm Post subject: |
|
|
Apprentice
Joined: 27 Nov 2009
Posts: 39
|
Well I can describe the approach we have adopted recently, it has proven to be beneficial to us at least.
When it comes to team structures, budget/funding, architects etc. pretty much all of that is still quite horrible so you're definitely a step ahead of us in that department.
We used to get contractors in every single time we needed to make a change (who didn't actually know anything about message broker) which we have thankfully stopped doing now but what we were left with was chaos, we had flows everywhere and they were all completely different and barely worked at times.
We then spent considerable time coming up with the following:
- We have a few light weight components sitting at the front of everything
- These components handle transformations, logging, security and routing to the component/service flows
- Most of this is common code and doesn't change very often except the transformations
- The transformations allow us to use a single message per component/service internally but the calling applications to use whatever message format best suites them
- We then get into the component/service flows that do all the actual work, they always get their messages from MQ
- The main flow is made up of nearly all common code, MQInput node plus common error handling logic and farms out the message to a label node matching the message type
- We then have a single subflow added to the main flow, all this contains is a collection of label nodes connected to other subflows
- From here each message type gets it's own subflow which is what does all the work
- How many subflows we end up with varies depending on how many similar functions we have, can be only one to dozens
This approach basically minimizes the number of flows we need to deploy but technically all message types still have their own independent flow, they are just subflows. We can add new subflows with minimal impact to any other subflows sharing the same queue because it forwards to them using label nodes matching the message name so we only need to alter the flow containing all the label nodes to add a new message. This approach also simplifies source control for us.
This approach may be pointless when we only have one message type and subsequent subflow but we still use it anyway for consistency. Performance is quite good and we handle about 3 million messages per week last time I looked. There may be better ways and time will tell how useful our approach is but it's served us well for the last year.
We haven't perfected an approach for message sets yet, but we only have half a dozen of them at the moment so it hasn't really been an issue. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sat Jan 15, 2011 12:32 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| jlaisbett wrote: |
We haven't perfected an approach for message sets yet, but we only have half a dozen of them at the moment so it hasn't really been an issue. |
Lucky you.
your site obviously is not into WebServices and other contact admin such as SAP Bapi & iDocs.
If you were your message set could would go through the roof.
IMHO, one of the biggest issues the OP is going to have is merging the different release processes into one streamlined, automated, measurable and most importantly manageable 'thing'.
If you do that then how the different dev teams do their coding becomes less important.
I'd start by looking at the two teams release processes and then thinking about the best practices and possible improvements. Adding something like Continuious Integration Testing, removing (if possible) barfile overrides to the mix.
Then you can come up with a 'new improved 2011' process whereby neither team feels that they have been taken over. Almost every site I've been on has gripes/bitches/complaints about the release & testing process. Get the two teams united by getting them involved in making the new process one that they appear to own. It is a common goal and will build trust and cooperation between them.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
|
|
