| Author |
Message
|
| p939276 |
 Posted: Thu Feb 04, 2016 1:36 pm Post subject: Implementing a three layered SOA design with IIBv10 |
 |
|
Novice
Joined: 17 Dec 2015
Posts: 10
|
Hi all,
I've been searching for a similar topic on the forum but I couldn't find any, so here I am creating my own.
In our client, we are implementing a SOA solution using IIBv10.
Main drivers of the solution are: Clients using different protocols (SOAP, HTTP, MQ, REST), Different backends (AS/400, SOAP, REST) and the requirement of having a well defined inventary of reusable services.
A SOA Architect with experience in other product (OSB) is suggesting to make a design in three layers:
- Proxy service: Transforms protocol to SOAP protocol and to a common schema (not canonical but at least, reusable and common to each business entity).
- Reusable service: Business service in charge of orchestrating requests to target services and maybe, other reusable services (composition). No schema transformation here; everything is "spoken" with this common message models.
- Target services: Specific for each backend and in charge of transforming from common message models to the models of the backend or 3rd party.
Everything is great on paper but when thinking on how to implement it on IIB we have several doubts, and we know that the decisions we take here a "one way". So, if any of you have implemented something similar with the product and can offer us your experienced point of view, we'd really appreciate it.
The questions are:
1) Would you think on three flows here (one for each layer in the simplest example with no complex orchestrations), a single flow with a subflow (for the reusable service with another subflow inside for the target service), or two flows for the first two layers and the second one having subflows inside for the target services? I know it should be balanced between performance and mainteinability, but probably your experience in IIB can help more than thinking on the obvious architecture reasons.
2) If more than one layer is a flow, should they communicate by SOAP? Even within the same server, is the impact on performance high due to so many message parsing/writing?
3) Thinking on deploy time for the "1 flow with subflows" approach, how should we separate applications, integration services and shared libraries to reduce coupling and easyness of deploy? If the application with a flow depends on a subflow on another layer, should this subflow be on a different application (or shared library?) and the first one depend on this one?
I know they are complex questions to answer here and of course you don't have the business context, but any experienced help on this will be awesome for us.
Thanks you all in advance.
Kind regards. |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Thu Feb 04, 2016 10:56 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
It sounds as if your architect has been drinking the SOA Snake Oil at bit to often.
One place I used to work, had such an infrastructure. It ran like a dog with no back legs.
Support was a nightmare due to the lack of a decent logging system. The designer came from the Java world where Log4J ruled. Ugh!
We moved to a two layer model for all new interfaces. Performed much better. Saved a lot of unnecessary transformations. The original model specified a message structure for communication between each level. This was clearly inadequate as we were forever updating it. In the end we dropped it entirely. We ended up with a model that as long as it was fully documented you could use and message structure you like.
The oroginal design was also folloing the SOA Snake Oil in that all communication between the layers was via SOAP. This as everyone knows is a synchronous thingy. This made getting data from a number of different sources so slow especially if the data from no 2 was not dependent upon data from No 1.
We moved to an MQ model for internal data and things worked a lot better.
The re-use everything is also a myth. We get a lot of re-use in IIB anyway.
If I were you, I'd experiment with a three layer, two layer and one layer. Find out how it all works in your environment and choose what suits you best.
I produced a demo where I could develop a single layer SOAP to SAP Update without having to write a line of code. Fat lot of good it was. The got rid of broker and went with a totally Java solution. I was glad to have left before that debacle happened.
Just my $0.02 worth. As you can see, I'm not a huge fan of SOAP everywhere. It sounds clean but it is usually very dirty. Pun intended.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions.
Last edited by smdavies99 on Fri Feb 05, 2016 1:31 am; edited 1 time in total |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Feb 04, 2016 11:59 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5849
Location: UK
|
I'm afraid that the "Cloud API" thought Police will be arriving soon to arrest you for such heresy.
How dare you suggest that technology of a nature not understood by someone fresh out of Uni should be used?
Asynchronous messaging and transactional integrity are now illegal.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| p939276 |
| Posted: Fri Feb 05, 2016 12:21 am Post subject: |
|
|
Novice
Joined: 17 Dec 2015
Posts: 10
|
Thanks you both for your responses (even you @zpat out of your jokes  )
I totally see @smdavies99 point. Too much SOAP integrations are nice in the docs but probably nightmare when it's time to implement and maintain it.
That's why we are now working on a three conceptual layers but two physical layers, being the third embedded as a subflow in the second one. It seems to be more suitable and mainteinable.
Using MQ to connect both layers were already suggested too and we could understand the benefits, but it's a very SOAP aligned company and we are probably going with the SOAP (and synchronous) way. After all, decoupling this two layers allows for a change later without impacting consumers.
Anyway, going back to application separation on IIB, should we suggest 1 flow per application (like integration services) or 1 application with (mostly) all the flows? We like the first one, but would it be a good solution to have 100+ application deployed?
Thanks again, guys. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Feb 05, 2016 1:37 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
I might add that there are at least two things that are essential to a system of the size you envisage and that is a
1) Services Registry - where each and every interface is documented in all their gory details including WSDL's and sample messages. If you fail to keep this up to date you will soon end up with anarchy and everyone doing their own thing and any ideas about service reuse will bite dust in a big way.
2) Service Versioning - I'll leave that to you to work out why.
Now I'll sit back and wait for the Cloud Police to come and take me away for a frontal lobotomy. Heresy is the name of the game. 
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| ruimadaleno |
| Posted: Fri Feb 05, 2016 3:02 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
"..and the requirement of having a well defined inventary of reusable services..."
My two cents on this requirement:
use WSRR (websphere service registry and repository) and take advantage of the "Endpoint Lookup" and "RegistryLookup" Nodes provided to you by IIB.
_________________
Best regards
Rui Madaleno |
|
| Back to top |
|
|
| p939276 |
| Posted: Fri Feb 05, 2016 3:24 am Post subject: |
|
|
Novice
Joined: 17 Dec 2015
Posts: 10
|
Thanks @ruimadaleno,
We've evaluated WSRR but unfortunately the client cannot afford it on its budget.
Maybe we'll use WSO2 for governance but only for design-time, so no runtime discovery of endpoints and services.
Thanks. |
|
| Back to top |
|
|
| ruimadaleno |
| Posted: Fri Feb 05, 2016 3:44 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
| p939276 wrote: |
Thanks @ruimadaleno,
We've evaluated WSRR but unfortunately the client cannot afford it on its budget.
Maybe we'll use WSO2 for governance but only for design-time, so no runtime discovery of endpoints and services.
Thanks. |
So, how are you thinking to deal with the services endpoint ? think about a service in the "reusable service layer". Think of a service that needs to execute a business function that requires to create one business entity (let's call it A) and update two other business entities (let's call it B and C).
So, your "reusable layer service" must be aware of the endpoint where the "target layer services" for A,B and C are available.
You can "hard code" the endpoints for A,B and C "target layer systems"
You can provide the enpoint for A, B and C on UDP (user defined properties)
Take in mind that A,B and C target layer services are highly reusable in this architecture, so you have to replicate this configuration for every "reusable layer service" you implement.
Also take a minute to think about change .. yes .. it will happen !!
If target layer service A changes and the contract is broken , how do you "update" all the "reusable services layers" ? do you have to redeploy them all ? do you deploy a new version of target layer service A (let's call it A v2.0) and change/deploy all the services in "reusable service layers" ? how comfortable you are in operating and maintaining several versions of services ? what is the tactic for versioning , how do you know what version of the service you are consuming ? by URL ? by namespace ?
_________________
Best regards
Rui Madaleno |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Feb 05, 2016 5:46 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
When building your layers you should also think about decoupling and scaling.
May be what you have set up as one flow would scale much better if it were implemented as 2 flows, knowing that each flow can and will scale differently... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Feb 05, 2016 5:55 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| p939276 wrote: |
| Anyway, going back to application separation on IIB, should we suggest 1 flow per application (like integration services) or 1 application with (mostly) all the flows? We like the first one, but would it be a good solution to have 100+ application deployed? |
This depends a lot on how you're building and governing the applications. If you've got a solid, automated, deploy process (or build and deploy process) then managing 100+ applications isn't that bad. It's not trivial and I'm not suggesting it is, but it does mean that keeping track of what application is at what version & which version is actually boots on the ground in the EG isn't a hideous nightmare from which there is no waking.
Separately and distinctly there's the question of reuse and amalgamation. Just because those drunkenly waving around bottle of SOAP Kool Aid say there are 100+ operations / services / what not does not mean you need 100+ deployable artifacts. Especially as you have v10 with Shareable Libraries. You make this observation in your original post and I think you've picked up a key point in maintaining maintainability.
My $0.02 - a few more and you'll have enough to retire and make this an SEP. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Feb 05, 2016 6:04 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
What you're really proposing is three separate input points.
I've seen this implemented, btw.
But the point of what you're calling the first layer was to provide a mechanism for applications that couldn't, or wouldn't, upgrade to using the common API. These should clearly be limited and strongly encouraged to upgrade. If only because it increases performance of their interactions and reduces maintenance/development costs.
If you don't separate the back side from the common API interface, then your common API interface has to grow to cover every implemented back end... Or has to only act as a routing mechanism.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| ruimadaleno |
| Posted: Fri Feb 05, 2016 8:40 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
"... Transforms protocol to SOAP protocol and to a common schema (not canonical but at least, reusable and common to each business entity)..."
maybe these common schema / canonical should be a set of xsd schema files, put it on a lib, and share it in every "Proxy Service Layer".
_________________
Best regards
Rui Madaleno |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Feb 05, 2016 8:47 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| ruimadaleno wrote: |
| maybe these common schema / canonical should be a set of xsd schema files, put it on a lib |
Also known as a "WSDL".
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Feb 05, 2016 9:58 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Tep.
We had that and at first it was enforced rigidly. However it soon became apparent that there was a class of service where it couldn't be used.
For example requesting a Domain Name from a registrar. Not easy to enforce a 'model' suited for a sales order on that.
This does apparently break all the SOA rules but... rules were there for breaking.
One transformation model clearly did not suit everything.
I can hear the SOA purists trashing about in their gazillion XML based config files as I speak.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| inMo |
| Posted: Mon Feb 08, 2016 7:16 am Post subject: |
|
|
Master
Joined: 27 Jun 2009
Posts: 216
Location: NY
|
| mqjeff wrote: |
| ...the point of what you're calling the first layer was to provide a mechanism for applications that couldn't, or wouldn't, upgrade to using the common API. |
I agree. It is really the only point to the stated first layer - especially when noting the 2nd layer is the common layer. It would almost be better to have the common layer be the primary layer, and consider your 1st layer a helper type layer for those apps desperately needing help to communicate with the primary common layer.
[/b] |
|
| Back to top |
|
|
|
|
