| Author |
Message
|
| sancsan |
 Posted: Fri Feb 26, 2016 2:28 am Post subject: WMB Compute Node Coding Standard |
 |
|
Newbie
Joined: 24 Feb 2016
Posts: 4
|
Hi,
I'm a newbie in IBM Broker development world and need an suggestion on coding standard for below mentioned scenario from performance perspective:
MQ Input --> Compute Node --> HTTP Request
MQ Input will receive 10 different types of events(create, update, delete..) in JSON format and each message type has different transformation rules associate with them to invoke REST API Calls.
Is it appropriate to have one compute node which contains all the transformation rules or create one compute node for each event.
Message size is not significant (in KBs) and also the requirement is to maintain the sequence of the message so at any moment only one message will be processed.
Appreciate your input. |
|
| Back to top |
|
 |
| ruimadaleno |
| Posted: Fri Feb 26, 2016 4:00 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
If you use a compute node to handle each one of the possible events you will end up with something like:
MQ Input --> Compute Node (create events) --> Compute Node (update events) --> .... --->HTTP Request
So, in every compute node a tree copying / tree parsing is performed and this will impact on performance.
My suggestion:
Divide to conquer and Think on reuse ! Use Librarys! separate between message format transformation and business rules.
Example:
LibraryXPTO -> this library contains a set of esql files. Each one of the esql files contains code to handle (transform) an event, maybe one function to handle each event. If using business rules, then code the business rules in another set of esql files (maybe another library). This way, when change happens (it will) you just need to make changes in library.
LibraryXPTO
|------- EventCreateMap.esql
|----------------- function HandleEventCreate(,,,)
|------- EventUpdateMap.esql
|------------------ function HandleEventUpdate(,,,)
|------- EventCreateRules.esql
|------- EventUpdateRules.esql
Then your message flow will look something like this:
MQ Input --> Compute Node (handle events - call functions in library) --->HTTP Request
Note that in this approach you have only one compute node, minimizing tree copying/parse operations.
Just my two cents, but other experts here can contribute
_________________
Best regards
Rui Madaleno |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Feb 26, 2016 4:52 am Post subject: Re: WMB Compute Node Coding Standard |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| sancsan wrote: |
Message size is not significant (in KBs) and also the requirement is to maintain the sequence of the message so at any moment only one message will be processed.
|
This won't scale and is in general (in these parts at least) referred to as 'Message Affinity'. There are some here who shudder when faced with this sort of thing. Not the best design IMHO.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| timber |
| Posted: Fri Feb 26, 2016 5:33 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
Personally, I would favour a design that is similar to a web service flow. One subflow ( or part of a flow, if you prefer ) per operation. A Compute node and a RouteToLabel node to select the operation. One label node for each subflow. Operation-specific logic in each subflow.
This design is flexible and maintainable, but does incur some avoidable tree copying. Whether that matters depends on your performance requirements. |
|
| Back to top |
|
|
| ruimadaleno |
| Posted: Fri Feb 26, 2016 6:26 am Post subject: Re: WMB Compute Node Coding Standard |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
| smdavies99 wrote: |
| sancsan wrote: |
Message size is not significant (in KBs) and also the requirement is to maintain the sequence of the message so at any moment only one message will be processed.
|
This won't scale and is in general (in these parts at least) referred to as 'Message Affinity'. There are some here who shudder when faced with this sort of thing. Not the best design IMHO. |
Absolutely right 
_________________
Best regards
Rui Madaleno |
|
| Back to top |
|
|
| inMo |
| Posted: Fri Feb 26, 2016 9:21 am Post subject: Re: WMB Compute Node Coding Standard |
|
|
Master
Joined: 27 Jun 2009
Posts: 216
Location: NY
|
| sancsan wrote: |
Message size is not significant (in KBs) and also the requirement is to maintain the sequence of the message so at any moment only one message will be processed.
|
2 cents: The idea that this won't scale is not necessarily a bad thing, it is just a fact for consideration. There are surely other facts we'd want to look at before asserting this as an issue. Is this a blanket mandate for the whole of the logical message processing layer or is this a mandate for a particular flow the OP is working on? Is the expected volume 10 messages/per day, 10,000, 10,000,000? Why does the single thread requirement exist? Does message affinity truly exist? If so, Is there true message affinity across all messages or just a handful that is forcing a rule change for all? |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Feb 26, 2016 9:30 am Post subject: Re: WMB Compute Node Coding Standard |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| inMo wrote: |
| Is this a blanket mandate for the whole of the logical message processing layer or is this a mandate for a particular flow the OP is working on? Is the expected volume 10 messages/per day, 10,000, 10,000,000? Why does the single thread requirement exist? Does message affinity truly exist? If so, Is there true message affinity across all messages or just a handful that is forcing a rule change for all? |
My 2 cents on top of yours:
The real problem with message affinity is that even if you only have 10 messages a day, you can't process message 3 if message 2 has a problem (e.g. bad data in the payload). This quickly leads you out of SLA and into badness. There's a process here which had affinity and was a big problem. They had groups of 5 - 10 messages that "had" to be processed in sequence, and got 4 such groups on a busy day.
Each message was a multi-million dollar wire transfer. I leave the s**t that went down when the messages couldn't be processed for over an hour to the reader's imagination.
This war story illustrates another important point. The affinity here was because they used to arrive in a file and were processed in record order. While message 1 added $10m to account A and message 2 removed $10m from account A and sent it to account B, as long as all the messages were processed before the end of day reconciliation no one actually cared.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| maurito |
| Posted: Fri Feb 26, 2016 10:06 am Post subject: |
|
|
Partisan
Joined: 17 Apr 2014
Posts: 358
|
| timber wrote: |
Personally, I would favour a design that is similar to a web service flow. One subflow ( or part of a flow, if you prefer ) per operation. A Compute node and a RouteToLabel node to select the operation. One label node for each subflow. Operation-specific logic in each subflow.
This design is flexible and maintainable, but does incur some avoidable tree copying. Whether that matters depends on your performance requirements. |
I would also favour this approach. The copying of the tree can be avoided by doing a
| Code: |
| PROPAGATE MESSAGE InputRoot |
|
|
| Back to top |
|
|
| timber |
| Posted: Fri Feb 26, 2016 10:27 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
| Quote: |
| The copying of the tree can be avoided... |
Good tip! Thanks for that. |
|
| Back to top |
|
|
| sancsan |
| Posted: Sun Feb 28, 2016 2:27 am Post subject: |
|
|
Newbie
Joined: 24 Feb 2016
Posts: 4
|
| Thanks everyone for your suggestions. Very useful. |
|
| Back to top |
|
|
|
|
