| Author |
Message
|
| Esa |
 Posted: Thu Feb 10, 2011 12:06 pm Post subject: using an external XML parser in a java compute node |
 |
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
Hello,
I tried to search the forum to find out if this topic has been discussed earlier and it seems it hasn't.
This is the second customer where I bump into java code where the input message has been parsed as BLOB in the very beginning of a javacompute and then parsed and transformed using another XML parser, Xerces or even MSXML. In my opinion this is very bad design which leads in bad performance and classloading related problems. For example other flows parsing messages with XMLNSC or using nodes that make use of XSL may not work in the same execution group. Or the other way around, depending on which of the flows run first after the execution group has been started and gets to load their versions of certain classes.
I hope this is just something that may happen in a small country where experienced Message Broker professionals are scarce, not to speak about those who can java as well. One reason for choosing this kind of approach seems to be the requirement - or, rather, convention - to unit test all classes separately. Which, of course, is not so easy to achieve with the orthodox approach of coding Message Broker java.
Another characteristic of this kind of coding - I hear they call it the pure code - is extensive use of open source utility libraries, often at least half a dozen apache.commons libraries. In many case just to save a line or two of code while doing some string manipulation. It is called productivity. In my opinion neither this fits Message Broker very well because it can lead to a classloading race between flows that want to load different versions of the same utility library. I think that might easily happen if the customer buys work from several contractors.
I spent weeks last year refactoring a flow like this, basically throwing away the external XML parser and as much of the open source libraries I could. The resulting code had at least 50 % less lines. The customer asked me to do this because they had had classloading problems.
But, before starting to preach the dangers of this approach in a too loud voice, I would like to hear if anybody has a second opinion. |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Thu Feb 10, 2011 12:32 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
At the risk of being showered with brickbats, I'd like to go a little further.
Good Java Devs don't make good Broker Devs.
There I've said it. (I can hear Vitor cheering already)
Seriously, you have described the Propeitary/Java/Open Source Conundrum.
To see some of the lengths some people go to to graft their favourite class library into an othewise stable OOTB Broker install is truly magnificent.
Then they do as you say, read everything as a BLOB then spend ages parsing the message using their own sometimes cryptic methods. Does this result in long term maintanable code? Not in my opinion.
Another controversial statement is coming:-
Most Java Devs don't grok Broker Folders or Abstract Message Models.
I did a presentation on Broker to a bunch of Java devs last year. They clearly didn't get things like On-demand parsing or using message models. The Mapping node was a complete mystery to them.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Feb 10, 2011 12:48 pm Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
Good point smdavies. How would you convey the meaning of a Mapping node to a fresher? If Java devs make poor Broker devs, what class of people make the best Broker devs?
I am down with it, grok'ing along with both of you that passing a BLOB in to a JCN just to parse the payload anyway is a totally bad idea. If the payload needs parsed, best to do it as why else does Broker exist but to parse messages?
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Feb 10, 2011 12:59 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| smdavies99 wrote: |
| There I've said it. (I can hear Vitor cheering already) |
Who, me?  
I'd qualify this slightly (whilst agreeing completely) that Java devs don't make good broker devs unless they accept that they're not in Kansas any more. A JCN is not an EJB, nor is it a standalone Java app, it runs in a different place for a different reason and you'd use one or the other according to need. One of the things that makes a JCN not an EJB is that it's surrounded by useful tools (like parsers) that you can use for the heavy lifting.
In my simplistic world this is much the same as developers (not always Java ones) going through hoops to prove that WMQ messages arrive safely because they're used to writing TCP/IP socket code that handshakes. It's right and valid that you do that in that envrionment, but pointless & wasteful in an MQ environment that has all sorts of features to ensure message delivery. But you tell them that and the room's suddenly knee deep in paranoia & odd scenarios where messages are eaten by The Data Monster that lives in the network.
As a minimum, this should give the brickbats a secondary target.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Feb 10, 2011 1:03 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| lancelotlinc wrote: |
| If Java devs make poor Broker devs, what class of people make the best Broker devs? |
Java or other language developers who accept that some of the techniques they're wedded to are no longer applicable in a WMB environment.
Which is not just a training issue (fresher -> experienced) but requires the developer to not just listen to the training but hear it.
Which I suppose could be considered a training issue as well.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Feb 10, 2011 1:06 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
The question is, did the customer buy Broker to use Broker, or did they voluntarily decide to spend $BIG for a JVM runtime?
If the customer's development team doesn't know the answer to that, perhaps the customer needs to discuss this with their development team.
And regardless of what kind of developer makes the best Broker developer, a simple demonstration - "Look, it took me fifteen minutes to create a production ready application to transform this message into that message" can usually go a long way. Especially because the Java developer is likely still writing the data bean classes. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Feb 10, 2011 1:20 pm Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
A gripe that goes along with this is, when the employment ad says WMB developer then all the interview questions revolve around Spring/Hibernate/EAR files, something is wrong. For every ten WMB job interviews I did in October last year, nine of the interviews had questions that touched no where near WMB.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Thu Feb 10, 2011 1:21 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| mqjeff wrote: |
| - "Look, it took me fifteen minutes to create a production ready application to transform this message into that message" can usually go a long way. |
Not far enough when you have a SAP/CRM guy constantly stating the I can webservice enavle this functionality is 5 minutes. That makes your 15 minutes look possibly snail like.
The problem is that people hear the tla 'ESB' and immediately think of a Java Appserver. Well that is natural. most of them are.
The fact that Broker is Websphere Advanced ESB and in NOT an Appserver does not even comprehent to many PHB's.
So the PHB's say 'this guy knows Java. He can pick up Broker in a few days. It's so simple isn't it (referring back to the 15 minutes quoted by mqjeff)?"
Then they take 3-6 months before they are half decent at broker dev.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| fatherjack |
| Posted: Thu Feb 10, 2011 1:31 pm Post subject: |
|
|
Knight
Joined: 14 Apr 2010
Posts: 522
Location: Craggy Island
|
| smdavies99 wrote: |
| I did a presentation on Broker to a bunch of Java devs last year. They clearly didn't get things like On-demand parsing or using message models. The Mapping node was a complete mystery to them. |
Hey! I've been doing WMB dev for nearly 10 years now and the Mapping Node is still a mystery.
_________________
Never let the facts get in the way of a good theory. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Feb 10, 2011 9:07 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| smdavies wrote: |
| Good Java Devs don't make good Broker Devs. |
I beg to differ on the definition of "good".
A "good" Java developer gets the definition of MBElement and the parallel with the XML document model node.
A "good" Java developer gets the fact that some specific XML oriented C/C++ libraries would parse faster than xerces or any Java based parser...
But then what I call a "good" Java developer and what is perceived by the Java Industry as a good Java developer are perhaps MILES apart.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Thu Feb 10, 2011 10:30 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| fjb_saper wrote: |
| smdavies wrote: |
| Good Java Devs don't make good Broker Devs. |
But then what I call a "good" Java developer and what is perceived by the Java Industry as a good Java developer are perhaps MILES apart. |
Only miles? How's about the distance from where you are to <insert place where your company is gong to offshore your development>?
I see literally dozens of CV with all sorts of Javeryness described in flowing detail. When in the interview, I show them the code in a simple JCN, they can't tell me how it works. As for parsing a message? That is one of my interview questions. Normally, you get
'I'll use <insert what I was taught to use at rote school> for parsing.
Then I ask,
"So what about the native parser for the framework. Why not use that?"
Pah.
Then I show them a piece of really nice ESQL and they go pale in the face when I say that code easily handles 300,000 messages per hour.
I'm not saying that there are not decent broker devs who grok Java(or vice versa) but frankly, IMHO they are rarer than fresh T-Rex (eggs or Lp's).
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Esa |
| Posted: Thu Feb 10, 2011 11:55 pm Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
Thanks everybody for the entertaining discussion, so far.
I especially liked this one:
| fatherjack wrote: |
Hey! I've been doing WMB dev for nearly 10 years now and the Mapping Node is still a mystery. |
Back to business:
We all seem to agree that parsing messages with external tools is a very bad idea.
But what about other libraries, open source utilities etc. ?
I am advicing my clients to review all libraries developers are about to use and produce a list of accepted library versions. To make sure their environment is consistent.
But am I over-reacting? Is there any real danger of flows in an execution group loading different versions of same classes? Is it possible that flows start failing after installing some future broker fix level? Because the execution group itself needs to load a different version of some open source class? In version 7? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Feb 11, 2011 2:31 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
In general, Broker tries to ensure that it has protected itself from classloader issues by using a layered structure of classloaders.
Particularly in v7, you can configure a JCN to use a JavaClassLoader configurable service, and thus create an application level classloader that will only be in effect when and where you want it to be.
In general, as you know, if Broker already does something, it is more manageable and a better idea to have Broker do it for you. So use a SOAPRequest node rather than embedding AXIS yourself and etc. etc. etc. |
|
| Back to top |
|
|
| Esa |
| Posted: Fri Feb 11, 2011 4:14 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| mqjeff wrote: |
| In general, as you know, if Broker already does something, it is more manageable and a better idea to have Broker do it for you. So use a SOAPRequest node rather than embedding AXIS yourself and etc. etc. etc. |
Yes, I agree. Only code those parts of your logic that cannot be implemented by configuring nodes. If you follow that rule of thumb there should not be much need for using external libraries. If you think you need them you should ask yourself: do I need it for implementing message routing and transformation logic or business logic? Can I move the business logic to a place that is achitecturally more correct (= outside Message Broker)?
But in this case I am trying to build guidelines for developers other than myself.
| mqjeff wrote: |
In general, Broker tries to ensure that it has protected itself from classloader issues by using a layered structure of classloaders.
Particularly in v7, you can configure a JCN to use a JavaClassLoader configurable service, and thus create an application level classloader that will only be in effect when and where you want it to be. |
Classloading has always been a bit obscure to me, but I think in version 6 it was impossible to have two flows in same execution group using different versions of same class.
But in version 7 it is possible if the other flow (or the JCN within it) is using a configured classloader service? |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Fri Feb 11, 2011 4:53 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| smdavies99 wrote: |
| fjb_saper wrote: |
| smdavies wrote: |
| Good Java Devs don't make good Broker Devs. |
But then what I call a "good" Java developer and what is perceived by the Java Industry as a good Java developer are perhaps MILES apart. |
Only miles? How's about the distance from where you are to <insert place where your company is gong to offshore your development>?
I see literally dozens of CV with all sorts of Javeryness described in flowing detail. When in the interview, I show them the code in a simple JCN, they can't tell me how it works. As for parsing a message? That is one of my interview questions. Normally, you get
'I'll use <insert what I was taught to use at rote school> for parsing.
Then I ask,
"So what about the native parser for the framework. Why not use that?"
Pah.
Then I show them a piece of really nice ESQL and they go pale in the face when I say that code easily handles 300,000 messages per hour.
I'm not saying that there are not decent broker devs who grok Java(or vice versa) but frankly, IMHO they are rarer than fresh T-Rex (eggs or Lp's). |
300,000 / 60 = 5,000 messages per minute. 5,000 / 60 = 83 per second. This sounds like z/OS. Am I right?
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER
Last edited by lancelotlinc on Fri Feb 11, 2011 4:56 am; edited 1 time in total |
|
| Back to top |
|
|
|
|
