| Author |
Message
|
| rodmill |
 Posted: Mon Nov 07, 2011 5:18 pm Post subject: MRM TDS All Elements Delimited want zero length strings |
 |
|
Novice
Joined: 01 Feb 2007
Posts: 13
|
Hi,
I have created a message definition using MRM TDS for a large and complex message
with some missing fields. Data Element Separation=All Elements Delimited.
For the missing fields (ie consecutive delimiters), the parser omits them from the
logical message as seen inside the flow.
The Infocenter tells me that if I set:
Nillable=true
TDS Encoding Null = NullLiteralValue
TDS Encoding NullValue = ''
which works, but it gives me fields with a value of NULL. What I want is fields of value zero-length-string.
As a last resort, I could write some ESQL to walk over the whole message and do an
assignment like this: field=COALESCE(field,''), but I feel there should be a parser option.
I am converting from MQSI 2.0.0.2 which used NEON, and very messy ESQL which I do not want to
touch, so I want to give the old ESQL the same message it was written for. Apparently, the NEON
parser did it this way.
Thanks in advance for any help,
Regards,
Rod |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Tue Nov 08, 2011 2:54 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I guess it's a question of how much change will interfere with your migration process.
The amount of change you've already incurred is significant, even without altering the ESQL.
Among other things, as you've discovered, the NEON parser is not remotely the same as MRM. You can still use the NEON parser, presuming you have a "Rules and Formatter" edition of broker.
However, even if you manage to achieve your goal of leaving the ESQL untouched, you are incurring a significant amount of risk that the functions that the ESQL calls are unchanged and are still valid. Which is not likely... consider at a minimum BITSTREAM vs ASBITSTREAM.
Plus, you still need to change all ESQL paths that reference the logical message tree to reference Root.MRM instead of Root.{whatever}. So changing any checks against '' to check against NULL is not a significantly larger change. At least in my book.
And you'll want to update loops to use references, instead of cardinality and counters.... and and and... |
|
| Back to top |
|
|
| shanson |
| Posted: Tue Nov 08, 2011 3:04 am Post subject: |
|
|
Partisan
Joined: 17 Oct 2003
Posts: 344
Location: IBM Hursley
|
rodmill, I have a couple of questions for you if I may...
Do you consider these missing strings to be 'required' or 'optional' ? I'm guessing 'required', otherwise if they were 'optional' you would not be bothered if they were omitted from the tree.
What behavior would you expect if the element was not a string, but say an integer or a datetime? Adding empty string to the tree for non-strings is not an option as it is not in the lexical space for the logical type. Are you ok with NULL for non-string types? |
|
| Back to top |
|
|
| rodmill |
| Posted: Tue Nov 08, 2011 1:29 pm Post subject: |
|
|
Novice
Joined: 01 Feb 2007
Posts: 13
|
mqjeff & Steve,
Thanks for your answers.
It is true that in going from MQSI 2.0.0.2 to WMB 7 there are some ESQL changes, but the only one I have had a problem with is the change so that decimal cast is not creating leading zeros.
I fix them manually. There is also a change in concatenating of null strings.
I am very impressed with that fact that code written 10 years ago still works reliably in WMB v7.
It has been running in production, with NO maintenance, for 10 years.
Regarding NEON: no, we will not have NEON in the new system, so I am replacing it with MRM.
Of course that might be obsolete soon, as I am planning on installing WMB 8 as soon as I can get it.
Regarding the paths: the author used InputBody, so switching the parser was easy.
Regarding the non-string types: the author of this code did not know that ESQL had types other than string (in the message). (He has got a few int variables in the ESQL.) So date manipulation is done in strings using
SUBSTR, TRIM, and POS (and no other functions!) (CAST is used to get leading zeroes in front of decimals! That is why no fixpacks were applied. That stopped working at the next fixpack.)
REPLACE is never used: he used a combination of POS & SUBSTR to achieve REPLACE.
So I do "consider these missing strings to be 'required'", as I want the code to run as it always has, that way I will not have to figure out the logic, as it is so convoluted.
So all I want is for the TDS layer of the MRM parser to give me a zero-length string when it finds consecutive
delimiters, in "All Elements Delimited" data. It will also make it easier for the testing phase, were we run the new & the old system in parallel, and compare the output messages automatically.
Thanks for your help,
Regards,
Rod |
|
| Back to top |
|
|
| shanson |
| Posted: Tue Nov 08, 2011 2:30 pm Post subject: |
|
|
Partisan
Joined: 17 Oct 2003
Posts: 344
Location: IBM Hursley
|
Rod, the DFDL parser in MB V8 will do this as long as the element is required (ie, minOccurs = 1). But it can only do this for xs:string or xs:hexBinary. If you have integers then you'll need to model them as xs:string if you want empty string in the tree. Empty string is not legal for xs:int etc.
You can put in a formal request to have this enhancement made to the MRM. Same deal applies for xs:int etc. But I can tell you what the reply will be - fixed in DFDL in V8.
Regards
Steve |
|
| Back to top |
|
|
| rodmill |
| Posted: Wed Nov 09, 2011 12:53 am Post subject: |
|
|
Novice
Joined: 01 Feb 2007
Posts: 13
|
Steve & mqjeff,
Many thanks for your comments, as following up your ideas led me to the discovery: I had a logic error in my conversion from NEON to MRM that led to some of the (deep) structures being not quite right. Once fixed, that 10 year old, MQSI 2.0.0.2 code worked perfectly, on WMB 7. Wonder how it will go on WMB 8.
Many thanks for your help,
and I am happy with TDS behaviour wrt missing fields & nulls,
Regards,
Rod |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Nov 09, 2011 3:51 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Refactor mercilessly.
I guarantee that the business need being supported by this code has changed in the last 10 years. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Nov 09, 2011 4:33 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Coding stuff that worked in 2.0.2 was a real PITA. Boy, was I glad to see the back of it at one site. They basically gave up trying to do anything in ESQL and did all the business critical stuff in Oracle Stored Procs. Then there were the limits to the size of result sets you could get back from Oracle.
As has been suggested the code you have ported from 2.0.2 may work BUT I'd expet that a good few of the 'tricks' that had to be employed to make things work int 2.0.2 have been closed/fixed/removed.
Refactor the code. Simply using REFERENCES will improve its operation by a huge amount. Tree traversing always impacted performance.
give one typical flow a go. Until you try it, you won't know how much better your code could be.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| rodmill |
| Posted: Wed Nov 09, 2011 12:32 pm Post subject: |
|
|
Novice
Joined: 01 Feb 2007
Posts: 13
|
Sorry, there was an ambiguity in my references to "10 year old code": I meant that the IBM supplied MQSI 2.0.0.2 product code was 10 years old. Since we all know that it has got better (in every way) every year for the last 10 years, it seems to me that is a very strong recommendation for the quality and reliability of the WMB 7 product. The customer has been modifying the flows over that period to keep up with business requirements. Then the only flow programmer left. So they are stuck now, as no one else is prepared to touch MQSI, as it is in an undocumented, unsupportable mess. I am not even certain that the code I am converting is the same as what is in production.
For legal reasons, they have got to install WMB 7 (or  . But that does not mean they are going to stick with it. Their plan is to replace it with Microsoft SQL Server Integration Services. That is why I am not refactoring. Just have to get the flows to work for the transition period. There are no performance issues anyhow.
Any comments on why someone should or should not replace WMB 8 with MS SSIS would be welcome. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Nov 09, 2011 12:35 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| rodmill wrote: |
| Any comments on why someone should or should not replace WMB 8 with MS SSIS would be welcome. |
WMB 8 will perform better and be more reliable. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Nov 09, 2011 12:37 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| rodmill wrote: |
| Any comments on why someone should or should not replace WMB 8 with MS SSIS would be welcome. |
#include anti-Microsoft.rant 
Does it have the track record of WMB?
Does it have the platform reach of WMB?
Does it have the protocol coverage of WMB?
Does it have the transformational power of WMB?
Does the organisation need any of those things?
Does SSIS cost less than WMB?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Nov 09, 2011 12:39 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| Does SSIS cost less than WMB? |
without having any knowledge on the price of SSIS, WMB v8 Express Edition will be very reasonable in cost. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Nov 09, 2011 12:48 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| Vitor wrote: |
| Does SSIS cost less than WMB? |
without having any knowledge on the price of SSIS, WMB v8 Express Edition will be very reasonable in cost. |
And I don't have any clue on comparitive pricing either; I was throwing a question into the air.
But it's a point well made that WMB comes in different flavours.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Nov 09, 2011 12:50 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
In addition to everything else, the .NET features in v8 may make it a lot more attractive to this particular customer instead of SSIS.
That is, one of the main reasons they may be considering SSIS is because it allows them to leverage their .NET skills... which MB v8 *also* allows... |
|
| Back to top |
|
|
| rodmill |
| Posted: Thu Nov 10, 2011 1:20 am Post subject: |
|
|
Novice
Joined: 01 Feb 2007
Posts: 13
|
mqjeff,
Thanks for your suggestion about WMB v8 Express Edition -- we are following that one up with IBM. |
|
| Back to top |
|
|
|
|
