| Author |
Message
|
| dyson |
 Posted: Thu Aug 23, 2012 4:48 pm Post subject: Message Set design - is there a way to ignore new fields? |
 |
|
Apprentice
Joined: 04 Apr 2011
Posts: 45
|
| I have a message set that converts from binary to the xml tree. It fails whenever a new record/field is added to the incoming binary. I was looking for a way to modify the message set that would allow it to ingnore/remove the new records is this doable? |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Thu Aug 23, 2012 4:55 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
There isn't a single generic fix that will allow you to always insert a new field anywhere in the message without impact.
There are techniques that you can use that will allow you to insert specific fields in specific locations.
It seems surprising that a binary format message is changing this often. This should be noted as a change that has impact, and thus managed under normal change management practices.
You should also ideally consider changing the binary message to an actual XML message at the source, which is much more flexible. |
|
| Back to top |
|
|
| dyson |
| Posted: Thu Aug 23, 2012 5:02 pm Post subject: |
|
|
Apprentice
Joined: 04 Apr 2011
Posts: 45
|
| This is short term solution we have plans to change to xml at the source. By techniques are you referring to wildcards? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Aug 23, 2012 5:05 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
As I said, there's no single, generic answer.
It depends entirely on the specific message and where the new fields are being added. |
|
| Back to top |
|
|
| mqsiuser |
| Posted: Fri Aug 24, 2012 1:01 am Post subject: Re: Message Set design - is there a way to ignore new fields |
|
|
Yatiri
Joined: 15 Apr 2008
Posts: 637
Location: Germany
|
| dyson wrote: |
| It fails whenever a new record/field is added to the incoming binary. I was looking for a way to modify the message set that would allow it to ingnore/remove the new records is this doable? |
I hightly guess you can do that with RegExps. You'd likely have to use RegExps on the complex types / group level and on the simply types / elements level !
I am actually using "\r?\n"... while the question mark will make the "return" optional (will parse the message if there or if not there). You'd likely have to use "*" also.
Ofc. XML comes with this type of extensibility for free and RegExps are hard to use, create and read... but: The RegExps that the Message Set are using are Perl-style which is what you want...
They are the most powerful choice you have for modelling flat file messages, meaning you should be able to model any/most possible flat file structures (including optional parts). |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Aug 24, 2012 2:48 am Post subject: Re: Message Set design - is there a way to ignore new fields |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqsiuser wrote: |
| dyson wrote: |
| It fails whenever a new record/field is added to the incoming binary. I was looking for a way to modify the message set that would allow it to ingnore/remove the new records is this doable? |
I hightly guess you can do that with RegExps. You'd likely have to use RegExps on the complex types / group level and on the simply types / elements level !
I am actually using "\r?\n"... while the question mark will make the "return" optional (will parse the message if there or if not there). You'd likely have to use "*" also.
Ofc. XML comes with this type of extensibility for free and RegExps are hard to use, create and read... but: The RegExps that the Message Set are using are Perl-style which is what you want...
They are the most powerful choice you have for modelling flat file messages, meaning you should be able to model any/most possible flat file structures (including optional parts). |
Note that if you can get this to work (I draw your attention to the use of the word "guess" at the start of the above suggestion), be aware that this is the most expensive way possible to parse a message. You'll pay in terms of resource use & performance loss, never mind the ongoing increase in maintenance complexity.
If this is a short term problem, a good change control system is probably better.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqsiuser |
| Posted: Fri Aug 24, 2012 3:48 am Post subject: Re: Message Set design - is there a way to ignore new fields |
|
|
Yatiri
Joined: 15 Apr 2008
Posts: 637
Location: Germany
|
| Vitor wrote: |
| Note that if you can get this to work (I draw your attention to the use of the word "guess" at the start of the above suggestion), |
Kind of "everything is possible with reg exps", but not for everyone (does anyone know of a service from IBM where one can delegate msg-set creation?)
I think if you take it seriously (to create/troubleshoot (regExp) msg-sets yourself) then you have to take a trace and properly know RegExps+TDS!
| Vitor wrote: |
| be aware that this is the most expensive way possible to parse a message. You'll pay in terms of resource use & performance loss, never mind the ongoing increase in maintenance complexity. |
expressiveness is expensive... but we are still talking about that a parser will go though a bitstream from the beginning (basically once) and then (according to rules expressed with RegExps) make (parsing) decisions. Depending on which RegExp you come up with this will be faster or slower... yes it is slowing (a bit) down but you get something in return for that !
And... you have maturity of technology: Perl-style reg-exps (the broker team likely uses a 'proven to be good' RegExp library internally)
@Vitor: May I remind you of this:
| kimbert wrote: |
| Vitor wrote: |
| Data Pattern should be a last resort when all other methods have failed. It's the most expensive of the parsers. |
That's true - if you can do without data patterns then there's no point in paying the CPU cost.
However, I think it's time to balance this up a little. Since v6 data patterns have become a little more efficient ( along with the rest of the TDS parser ). In any case, if there is no other way to do the job, there is no shame in creating a message set that contains data patterns. |
|
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Aug 24, 2012 5:00 am Post subject: Re: Message Set design - is there a way to ignore new fields |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqsiuser wrote: |
@Vitor: May I remind you of this:
| kimbert wrote: |
| Vitor wrote: |
| Data Pattern should be a last resort when all other methods have failed. It's the most expensive of the parsers. |
That's true - if you can do without data patterns then there's no point in paying the CPU cost.
However, I think it's time to balance this up a little. Since v6 data patterns have become a little more efficient ( along with the rest of the TDS parser ). In any case, if there is no other way to do the job, there is no shame in creating a message set that contains data patterns. |
|
I remembered that when writing. There's no shame in a data pattern; it's still the most expensive of the parsing options and is a lot of trouble for a short term problem that would yield to good change control!
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Aug 24, 2012 5:33 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Let's suppose that we have a bog standard COBOL message.
It looks vaguely like - Header (one occurance)
- Detail (repeats indefinitely)
- DetailField1
- ...
- DetailFieldN
Now, let's think about all of the ways that a new field could be added: - a new field added to the start of the header
- a new field added to the middle of the header
- a new field added to the end of the header
- a new field added to the start of the detail
- a new field added to the middle of the detail
- a new field added to the end of the detail
- a new record added before the header
- a new record added between the header and the detail
- a new record added after each detail
- a new record added after the last detail
- a new record added after some but not all details
There's no single method that can be chosen that can accommodate all of those potential changes.
There's no single method that can be chosen to resolve all possible forms of ONE of those potential changes.
If specific detail is given about the specific message being modified - INCLUDING detail about how parts are identified (fixed length, tagged/fixed length, delimited, etc. etc. etc.), and specific detail is given about which type of change is generally being made, then a meaningful suggestion can be made.
Otherwise just stick a variable length field in all of those places, that is treated as a blob, and hope that this doesn't cause the parser to fail to identify the pieces of your message properly.
Every time I hope something like that, I get called at 2am on saturday after a long night at the pub. |
|
| Back to top |
|
|
| dyson |
| Posted: Fri Aug 24, 2012 8:55 am Post subject: |
|
|
Apprentice
Joined: 04 Apr 2011
Posts: 45
|
| Quote: |
| it's still the most expensive of the parsing options and is a lot of trouble for a short term problem that would yield to good change control! |
This might be one of the times where it's worth the price ... we have up to 8 releases per year each will have one new record so this will become a career if I don't find a way to ignore/remove the new records. And by short term I mean 1 or 2 years but then again who knows. The records are tag delimited with a colon and group indicators that are different for each new record (e.g. <u+00ED>:<u+00F1>: ) |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Aug 24, 2012 8:59 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| dyson wrote: |
| by short term I mean 1 or 2 years but then again who knows. |
Ok, the phrase "short term" conveys something entirely different to me. Use the data pattern.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqsiuser |
| Posted: Fri Aug 24, 2012 9:00 am Post subject: |
|
|
Yatiri
Joined: 15 Apr 2008
Posts: 637
Location: Germany
|
Dont hope... test  (ofc this likely was a "I hope it just works (but no time/budget to approve/test) feature"!)
And try out reg exp parsing.
@OP: Do they just append at the end or begining of the msg (that certainly is easier to cover in your msg set def) |
|
| Back to top |
|
|
| dyson |
| Posted: Fri Aug 24, 2012 9:12 am Post subject: |
|
|
Apprentice
Joined: 04 Apr 2011
Posts: 45
|
| No they add it right in the middle someplace. My consolation is that it's restricted to two specific areas in the message set. I will look at the reg exp option, thanks! |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Aug 24, 2012 9:43 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
If the records are tagged-delimited, are you using the tags to resolve a choice?
If so, you can alter the structure of the sequence containing the choice to include a variable length optional field at the end (call it FILLER or something). The unmodelled records will then be consumed by the FILLER. |
|
| Back to top |
|
|
| NealM |
| Posted: Fri Aug 24, 2012 11:06 am Post subject: |
|
|
Master
Joined: 22 Feb 2011
Posts: 230
Location: NC or Utah (depends)
|
Here is another simple way of handling:
1. Assuming your Message set = AA, keep a constant for length of AA (L_AA) that you adjust whenever AA grows.
2. Have your input come in as a BLOB. Set L_Blob = LENGTH(...BLOB)
3. CREATE LASTCHILD OF Environment.myMessage.<msgName> DOMAIN 'MRM'
PARSE(SUBSTRING(InputRoot.BLOB.BLOB FROM 1 FOR L_Blob)
ENCODING InputRoot.MQMD.Encoding
CCSID InputRoot.MQMD.CodedCharSetId
SET 'AA'
TYPE '<AA's messagetype>'
FORMAT '<binary?>');
...so now your known message is in the Environment and ready for use...
4. If L_AA <> L_Blob, you know something has changed, create a warning message (email, trace, whatever) so that a developer is alerted that it's time to change AA again. |
|
| Back to top |
|
|
|
|
