| Author |
Message
|
| dpaul |
 Posted: Tue Jan 31, 2017 7:08 am Post subject: MRM TDS Format Fixed Length |
 |
|
Newbie
Joined: 31 Jan 2017
Posts: 8
|
Hi,
Thanks before hand! (new to IIB).
I have to use MessageSets. I have a fixed length file that I need to parse. There are no tags. It consists of 1 header record, 0 or more detail records and 1 trailer record. Each records is seperated by CRLF. Each type of record is of a different length, eg, header = 20 characters, Detail = 30 characters and Trailer = 10 Characters. The individual fields vary in length.
Example
HeaderField1HeaderField2HeaderField3<CRLF>
DetailField1DetailField2DetailField3DetailField4DetailField5<CRLF>
DetailField1DetailField2DetailField3DetailField4DetailField5<CRLF>
TrailerField1TrailerField2<CRLF>
I don't know how to indicate to the parser each type of record when it starts and when it ends. I don't know how to set up the Group Indicators, Group Terminators?
Can you specify a group to be the total length of the record? So that the parser would know if it encounters record length of 20 it is a Header record of 30 it is a detail record and of 10 it is a Trailer record?
What is the best way to handle this? I can't seem to find any examples on the web.
Thanks Much!
Dave |
|
| Back to top |
|
 |
| Vitor |
| Posted: Tue Jan 31, 2017 7:16 am Post subject: Re: MRM TDS Format Fixed Length |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| dpaul wrote: |
| I have to use MessageSets. |
Don't. Use DFDL.
| dpaul wrote: |
| What is the best way to handle this? |
Use DFDL and set indicators / terminators.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| dpaul |
| Posted: Tue Jan 31, 2017 7:24 am Post subject: |
|
|
Newbie
Joined: 31 Jan 2017
Posts: 8
|
Hi Vitor
I'll check to see if we can mix the DFDL into this service.
Is it possible to do this type of parsing with Message Sets?
dp |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jan 31, 2017 7:30 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| dpaul wrote: |
| I'll check to see if we can mix the DFDL into this service. |
Mix what? How would the service know or care if you're creating the message tree with MRM or DFDL?
| dpaul wrote: |
| Is it possible to do this type of parsing with Message Sets? |
I think you'd tie yourself in knots trying. If you could do that kind of "determine record type from length", it would almost certainly need to be a Parsed Record Sequence. That is a parsing method of last resort because the efficiency is (obviously) the lowest and you'll pay a steep price if you have any data size at all.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jan 31, 2017 8:04 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Why would you possibly need two types of models?
If you have old TDS message models, convert them to DFDL. IMO, in all cases, not just this one.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jan 31, 2017 8:25 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| If you have old TDS message models, convert them to DFDL. IMO, in all cases, not just this one. |
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| dpaul |
| Posted: Tue Jan 31, 2017 9:58 am Post subject: |
|
|
Newbie
Joined: 31 Jan 2017
Posts: 8
|
The design calls for Message set, so I can't change the design without meeting with people is what I meant.
Okay, so it is possible but you think it is a bad idea and should use DFDL.
Thanks
Dave |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jan 31, 2017 10:22 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| dpaul wrote: |
| The design calls for Message set, so I can't change the design without meeting with people is what I meant. |
You should tell people it's a very odd thing to specify data modeling technology in a design document. Data format and layout for sure but not how the solution will parse it. What's going to happen if you decide to port the design to another solution, like Java in WAS?
| dpaul wrote: |
| Okay, so it is possible but you think it is a bad idea and should use DFDL. |
It's a bad idea to do any new development in MRM. DFDL is better and any MRM should be limited to existing MRM sets where you're making a trivial change. As my most worthy associate points out, if you have a development task in progress with all of the associated testing and control processes that implies then it's an ideal opportunity to convert existing MRM to DFDL.
Specific advantages that DFDL offers in your case is that DFDL can be both tested in the Toolkit without deployment and the DFDL Trace offers rich insights into how the DFDL parser is interpreting the data. This will save you a lot of time with the kind of complex data structure you're describing,
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jan 31, 2017 10:33 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Does the design say "message set"? or "TDS"?
DFDL can be considered a message set...
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| dpaul |
| Posted: Tue Jan 31, 2017 10:52 am Post subject: |
|
|
Newbie
Joined: 31 Jan 2017
Posts: 8
|
Hi mqJeff
The design does say message set. This is an addition to a project that is all message sets. I don't think it will be a big deal. I'll start reading up on DFDL.
I'll let you know if I have issues.
Thanks
Dave |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jan 31, 2017 10:53 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| dpaul wrote: |
| This is an addition to a project that is all message sets. |
See my comments above, which go double if all these message sets are recent new development.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| dpaul |
| Posted: Sat Feb 04, 2017 8:02 am Post subject: Continuation of DFDL, discriminators |
|
|
Newbie
Joined: 31 Jan 2017
Posts: 8
|
Hi guys,
I studied up on DFDL. Thanks for the advice. it's nice to be able to test the parser without deploying. I got once file parsed. For the next file I am working on, it is comma delimited with this format:
HeaderRecord
DetailRecords 0 or many
TrailerRecord
Header and Detail records have an indicator at the beginning of the records that I can put an "Initiator" on.
However the format of the trailer record is:
Column Description Type Size Valid Values
1 Input file Created Date CHAR 11 i.e. 24 AUG 2011
2 Number of Exception Records NUMERIC 10 0-9999999999
The first field is a date. I can't figure out how to create an initiator for a date that changes. So how will the parser know that the last detail record has occurred and the next record is the trailer?
Is it possible to use a Discriminator with a regular expression that will act like an Initiator for the parser so it knows the next record is a Trailer record?
I can't find many examples of using Discriminators with syntax I am looking for. I'll mess around with it, but if you know answers to these questions it could save me some time.
Thanks
Dave |
|
| Back to top |
|
|
| dpaul |
| Posted: Sat Feb 04, 2017 8:15 am Post subject: Never Mind |
|
|
Newbie
Joined: 31 Jan 2017
Posts: 8
|
What do ya know. The parser figured it out without an initiator at all. Guess i should have tried it out without posting first. Overthought it.
Thanks |
|
| Back to top |
|
|
| timber |
| Posted: Sat Feb 04, 2017 4:48 pm Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1280
|
I expect that the DFDL parser is trying to parse the trailer record as a Detail record. Because it fails, DFDL backs out and tries the next alternative which is the trailer. I recommend that you look at the DFDL Trace in the Test Parse view and check whether my guess is correct.
I also recommend that you consider whether your test data is representative of real-world data. Will it ever be possible for the Trailer record to look like a valid Detail record, thus deceiving the DFDL parser into parsing it successfully as a Detail record?
| Quote: |
| Is it possible to use a Discriminator with a regular expression that will act like an Initiator for the parser so it knows the next record is a Trailer record? |
Not quite, but almost. IBM DFDL supports only one type of discriminator, and it acts on the 'info set' and not on the bit stream. In other words, you can let DFDL parse an element, then use a discriminator to examine the parsed data and decide whether it should have done so. If the discriminator evaluates to false, DFDL will obediently back out the parsing of that element.
The DFDL specification allows for a different type of discriminator; one that acts on the raw bitstream using a regular expression. This type is not yet supported by IBM DFDL. |
|
| Back to top |
|
|
| dpaul |
| Posted: Sun Feb 05, 2017 7:21 am Post subject: |
|
|
Newbie
Joined: 31 Jan 2017
Posts: 8
|
Thanks Timber! I will check out the trace. Pretty sophisticated parser. Fortunately the detail and trailer records are very different.
Also, thanks for the clarification on the discriminator. I think I have this correct. So based on your comment, since the parser was able to parse the trailer record into the Infoset, I could add a discriminator to the trailer element that could use a regex to verify it further in order to decide to keep it or not.
Thanks
Dave |
|
| Back to top |
|
|
|
|
