| Author |
Message
|
| mk140589 |
 Posted: Tue Oct 08, 2013 10:44 pm Post subject: DFDL development : multiple version of a fixed length record |
 |
|
Novice
Joined: 17 Jun 2013
Posts: 20
|
Hi Guys,
I have a question which I can't solve.
My format contains several fields which can be given or not. (If given, it is not necessarily with data in it, but consists of whitespaces)
For example :
| Code: |
Record 1
Fixed field 1
Fixed field 2 (here a file can end)
Optional field 1
Optional field 2 (here a file can end)
Second Optional field 1
Second Optional field 2 (here a file can end)
|
The Second Optional field can't be given without the first Optional fields.
So I tried to use nill values with a %NL terminator, but this only works if it concerns 1 optional field, not multiple.
Does anyone know how to do something like this?
Thanks for your help,
Best regards,
Maarten[/list] |
|
| Back to top |
|
 |
| dogorsy |
| Posted: Tue Oct 08, 2013 10:58 pm Post subject: Re: DFDL development : multiple version of a fixed length re |
|
|
Knight
Joined: 13 Mar 2013
Posts: 553
Location: Home Office
|
| mk140589 wrote: |
Hi Guys,
I have a question which I can't solve.
My format contains several fields which can be given or not. (If given, it is not necessarily with data in it, but consists of whitespaces)
For example :
| Code: |
Record 1
Fixed field 1
Fixed field 2 (here a file can end)
Optional field 1
Optional field 2 (here a file can end)
Second Optional field 1
Second Optional field 2 (here a file can end)
|
The Second Optional field can't be given without the first Optional fields.
So I tried to use nill values with a %NL terminator, but this only works if it concerns 1 optional field, not multiple.
Does anyone know how to do something like this?
Thanks for your help,
Best regards,
Maarten[/list] |
You need to get your ideas clear: the subject title says the record "fixed length".
then you say
| Quote: |
Fixed field 2 (here a file can end)
|
so, is it fixed length or not ?
If it is fixed length, then you define all fields, with their respective lengths, whether they come as spaces or not is irrelevant.
If it is not, you will need to describe better your input record ( and your problem ). |
|
| Back to top |
|
|
| mk140589 |
| Posted: Tue Oct 08, 2013 11:07 pm Post subject: |
|
|
Novice
Joined: 17 Jun 2013
Posts: 20
|
Hi,
I thought that my subject was well declared.
We have 3 different versions of one record.
Each version has a fixed length :
V1 has a length of 100 characters
V2 has a length of 160 characters
V3 has a length of 250 characters
The only thing that I have to distinguish the version of the record is the length of the entire record or the presence/absence of the Optional fields.
Kind regards,
Maarten |
|
| Back to top |
|
|
| dogorsy |
| Posted: Tue Oct 08, 2013 11:25 pm Post subject: |
|
|
Knight
Joined: 13 Mar 2013
Posts: 553
Location: Home Office
|
| mk140589 wrote: |
Hi,
I thought that my subject was well declared.
We have 3 different versions of one record.
Each version has a fixed length :
V1 has a length of 100 characters
V2 has a length of 160 characters
V3 has a length of 250 characters
The only thing that I have to distinguish the version of the record is the length of the entire record or the presence/absence of the Optional fields.
Kind regards,
Maarten |
ok, that is completely different to what you described in the first post.
In your case, you need to define in DFDL a choice, something like
| Code: |
myRecord
choice
V1
V1 fields
end V1
V2
V2 fields
end V2
V3
V3 fields
end V3
end choice
end |
You may also need to use discriminators/assertions to determine which choice to use, or reorder the V1, V2, V3. DFDL picks the first that matches the definition.
PS: you still do not have clear the notion of fixed length/optional
if V1 is 100 characters, then it is 100 chars and has not optional fields. If a field is spaces, it has been specified, IT IS NOT OPTIONAL. Having a blank space is not the same as having nothing.
so if V1 is 100 chars, V2 160 and V3 250, the length can be used to determine the type of record. THERE ARE NO OPTIONAL FIELDS.
If the records are truly fixed length, then I would place the longest first, then the 2nd longest, etc. |
|
| Back to top |
|
|
| kimbert |
| Posted: Wed Oct 09, 2013 8:21 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
So...let me try to state the problem in my own words...
- your message format consists of lines that are terminated by CR/LF
- Each line is a 'record'
- One of the record types can have 'optional' fields after the first 100 characters. The 'optional' fields are either completely missing ( the record is truncated ) or they are filled with spaces.
Questions:
- Is it valid to get a line that contains exactly 107 characters, with 7 trailing spaces. What about 157 ( with trailing spaces after the first 100)? or 249?
- Are the 'missing' fields reliably filled with spaces, or can they contain other types of 'filler'?
- Is it possible to have spaces in characters 101-160 and then real data from 161 to 250?
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| dogorsy |
| Posted: Wed Oct 09, 2013 8:46 am Post subject: |
|
|
Knight
Joined: 13 Mar 2013
Posts: 553
Location: Home Office
|
| @kimbert: I am glad it is not only me. Even though the OP said he/she thought the "subject was well declared", I was only guessing. However he/she has not come back. |
|
| Back to top |
|
|
| mk140589 |
| Posted: Wed Oct 09, 2013 9:18 am Post subject: |
|
|
Novice
Joined: 17 Jun 2013
Posts: 20
|
Hi guys,
I'm sorry that I didn't come back earlier.
@kimbert :
- It is not valid to have 107 or 157 characters, it should be the exact number. It is possible that it is only filled with spaces, but in this case it should be registered as given but empty.
- Spaces are the only valid type of filler
- it is possible that characters 101 - 160 only contain spaces and the fields behind it are filled with real data.
The previously mentioned number of characters where only an example.
In my test XSD i've used the real numbers.
I made a test XSD with only 2 record versions but for some reason the choice isn't working as it should, but maybe this is because of my limited knowledge of DFDL.
My Xsd :
| Code: |
<?xml version="1.0" encoding="UTF-8"?><xsd:schema xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/" xmlns:ibmDfdlExtn="http://www.ibm.com/dfdl/extensions" xmlns:ibmSchExtn="http://www.ibm.com/schema/extensions" xmlns:recFixLengthFieldsFmt="http://www.ibm.com/dfdl/RecordFixLengthFieldFormat" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<xsd:import namespace="http://www.ibm.com/dfdl/RecordFixLengthFieldFormat" schemaLocation="IBMdefined/RecordFixLengthFieldFormat.xsd"/>
<xsd:annotation>
<xsd:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:format byteOrder="{$dfdl:byteOrder}" documentFinalTerminatorCanBeMissing="yes" encoding="{$dfdl:encoding}" escapeSchemeRef="recFixLengthFieldsFmt:RecordEscapeScheme" occursCountKind="fixed" ref="recFixLengthFieldsFmt:RecordFixLengthFieldsFormat" separatorSuppressionPolicy="anyEmpty"/>
</xsd:appinfo>
</xsd:annotation>
<xsd:element dfdl:lengthKind="delimited" ibmSchExtn:docRoot="true" name="IFTTOI">
<xsd:complexType>
<xsd:sequence dfdl:separator="%NL;">
<xsd:element dfdl:lengthKind="implicit" dfdl:occursCountKind="implicit" maxOccurs="unbounded" name="HREC03">
<xsd:complexType>
<xsd:choice>
<xsd:sequence>
<xsd:element dfdl:length="317" dfdl:lengthUnits="characters" name="SEQ1" type="xsd:string"/>
<xsd:element dfdl:length="50" dfdl:lengthUnits="characters" name="SEQ2" type="xsd:string"/>
<xsd:element dfdl:length="50" dfdl:lengthUnits="characters" name="SEQ3" type="xsd:string"/>
</xsd:sequence>
<xsd:sequence>
<xsd:element dfdl:length="317" dfdl:lengthUnits="characters" name="SEQ1_2" type="xsd:string"/>
</xsd:sequence>
</xsd:choice>
</xsd:complexType>
</xsd:element>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
</xsd:schema> |
My test-message (2nd line record is the one which is longer) :
| Code: |
0003001003CZ 882400053100 XPEDYS NV Houtdok 25A Antwerpen 2030 BE
0003002003CN 840000000000 Nyrstar Budel BV Hoofdstraat 1 Budel 6024 AA NL BU1A130821
0003003003N1 888277600139 XPEDYS NV 100 |
If i look at the parser, it is saying :
9 Oct 2013 19:17:41 fatal: CTDP3058E: Separator '%NL;' not found at offset '417' for sequence or choice within element '/IFTTOI[1]'.
In the structure, it used the long record for the first line.
Thank you for your patience and help,
Best regards,
Maarten |
|
| Back to top |
|
|
| kimbert |
| Posted: Wed Oct 09, 2013 9:19 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
In my experience it is extremely rare to get a description of a message format that leaves no room for further questions. This is problem description is quite good compared with the usual.
As I have often said, the problem is the human brain. It makes parsing seem like a simple thing, because the brain is so effortlessly brilliant at it.
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| dogorsy |
| Posted: Wed Oct 09, 2013 9:33 am Post subject: |
|
|
Knight
Joined: 13 Mar 2013
Posts: 553
Location: Home Office
|
| Quote: |
9 Oct 2013 19:17:41 fatal: CTDP3058E: Separator '%NL;' not found at offset '417' for sequence or choice within element '/IFTTOI[1]'.
|
How do you know your separator is %NL ? it is more likely to be %CR;%LF; or %LF; |
|
| Back to top |
|
|
| mk140589 |
| Posted: Wed Oct 09, 2013 9:46 am Post subject: |
|
|
Novice
Joined: 17 Jun 2013
Posts: 20
|
| Quote: |
| How do you know your separator is %NL ? it is more likely to be %CR;%LF; or %LF; |
Because %CR;%LF; and %LF; didn't work with the other records and %NL; does. |
|
| Back to top |
|
|
| dogorsy |
| Posted: Wed Oct 09, 2013 10:02 am Post subject: |
|
|
Knight
Joined: 13 Mar 2013
Posts: 553
Location: Home Office
|
| mk140589 wrote: |
| Quote: |
| How do you know your separator is %NL ? it is more likely to be %CR;%LF; or %LF; |
Because %CR;%LF; and %LF; didn't work with the other records and %NL; does. |
but it does not work with this one. So, you don't know what the separator is. Just guessing.
And you never said before that you managed to get something working. As you can imagine, we are trying to help, but you are not helping us to help you. |
|
| Back to top |
|
|
| mk140589 |
| Posted: Wed Oct 09, 2013 10:04 am Post subject: |
|
|
Novice
Joined: 17 Jun 2013
Posts: 20
|
This is only one type off record within a file with more than 20 different records.
All records work perfect with the %NL separator.
I am only having troubles with recognizing the different versions of HREC03. (The other one only have one version) |
|
| Back to top |
|
|
| dogorsy |
| Posted: Wed Oct 09, 2013 10:21 am Post subject: |
|
|
Knight
Joined: 13 Mar 2013
Posts: 553
Location: Home Office
|
I think you need to be more precise with your language.
You are using the term "record" both for files and records.
Are you trying to say that if you get FILES containing the same type of RECORDS then the parsing works?
and if you get FILES with different type of RECORDS then it does not ?
and another question:
if you go to one shop and pay 10 p for a loaf of bread. The next day go to a different shop and again, pay 10 p for a loaf of bread. Will you then conclude that a loaf of bread will cost you 10 p anywhere in the world ?
Last edited by dogorsy on Wed Oct 09, 2013 10:26 am; edited 1 time in total |
|
| Back to top |
|
|
| mk140589 |
| Posted: Wed Oct 09, 2013 10:24 am Post subject: |
|
|
Novice
Joined: 17 Jun 2013
Posts: 20
|
I get one FILE with several different RECORDS.
As I said before :
My file can get parsed if I leave out the HREC03-records, because these have the versioning.
But it still contains 19 other different types of records. |
|
| Back to top |
|
|
| dogorsy |
| Posted: Wed Oct 09, 2013 10:29 am Post subject: |
|
|
Knight
Joined: 13 Mar 2013
Posts: 553
Location: Home Office
|
| mk140589 wrote: |
I get one FILE with several different RECORDS.
As I said before :
My file can get parsed if I leave out the HREC03-records, because these have the versioning.
But it still contains 19 other different types of records. |
Can you please anwer my questions ? there are 3 questions. I don't care what you said before. |
|
| Back to top |
|
|
|
|
