| Author |
Message
|
| ottfried11 |
 Posted: Fri Jun 28, 2013 3:39 am Post subject: Model message with DFDL |
 |
|
Apprentice
Joined: 28 Jun 2013
Posts: 29
|
Hi all,
I'm new to the message broker and I'm trying to model the following message with DFDL
| Code: |
#1433200117240007003EUL8003EUR60010600108DDD600108DDD8DDD
2DDD8DDD6001060010600108DDD2DDD2DDD600108DDD8DDD7DDD7DD
D7DDD#1101300067240007008K981895I7DDD7DDD7017GB w/
Kumulierung8002LZ7DDD#1642600168240007003EUL8003EUR8DDD6001
02DDD8DDD8DDD600106001060010600106001060010600108DDD7DDD8
DDD7DDD2DDD8DDD2DDD8DDD8DDD7DDD7DDD7DDD2DDD60010600107
DDD2DDD2DDD8DDD8DDD#15244000212400060010#148520028924000
7004HTDE7DDD8004HAND8002KF200108003EUR40102013-06-
07300817.32.4440102013-06-07300817.32.0040102013-06-
133XXX8004VERS8004GULT7DDD10083443519640102013-06-
07300817.32.412002307003STK7DDD0DDD7DDD4DDD3DDD4DDD3DDD8D
DD0DDD7DDD8004A2207004A2207DDD8002KT8DDD600102DDD60010000
1060011......... |
The message is set up as follows
structure ID: #14332 (always # + 5 digits)
length of the strukture: 00117 (inklusive länge der Struktur Id) -> the length is not fix
5 digit ID: 24000 (I don't know what this id is for. lets assume it is not always the same)
structure body :
7003EUL -> the first character is always the domain (0 = Short, 1 = Long, 2 = Real, 3 = Zeit, 4 = Datum, 6 = Bool., 7 = String, 8 = Key) I'm not sure what 5 stands for. It does not really matter
-> characters 2 - 4 = field length: here the length is 3. when the field is empty, he lenght of the field is described with DDD
-> field value.
The number of structures in the message can vary
The structures do not always come in the same order. The vlues in a structure are fix.
The hash # can not be used as a delimiter because it could also appear in text fields.
A message always starts with a hash # and structure id. One has to then determine the length of the structure to find the next one.
For later mapping in the target data we only need some of the structures in the message.
I hope someone can help me on this
Cheers
Michael
Last edited by ottfried11 on Fri Jun 28, 2013 3:47 am; edited 1 time in total |
|
| Back to top |
|
 |
| lancelotlinc |
| Posted: Fri Jun 28, 2013 3:40 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Jun 28, 2013 4:59 am Post subject: Re: Model message with DFDL |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ottfried11 wrote: |
| For later mapping in the target data we only need some of the structures in the message. |
But you still need to describe the full message, especially if bits of the input are optional
| ottfried11 wrote: |
| I hope someone can help me on this |
Help you with what? You've described (and described quite well) the format of the input message. This is well within the scope of DFDL to parse so what happens when you try it? What errors are you getting? Are you getting a parsed message but with some of the fields unrecognised? Why not post the DFDL schema as well as the sample data (again tagging it for readability)?
Better information, better advice.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| ottfried11 |
| Posted: Fri Jun 28, 2013 5:10 am Post subject: |
|
|
Apprentice
Joined: 28 Jun 2013
Posts: 29
|
Hi Vitor,
thanks for the quick reply.
As of now, I have no DFDL schema. The only good examples I found were for csv or delimited messages.
I do not know how to define a DFDL schema for such a message. How is the lenght field desrcribed? How do I then find the end of the structure. As I wrote: I'm new to the WMB and DFDL. Can you give me an example or a start up which I could play around with?
That would help a lot. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Jun 28, 2013 5:20 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Fri Jun 28, 2013 5:21 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
|
| Back to top |
|
|
| kimbert |
| Posted: Mon Jul 01, 2013 12:38 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
This is an example of a TLV data format ( http://en.wikipedia.org/wiki/Type-length-value ).
It is actually quite a tricky format to model, whether you are using DFDL or any other language. And as always, the details really matter. Here are a few tips:
| Quote: |
| length of the strukture: 00117 (inklusive länge der Struktur Id) -> the length is not fix |
You need to set lengthKind='explicit' and set the length to "{xs:nonNegativeInteger(../structureLength)}". You may need to add a bit of calculation into the XPath expression if the length includes the length of the prefix field. Alternatively ( and more simply ) you may be able to use lengthKind="prefixed". It depends on exactly how the structure length is calculated. You should be able to read the specification and work out the correct choice.
| Quote: |
7003EUL -> the first character is always the domain (0 = Short, 1 = Long, 2 = Real, 3 = Zeit, 4 = Datum, 6 = Bool., 7 = String, 8 = Key) I'm not sure what 5 stands for. It does not really matter
-> characters 2 - 4 = field length: here the length is 3. when the field is empty, he lenght of the field is described with DDD
-> field value. |
You have a number of options for dealing with the fields.
a) model them using a single element with maxOccurs="unbounded" and occursCountKind="implicit". This will produce a message tree like this
| Code: |
InputRoot
DFDL
Structure[0]
TLV_generic_element[0]
value
TLV_generic_element[1]
value
...
TLV_generic_element[N]
value |
b) If you know the names and type of the elements in each structure, you can construct a more detailed model that contains meaningful names. You will need a type library for the various TLV types. There are techniques for doing this in a maintainable way - let me know if you want some help with it.
a) is less work. b) will produce easier to maintain message flows. |
|
| Back to top |
|
|
| ottfried11 |
| Posted: Mon Jul 01, 2013 12:56 am Post subject: |
|
|
Apprentice
Joined: 28 Jun 2013
Posts: 29
|
Thanks a lot for your help. I'll try my luck the next days and will keep you informed  |
|
| Back to top |
|
|
| shanson |
| Posted: Tue Jul 02, 2013 1:13 am Post subject: |
|
|
Partisan
Joined: 17 Oct 2003
Posts: 344
Location: IBM Hursley
|
| It would help us if you could say whether the message format is for a particular industry standard. Is there a public link to a specification? |
|
| Back to top |
|
|
| ottfried11 |
| Posted: Tue Jul 02, 2013 1:28 am Post subject: |
|
|
Apprentice
Joined: 28 Jun 2013
Posts: 29
|
Hi,
its not really a standard. Its a so called SWEIO message used by the order management system GEOS. It contains order information which will then be mapped to a FIX message to route this to an exchange (e.g. NYSE) |
|
| Back to top |
|
|
| ottfried11 |
| Posted: Wed Jul 03, 2013 12:34 pm Post subject: |
|
|
Apprentice
Joined: 28 Jun 2013
Posts: 29
|
| Code: |
<?xml version="1.0" encoding="UTF-8"?><xs:schema xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/" xmlns:ibmDfdlExtn="http://www.ibm.com/dfdl/extensions" xmlns:ibmSchExtn="http://www.ibm.com/schema/extensions" xmlns:recSepFieldsFmt="http://www.ibm.com/dfdl/RecordSeparatedFieldFormat" xmlns:tns="http://www.ibm.com/dfdl/ISO8583Types" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<xs:import namespace="http://www.ibm.com/dfdl/RecordSeparatedFieldFormat" schemaLocation="IBMdefined/RecordSeparatedFieldFormat.xsd"/>
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:format byteOrder="{$dfdl:byteOrder}" encoding="{$dfdl:encoding}" escapeSchemeRef="recSepFieldsFmt:RecordEscapeScheme" occursCountKind="fixed" ref="recSepFieldsFmt:RecordSeparatedFieldsFormat"/>
</xs:appinfo>
</xs:annotation>
<xs:element dfdl:emptyValueDelimiterPolicy="none" dfdl:initiator="" dfdl:lengthKind="implicit" name="SWEIO">
<xs:complexType>
<xs:sequence dfdl:separator="" dfdl:sequenceKind="ordered">
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:sequence/>
</xs:appinfo>
</xs:annotation>
<xs:element dfdl:emptyValueDelimiterPolicy="none" dfdl:initiator="" dfdl:lengthKind="implicit" name="Structure14332">
<xs:complexType>
<xs:sequence dfdl:separator="" dfdl:sequenceKind="ordered">
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:sequence/>
</xs:appinfo>
</xs:annotation>
<xs:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="1" dfdl:lengthKind="explicit" dfdl:nilValueDelimiterPolicy="none" fixed="#" name="leadingHash">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:minLength value="1"/>
<xs:maxLength value="1"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="5" dfdl:lengthKind="explicit" dfdl:nilValueDelimiterPolicy="none" ibmDfdlExtn:sampleValue="aaaaa" name="structureId">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:minLength value="5"/>
<xs:maxLength value="5"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="5" dfdl:lengthKind="explicit" dfdl:nilValueDelimiterPolicy="none" ibmDfdlExtn:sampleValue="aaa" name="structureLength">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:minLength value="5"/>
<xs:maxLength value="5"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="5" dfdl:lengthKind="explicit" dfdl:nilValueDelimiterPolicy="none" ibmDfdlExtn:sampleValue="aaaaa" name="unknownId">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:minLength value="5"/>
<xs:maxLength value="5"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="{xs:nonNegativeInteger(/SWEIO/Structure14332/structureLength)-16}" dfdl:lengthKind="explicit" dfdl:nilValueDelimiterPolicy="none" ibmDfdlExtn:maxOccurs="1" ibmDfdlExtn:sampleValue="aaaaaaaaaaaaaaaaaaaa" name="messageStream">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:minLength value="0"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element dfdl:emptyValueDelimiterPolicy="none" dfdl:initiator="" dfdl:lengthKind="implicit" name="Structure11013">
<xs:complexType>
<xs:sequence dfdl:separator="" dfdl:sequenceKind="ordered">
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:sequence/>
</xs:appinfo>
</xs:annotation>
<xs:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="1" dfdl:lengthKind="explicit" dfdl:nilValueDelimiterPolicy="none" fixed="#" name="leadingHash">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:minLength value="1"/>
<xs:maxLength value="1"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="5" dfdl:lengthKind="explicit" dfdl:nilValueDelimiterPolicy="none" ibmDfdlExtn:sampleValue="aaaaa" name="structureId">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:minLength value="5"/>
<xs:maxLength value="5"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="5" dfdl:lengthKind="explicit" dfdl:nilValueDelimiterPolicy="none" ibmDfdlExtn:sampleValue="aaa" name="structureLength">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:minLength value="5"/>
<xs:maxLength value="5"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="5" dfdl:lengthKind="explicit" dfdl:nilValueDelimiterPolicy="none" ibmDfdlExtn:sampleValue="aaaaa" name="unknownId">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:minLength value="5"/>
<xs:maxLength value="5"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="{xs:nonNegativeInteger(/SWEIO/Structure11013/structureLength)-16}" dfdl:lengthKind="explicit" dfdl:nilValueDelimiterPolicy="none" ibmDfdlExtn:maxOccurs="1" ibmDfdlExtn:sampleValue="aaaaaaaaaaaaaaaaaaaa" name="messageStream">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:minLength value="0"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>
|
I'm trying to define the different structures. How can I define the length of
| Code: |
| <xs:element dfdl:emptyValueDelimiterPolicy="none" dfdl:initiator="" dfdl:lengthKind="implicit" name="Structure14332"> |
so that its explicit and points to
| Code: |
| <xs:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="5" dfdl:lengthKind="explicit" dfdl:nilValueDelimiterPolicy="none" ibmDfdlExtn:sampleValue="aaa" name="structureLength"> |
?
The way the schema is defined at the moment, the structures have to be in a fixed order. How can I define them, so that the order can vary? |
|
| Back to top |
|
|
| kimbert |
| Posted: Wed Jul 03, 2013 1:28 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
OK - at least two separate questions there.
1. How to define the length of element 'Structure14332' so that it is dynamically calculated from its child element 'structureLength'
When you use a DFDL expression to calculate a length, the length must be evaluated immediately. So the length expression cannot refer to a child of the element. The solution is to put the first few fields of 'Structure14332' into an element 'structureHeader' and the remaining fields into element 'StructureBody'. Then you can calculate the length of 'StructureBody' using the ( already-parsed ) 'structureLength' field. See my previous post for the correct syntax. Remember to adjust the expression to subtract the length of the header fields if necessary.
2. How to handle unordered data
IBM DFDL does not yet support sequenceKind='unordered' ( see the other thread on this forum ). But you can simulate it by creating a choice group that contains all of the fields. Wrap the choice group in an element with maxOccurs='unbounded' and you should be OK. |
|
| Back to top |
|
|
| ottfried11 |
| Posted: Wed Jul 03, 2013 10:24 pm Post subject: |
|
|
Apprentice
Joined: 28 Jun 2013
Posts: 29
|
| thx a lot. sounds good. I'll try it right away |
|
| Back to top |
|
|
| ottfried11 |
| Posted: Fri Jul 05, 2013 2:42 am Post subject: |
|
|
Apprentice
Joined: 28 Jun 2013
Posts: 29
|
I got the structure part working. Now I'm having a look at the body of each structure.
| Quote: |
structure body :
7003EUL -> the first character is always the domain (0 = Short, 1 = Long, 2 = Real, 3 = Zeit, 4 = Datum, 6 = Bool., 7 = String, 8 = Key) I'm not sure what 5 stands for. It does not really matter
-> characters 2 - 4 = field length: here the length is 3. when the field is empty, he lenght of the field is described with DDD
-> field value.
|
I solved the problem the following way:
| Quote: |
| #1101300067240007008K981895I7DDD7DDD7017GB w/ Kumulierung8002LZ7DDD |
can be parsed by
| Code: |
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:initiator="" dfdl:lengthKind="implicit" name="structureBody11013">
<xsd:complexType>
<xsd:sequence dfdl:separator="" dfdl:sequenceKind="ordered">
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="{xs:nonNegativeInteger(../../structureHeader/structureLength)-16}" dfdl:lengthKind="explicit" dfdl:nilValueDelimiterPolicy="none" ibmDfdlExtn:maxOccurs="1" ibmDfdlExtn:sampleValue="aaaaaaaaaaaaaaaaaaaa" name="messageStream">
<xsd:complexType>
<xsd:sequence dfdl:separator="" dfdl:sequenceKind="ordered">
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="1" dfdl:lengthKind="explicit" name="KONTRKuerzelDomain" type="xsd:string"/>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="3" dfdl:lengthKind="explicit" name="KONTRKuerzelLength" type="xsd:string"/>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="{xs:nonNegativeInteger(../KONTRKuerzelLength)}" dfdl:lengthKind="explicit" minOccurs="1" name="KONTRKuerzelValue" type="xsd:string"/>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="1" dfdl:lengthKind="explicit" name="KTAUFWeitere_WeisungenDomain" type="xsd:string"/>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="3" dfdl:lengthKind="explicit" name="KTAUFWeitere_WeisungenLength" type="xsd:string"/>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="{xs:nonNegativeInteger(../KTAUFWeitere_WeisungenLength)}" dfdl:lengthKind="explicit" dfdl:occursCountKind="implicit" minOccurs="0" name="KTAUFWeitere_WeisungenValue" type="xsd:string"/>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="1" dfdl:lengthKind="explicit" name="KTAUFMitteilung_KntrhntDomain" type="xsd:string"/>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="3" dfdl:lengthKind="explicit" name="KTAUFMitteilung_KntrhntLength" type="xsd:string"/>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="{xs:nonNegativeInteger(../KTAUFMitteilung_KntrhntLength)}" dfdl:lengthKind="explicit" dfdl:occursCountKind="implicit" minOccurs="0" name="KTAUFKTAUFMitteilung_KntrhntValue" type="xsd:string"/>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="1" dfdl:lengthKind="explicit" name="SETLVBezeichnungDomain" type="xsd:string"/>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="3" dfdl:lengthKind="explicit" name="SETLVBezeichnungLength" type="xsd:string"/>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="{xs:nonNegativeInteger(../SETLVBezeichnungLength)}" dfdl:lengthKind="explicit" dfdl:occursCountKind="implicit" minOccurs="0" name="SETLVBezeichnungValue" type="xsd:string"/>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="1" dfdl:lengthKind="explicit" name="SETLVSettlement_ArtDomain" type="xsd:string"/>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="3" dfdl:lengthKind="explicit" name="SETLVSettlement_ArtLength" type="xsd:string"/>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="{xs:nonNegativeInteger(../SETLVSettlement_ArtLength)}" dfdl:lengthKind="explicit" dfdl:occursCountKind="implicit" minOccurs="0" name="SETLVSettlement_ArtValue" type="xsd:string"/>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="1" dfdl:lengthKind="explicit" name="KTAUFSettlement_TextDomain" type="xsd:string"/>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="3" dfdl:lengthKind="explicit" name="KTAUFSettlement_TextLength" type="xsd:string"/>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="{xs:nonNegativeInteger(../KTAUFSettlement_TextLength)}" dfdl:lengthKind="explicit" dfdl:occursCountKind="implicit" minOccurs="0" name="KTAUFSettlement_TextValue" type="xsd:string"/>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
|
I was not sure how to solve this with a prefixed length type.
1. Is there another, more simpler way of doing it?
2. Can I run into problems with the length of DDD? It seems that nonNegativeInteger of DDD equals 0. |
|
| Back to top |
|
|
| kimbert |
| Posted: Fri Jul 05, 2013 5:32 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| I was not sure how to solve this with a prefixed length type |
That's because it's not really a prefixed-length format. It's a data format that contains a length field, which is almost, but not quite, the same thing. This is exactly why DFDL allows the 'length' property to be a DFDL expression.
| Quote: |
| Is there another, more simpler way of doing it? |
Yes - you can define a global complex type that contains three child elements with names 'type', 'length' and 'value'. Then your structure definition will be a simple list of named elements that use that complex type.
If I was doing it, I would define 7 variants of the complex type - one for each simple type value in the 'type' field. I would also put some enumeration facets or a maxInclusive facet on the 'type' field.
| Quote: |
| Can I run into problems with the length of DDD? It seems that nonNegativeInteger of DDD equals 0. |
Yes, you can run into problems. You should really be getting an error when you try to cast 'DDD' to an integer. I suggest that you change your DFDL expression to
| Code: |
| xs:nonNegativeInteger(if (../fieldLength eq 'DDD') then 0 else ../fieldLength) |
|
|
| Back to top |
|
|
|
|
