| Author |
Message
|
| ottfried11 |
 Posted: Tue Mar 18, 2014 8:14 am Post subject: DFDL parse/encoding problem |
 |
|
Apprentice
Joined: 28 Jun 2013
Posts: 29
|
Hi all,
I have the following problem:
I am using DFDL to parse FIX 4.2 (fixprotocol.org) messages. It works fine, as long as I don't have any special characters in a text field.
A FIX message is set up as follows:
Tagnumber equals value delimited by hex01.
The message starts with a header containing version and length of message (without header and checksum), then follows the message body followed by a checksum value.
Here an example:
| Quote: |
| 8=FIX.4.29=27835=D52=20140318-13:15:1349=HSBCTUBO56=HSBCGFIX57=HBCAN115=TRINDUS11=GEOE2P9254602458=!"$§%&/()=?{[]}\+*~#,.-;:_|ÄÜÖäüö1=5504028621=3100=TO207=TO55=CA067901108448=CA067901108422=454=260=20140225-09:21:2138=33315=CAD40=259=6126=20141231-23:59:5944=3010=250 |
In this example, tag 58 with the testsvalues !"$§%&/()=?{[]}\+*~#,.-;:_|ÄÜÖäüö is my problem. The number of characters is 33. The byte length is 40 because of the double values §ÄÖÜäöü.
I want my dfdl only to count the character length units.
Here my code:
| Code: |
<?xml version="1.0" encoding="UTF-8"?><xsd:schema xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/" xmlns:ibmDfdlExtn="http://www.ibm.com/dfdl/extensions" xmlns:ibmSchExtn="http://www.ibm.com/schema/extensions" xmlns:recSepFieldsFmt="http://www.ibm.com/dfdl/RecordSeparatedFieldFormat" xmlns:tns="http://www.ibm.com/dfdl/ISO8583Types" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<xsd:import namespace="http://www.ibm.com/dfdl/RecordSeparatedFieldFormat" schemaLocation="IBMdefined/RecordSeparatedFieldFormat.xsd"/>
<xsd:annotation>
<xsd:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:format byteOrder="{$dfdl:byteOrder}" encoding="{$dfdl:encoding}" escapeSchemeRef="recSepFieldsFmt:RecordEscapeScheme" occursCountKind="fixed" ref="recSepFieldsFmt:RecordSeparatedFieldsFormat"/>
</xsd:appinfo>
</xsd:annotation>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:initiator="" dfdl:lengthKind="implicit" ibmDfdlExtn:docRoot="true" name="FIX">
<xsd:complexType>
<xsd:sequence dfdl:separator="">
<xsd:element dfdl:lengthKind="implicit" dfdl:occursCountKind="implicit" maxOccurs="unbounded" minOccurs="1" name="FIXchoiceElement">
<xsd:complexType>
<xsd:choice dfdl:initiatedContent="yes">
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:initiator="8=" dfdl:lengthKind="implicit" name="FIXMessage">
<xsd:complexType>
<xsd:sequence dfdl:separator="" dfdl:sequenceKind="ordered">
<xsd:element ref="messageHeader"/>
<xsd:element ref="messageBody"/>
<xsd:element ref="CheckSum"/>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
</xsd:choice>
</xsd:complexType>
</xsd:element>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:initiator="" dfdl:lengthKind="implicit" name="messageHeader">
<xsd:complexType>
<xsd:sequence dfdl:separator="%SOH;" dfdl:sequenceKind="ordered">
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:lengthKind="delimited" name="BeginString">
<xsd:simpleType>
<xsd:restriction base="xsd:string">
<xsd:minLength value="0"/>
</xsd:restriction>
</xsd:simpleType>
</xsd:element>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:initiator="9=" dfdl:lengthKind="delimited" dfdl:terminator="%SOH;" name="BodyLength">
<xsd:simpleType>
<xsd:restriction base="xsd:string">
<xsd:minLength value="0"/>
</xsd:restriction>
</xsd:simpleType>
</xsd:element>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="{xs:nonNegativeInteger(../messageHeader/BodyLength)}" dfdl:lengthKind="explicit" dfdl:nilValueDelimiterPolicy="none" ibmDfdlExtn:maxOccurs="1" name="messageBody">
<xsd:complexType>
<xsd:sequence dfdl:separator="">
<xsd:element dfdl:lengthKind="implicit" dfdl:occursCountKind="implicit" maxOccurs="unbounded" minOccurs="1" name="choiceElement">
<xsd:complexType>
<xsd:choice dfdl:initiatedContent="yes">
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:initiator="1=" dfdl:lengthKind="delimited" dfdl:terminator="%SOH;" minOccurs="1" name="Account" type="xsd:string"/>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:initiator="2=" dfdl:lengthKind="delimited" dfdl:terminator="%SOH;" minOccurs="1" name="AdvId" type="xsd:string"/>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:initiator="3=" dfdl:lengthKind="delimited" dfdl:terminator="%SOH;" minOccurs="1" name="AdvRefID" type="xsd:string"/>
....
....
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:initiator="58=" dfdl:lengthKind="delimited" dfdl:terminator="%SOH;" minOccurs="1" name="Text" type="xsd:string"/>
....
....
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:initiator="446=" dfdl:lengthKind="delimited" dfdl:terminator="%SOH;" minOccurs="1" name="EncodedListStatusText" type="xsd:string"/> </xsd:choice>
</xsd:complexType>
</xsd:element>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:initiator="10=" dfdl:lengthKind="delimited" dfdl:terminator="%SOH;" name="CheckSum">
<xsd:simpleType>
<xsd:restriction base="xsd:string">
<xsd:minLength value="0"/>
</xsd:restriction>
</xsd:simpleType>
</xsd:element>
</xsd:schema>
|
The default encoding="{$dfdl:encoding}" is 1208 (UTF-8).
The error occurs also in the WMB DFDL Test - Parse. So it can easily be reproduced.
I would be very greatful for any help. |
|
| Back to top |
|
 |
| kimbert |
| Posted: Wed Mar 19, 2014 2:49 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
This one sounds quite any easy one. You need to set the DFDL property 'lengthUnits' to 'characters'. I expect it is being set to 'bytes' in RecordSeparatedFieldFormat.xsd.
If I don't stop you, you will now go and edit RecordSeparatedFieldFormat.xsd. That would be the wrong option. The defaults for the DFDL properties in your xsd are in this annotation:
| Code: |
<xsd:annotation>
<xsd:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:format byteOrder="{$dfdl:byteOrder}" encoding="{$dfdl:encoding}" escapeSchemeRef="recSepFieldsFmt:RecordEscapeScheme" occursCountKind="fixed" ref="recSepFieldsFmt:RecordSeparatedFieldsFormat"/>
</xsd:appinfo>
</xsd:annotation> |
So you should add 'lengthUnits="characters" after the settings for byteOrder and encoding. That will ensure that all elements in your xsd inherit that setting.
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| ottfried11 |
| Posted: Wed Mar 19, 2014 3:30 am Post subject: |
|
|
Apprentice
Joined: 28 Jun 2013
Posts: 29
|
Hi Kimbert,
thanks for the response.
when runnig the message flow with the new DFDL, I still get the same error message:
| Quote: |
Text:CHARACTER:CTDU4023E: The DFDL serializer cannot output the physical representation of element '#xscd(/schemaElement::messageBody/type::0/model::sequence/schemaElement::choiceElement/type::0/model::choice/schemaElement::Text)' at offset '329'. The length '83' in units of 'bytes' exceeds the maximum allowed length of parent element '#xscd(/schemaElement::messageBody)'.
|
The length 83 refers to the text field, which is of length 79 measured in characters.
| Quote: |
Sonderzeichen Mitteilung Kontrahent: äöü!"§$%&/()=?`;:_,.-#'+*~\r\n@<>|{[]}\\´
|
When testing the new DFDL with the DFDL Test - Parse eclipse throws the exception
| Quote: |
Problems occurred when invoking code from plug-in: "org.eclipse.core.resources".
java.lang.NullPointerException
at org.eclipse.xsd.util.XSDResourceImpl.getEObject(Unknown Source)
at com.ibm.dfdl.internal.ui.properties.DFDLMarkerManager.getSchemaObject(Unknown Source)
at com.ibm.dfdl.internal.ui.properties.DFDLMarkerManager.registerMarker(Unknown Source)
at com.ibm.dfdl.internal.ui.properties.DFDLMarkerManager.addMarker(Unknown Source)
at com.ibm.dfdl.internal.ui.properties.DFDLMarkerManager$1.processDelta(Unknown Source)
at com.ibm.dfdl.internal.ui.properties.DFDLMarkerManager$1.handleDelta(Unknown Source)
at com.ibm.dfdl.internal.ui.properties.DFDLMarkerManager$1.handleDelta(Unknown Source)
at com.ibm.dfdl.internal.ui.properties.DFDLMarkerManager$1.handleDelta(Unknown Source)
at com.ibm.dfdl.internal.ui.properties.DFDLMarkerManager$1.resourceChanged(Unknown Source)
at org.eclipse.core.internal.events.NotificationManager$2.run(Unknown Source)
at org.eclipse.core.runtime.SafeRunner.run(Unknown Source)
at org.eclipse.core.internal.events.NotificationManager.notify(Unknown Source)
at org.eclipse.core.internal.events.NotificationManager.broadcastChanges(Unknown Source)
at org.eclipse.core.internal.resources.Workspace.broadcastPostChange(Unknown Source)
at org.eclipse.core.internal.resources.Workspace.endOperation(Unknown Source)
at org.eclipse.core.internal.resources.InternalWorkspaceJob.run(Unknown Source)
at org.eclipse.core.internal.jobs.Worker.run(Unknown Source)
|
It seems, that the lengthUnits definition is ignored in the annotation
| Code: |
<dfdl:format byteOrder="{$dfdl:byteOrder}" encoding="{$dfdl:encoding}" lengthUnits="characters" escapeSchemeRef="recSepFieldsFmt:RecordEscapeScheme" occursCountKind="fixed" ref="recSepFieldsFmt:RecordSeparatedFieldsFormat"/>
|
|
|
| Back to top |
|
|
| kimbert |
| Posted: Wed Mar 19, 2014 4:11 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
Sounds like a defect, but it's hard to be sure without actually trying it out.
Have you checked that the DFDL editor correctly shows 'Length Units' as 'Characters' throughout the xsd?
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| ottfried11 |
| Posted: Wed Mar 19, 2014 6:35 am Post subject: |
|
|
Apprentice
Joined: 28 Jun 2013
Posts: 29
|
The DFDL Test -Parse exception came from the leading quotation mark in the tag 58 textfield. After removing that, it worked fine.
The message broker message flow still does not work though.
What irritates me, is the error note
| Quote: |
units of 'bytes' exceeds the maximum allowed length
|
| Quote: |
Text:CHARACTER:CTDU4023E: The DFDL serializer cannot output the physical representation of element '#xscd(/schemaElement::messageBody/type::0/model::sequence/schemaElement::choiceElement/type::0/model::choice/schemaElement::Text)' at offset '329'. The length '83' in units of 'bytes' exceeds the maximum allowed length of parent element '#xscd(/schemaElement::messageBody)'.
|
I defined every element with lengthUnits="characters" and not as "bytes"
| Code: |
<xsd:import namespace="http://www.ibm.com/dfdl/RecordSeparatedFieldFormat" schemaLocation="IBMdefined/RecordSeparatedFieldFormat.xsd"/>
<xsd:annotation>
<xsd:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:format byteOrder="{$dfdl:byteOrder}" encoding="{$dfdl:encoding}" lengthUnits="characters" escapeSchemeRef="recSepFieldsFmt:RecordEscapeScheme" occursCountKind="fixed" ref="recSepFieldsFmt:RecordSeparatedFieldsFormat"/>
</xsd:appinfo>
</xsd:annotation>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:initiator="" dfdl:lengthKind="implicit" ibmDfdlExtn:docRoot="true" name="FIX">
<xsd:complexType>
<xsd:sequence dfdl:separator="">
<xsd:element dfdl:lengthKind="implicit" dfdl:occursCountKind="implicit" maxOccurs="unbounded" minOccurs="1" name="FIXchoiceElement">
<xsd:complexType>
<xsd:choice dfdl:initiatedContent="yes">
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:initiator="8=" dfdl:lengthKind="implicit" name="FIXMessage">
<xsd:complexType>
<xsd:sequence dfdl:separator="" dfdl:sequenceKind="ordered">
<xsd:element ref="messageHeader"/>
<xsd:element ref="messageBody"/>
<xsd:element ref="CheckSum"/>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
</xsd:choice>
</xsd:complexType>
</xsd:element>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
<!-- Global elements for re-use -->
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:initiator="" dfdl:lengthKind="implicit" name="messageHeader">
<xsd:complexType>
<xsd:sequence dfdl:separator="%SOH;" dfdl:sequenceKind="ordered">
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:lengthKind="delimited" dfdl:lengthUnits="characters" name="BeginString">
<xsd:simpleType>
<xsd:restriction base="xsd:string">
<xsd:minLength value="0"/>
</xsd:restriction>
</xsd:simpleType>
</xsd:element>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:initiator="9=" dfdl:lengthKind="delimited" dfdl:lengthUnits="characters" dfdl:terminator="%SOH;" name="BodyLength">
<xsd:simpleType>
<xsd:restriction base="xsd:string">
<xsd:minLength value="0"/>
</xsd:restriction>
</xsd:simpleType>
</xsd:element>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
<!--xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:lengthKind="implicit" dfdl:maxLength="{xs:nonNegativeInteger(../messageHeader/BodyLength)}" dfdl:nilValueDelimiterPolicy="none" ibmDfdlExtn:maxOccurs="1" name="messageBody"-->
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:length="{xs:nonNegativeInteger(../messageHeader/BodyLength)}" dfdl:lengthKind="explicit" dfdl:lengthUnits="characters" dfdl:nilValueDelimiterPolicy="none" ibmDfdlExtn:maxOccurs="1" name="messageBody">
<xsd:complexType>
<xsd:sequence dfdl:separator="">
<xsd:element dfdl:lengthKind="implicit" dfdl:lengthUnits="characters" dfdl:occursCountKind="implicit" maxOccurs="unbounded" minOccurs="1" name="choiceElement">
<xsd:complexType>
<xsd:choice dfdl:initiatedContent="yes">
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:initiator="1=" dfdl:lengthKind="delimited" dfdl:lengthUnits="characters" dfdl:terminator="%SOH;" minOccurs="1" name="Account" type="xsd:string"/>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:initiator="2=" dfdl:lengthKind="delimited" dfdl:lengthUnits="characters" dfdl:terminator="%SOH;" minOccurs="1" name="AdvId" type="xsd:string"/>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:initiator="3=" dfdl:lengthKind="delimited" dfdl:lengthUnits="characters" dfdl:terminator="%SOH;" minOccurs="1" name="AdvRefID" type="xsd:string"/>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:initiator="4=" dfdl:lengthKind="delimited" dfdl:lengthUnits="characters" dfdl:terminator="%SOH;" minOccurs="1" name="AdvSide" type="xsd:string"/>
.....
|
In the DFDL editor, the lengthUnit = Characters in the above configuration, only shows in the messageBody. The messageBody is the only field with lengthKind = explicit. The rest is either implicit or delimited.
Either the lengthUnits are ignored by the parser or I still have an encoding problem. |
|
| Back to top |
|
|
| kimbert |
| Posted: Fri Mar 21, 2014 5:38 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
|
| Back to top |
|
|
| ottfried11 |
| Posted: Fri Mar 21, 2014 5:44 am Post subject: |
|
|
Apprentice
Joined: 28 Jun 2013
Posts: 29
|
I set the codepage to a windows CCSID 1252 and removed CRLF from the message before parsing it. It works, but its not very nice.
I'll open a PMR.
Thanks for your help, again  |
|
| Back to top |
|
|
| kimbert |
| Posted: Fri Mar 21, 2014 9:03 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| I set the codepage to a windows CCSID 1252 |
That is a fixed-width ( single-byte) encoding. That's why it appears to work. It will not work correctly if the element contains any multi-byte characters.
Does the complex element really need to define its length explicitly? It looks as if every branch of the choice group has a terminator, so the length may be redundant. Removing the explicit length would avoid the problem.
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| ottfried11 |
| Posted: Mon Mar 24, 2014 12:14 am Post subject: |
|
|
Apprentice
Joined: 28 Jun 2013
Posts: 29
|
It works for the multi-byte "german Umlaute" such as äöü. CRLF did nor work, so I replaced it. I've tested it with all possible values which can be entered by the users.
The problem is, I have to define a correct FIX message. The FIX tag BodyLength contains the message length. When validating the message, the set length has to be correct. Otherwise, the receiving party will not be able to handle the message. |
|
| Back to top |
|
|
| Senthamizh |
| Posted: Fri Jul 18, 2014 12:12 am Post subject: |
|
|
Apprentice
Joined: 21 Dec 2009
Posts: 47
|
Are there any updates on the PMR. I am facing a similar issue.
I just have to parse a single line of 74 characters. This is the xsd
| Code: |
<?xml version="1.0" encoding="UTF-8"?><!-- edited with XMLSpy v2013 sp1 (http://www.altova.com) --><xsd:schema xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/" xmlns:fmt="http://www.ibm.com/dfdl/GeneralPurposeFormat" xmlns:ibmSchExtn="http://www.ibm.com/schema/extensions" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<xsd:import namespace="http://www.ibm.com/dfdl/GeneralPurposeFormat" schemaLocation="IBMdefined/GeneralPurposeFormat.xsd"/>
<xsd:import namespace="http://www.ibm.com/dfdl/GeneralPurposeFormat" schemaLocation="IBMdefined/GeneralPurposeFormat.xsd"/>
<xsd:element dfdl:byteOrder="{$dfdl:byteOrder}" dfdl:emptyValueDelimiterPolicy="none" dfdl:encoding="UTF-16" dfdl:length="74" dfdl:lengthKind="explicit" dfdl:lengthUnits="characters" dfdl:outputNewLine="{%CR;%LF;}" dfdl:utf16Width="variable" ibmSchExtn:docRoot="true" name="Message">
<xsd:complexType>
<xsd:sequence dfdl:byteOrder="bigEndian" dfdl:separator="">
<xsd:element dfdl:byteOrder="{$dfdl:byteOrder}" dfdl:encoding="UTF-16" dfdl:length="1" dfdl:lengthKind="explicit" dfdl:lengthUnits="characters" dfdl:textPadKind="padChar" dfdl:utf16Width="variable" name="Flow_Code" type="xsd:string">
<xsd:annotation/>
</xsd:element>
<xsd:element dfdl:byteOrder="{$dfdl:byteOrder}" dfdl:encoding="UTF-16" dfdl:length="15" dfdl:lengthKind="explicit" dfdl:lengthUnits="characters" dfdl:textPadKind="padChar" dfdl:utf16Width="variable" name="IMP" type="xsd:string"/>
<xsd:element dfdl:byteOrder="{$dfdl:byteOrder}" dfdl:encoding="UTF-16" dfdl:length="9" dfdl:lengthKind="explicit" dfdl:lengthUnits="characters" dfdl:textPadKind="padChar" dfdl:utf16Width="variable" name="Ticket_Nr" type="xsd:string"/>
<xsd:element dfdl:byteOrder="{$dfdl:byteOrder}" dfdl:encoding="UTF-16" dfdl:length="6" dfdl:lengthKind="explicit" dfdl:lengthUnits="characters" dfdl:textPadKind="padChar" dfdl:utf16Width="variable" name="NIP" type="xsd:string"/>
<xsd:element dfdl:byteOrder="{$dfdl:byteOrder}" dfdl:encoding="UTF-16" dfdl:length="5" dfdl:lengthKind="explicit" dfdl:lengthUnits="characters" dfdl:textPadKind="padChar" dfdl:utf16Width="variable" name="NIF" type="xsd:string"/>
<xsd:element dfdl:byteOrder="{$dfdl:byteOrder}" dfdl:encoding="UTF-16" dfdl:length="10" dfdl:lengthKind="explicit" dfdl:lengthUnits="characters" dfdl:outputNewLine="{$dfdl:outputNewLine}" dfdl:textPadKind="padChar" dfdl:utf16Width="variable" name="Lot_Nr" type="xsd:string"/>
<xsd:element dfdl:byteOrder="{$dfdl:byteOrder}" dfdl:encoding="UTF-16" dfdl:length="6" dfdl:lengthKind="explicit" dfdl:lengthUnits="characters" dfdl:textPadKind="padChar" dfdl:utf16Width="variable" name="UDC_Quantity" type="xsd:string"/>
<xsd:element dfdl:byteOrder="{$dfdl:byteOrder}" dfdl:encoding="UTF-16" dfdl:length="2" dfdl:lengthKind="explicit" dfdl:lengthUnits="characters" dfdl:textPadKind="padChar" dfdl:utf16Width="variable" name="UDC_Type" type="xsd:string"/>
<xsd:element dfdl:byteOrder="{$dfdl:byteOrder}" dfdl:encoding="UTF-16" dfdl:length="12" dfdl:lengthKind="explicit" dfdl:lengthUnits="characters" dfdl:textPadKind="padChar" dfdl:utf16Width="variable" name="Container_Id" type="xsd:string"/>
<xsd:element dfdl:byteOrder="{$dfdl:byteOrder}" dfdl:encoding="UTF-16" dfdl:length="8" dfdl:lengthKind="explicit" dfdl:lengthUnits="characters" dfdl:textPadKind="padChar" dfdl:utf16Width="variable" name="HDV_Date" type="xsd:string"/>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
<xsd:annotation>
<xsd:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:format encoding="UTF-16" length="74" lengthKind="explicit" ref="fmt:GeneralPurposeFormat" separator=""/>
<dfdl:defineFormat name="Group_Terminator">
<dfdl:format encoding="UTF-16" length="74" lengthKind="explicit" lengthUnits="characters" representation="text" terminator="{/Message/HDV_Date}"/>
</dfdl:defineFormat>
</xsd:appinfo>
</xsd:annotation>
</xsd:schema> |
Not sure why the broker is using byte count for validation when it is specified as character
| Code: |
Text:CHARACTER:CTDU4023E: The DFDL serializer cannot output the physical representation of element '#xscd(/schemaElement::Message/type::0/model::sequence/schemaElement::Lot_Nr)' at offset '92'. The length '20' in units of 'bytes' exceeds the maximum allowed length of parent element '#xscd(/schemaElement::Message)'.
|
|
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Jul 18, 2014 4:56 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20697
Location: LI,NY
|
It looks like the length of the parent element of LotNr has been defined as explicit and the length of the child element LotNr causes that length to be exceeded.
You have defined the message to have a total length of 74 char and the system is telling you that you will exceed that length... Remember that 74 should include the length of all prefixes and suffixes/ initiators and terminators on top of the length of the elements...
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| kimbert |
| Posted: Fri Jul 18, 2014 6:37 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
Please note this question/advice:
| Quote: |
| Does the complex element really need to define its length explicitly? It looks as if every branch of the choice group has a terminator, so the length may be redundant. Removing the explicit length would avoid the problem. |
I have noticed that people routinely put lengths on their structures when it is not necessary. I understand how it happens ( lengths were there in the source data, and get copied into the schema along with everything else ). But it does make extra work for the parser, and in this case it also triggers a bug in the DFDL component.
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| Senthamizh |
| Posted: Sun Jul 20, 2014 1:14 pm Post subject: |
|
|
Apprentice
Joined: 21 Dec 2009
Posts: 47
|
Thanks for your replies..
Infact what am trying to do is to convert a UTF8 xml into a UTF 16 text file.
| Code: |
.Z.P.G.0 .2.3.2.1 005A0050 00470030 00320033 00320031
.R.F. . . . . . 00520046 00200020 00200020 00200020
.E.T.Q.P .O.L.0.0 00450054 00510050 004F004C 00300030
.1.0.2.3 .2.1.0.0 00310030 00320033 00320031 00300030
.6.9.0.0 .L.O.T.P 00360039 00300030 004C004F 00540050
.O.L.0.0 .0.1.9.5 004F004C 00300030 00300031 00390035
.0. . . .B.B.C.N 00300020 00200020 00420042 0043004E
.T.P.O.L .0.0.0.0 00540050 004F004C 00300030 00300030
. . .0.7 .1.2.2.0 00200020 00300037 00310032 00320030
.0.8 00300038
|
Infact the DFDL i created is parsing this text file correctly when i do it stand alone. The is the outpiut from an RFHutil which contains both character and hex values.. Here 1 character is taken as 2 bytes...
But when i try to convert the UFT8 to UTF16 in ESQL using some CAST statments, it failing becuase i do CAST as char..
| Code: |
SET OutputRoot.DFDL.Message.Flow_Code = CAST(CAST(flowcode AS BLOB CCSID 1200 ENCODING InputRoot.Properties.Encoding) AS CHAR ENCODING InputRoot.Properties.Encoding);
|
Here I try to cast as BLOB and then back to CHAR to change it to UT16, infact, the conversion is good, but since is CAST it to char, the DFDL expects 2 character and not 2 bytes(1char)..
Is there another way to do this casting and parse the DFDL message ? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Jul 21, 2014 4:33 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20697
Location: LI,NY
|
There seems to be some misconception here. Why would you need to parse, then cast to blob and then parse again in a different CCSID from within the same flow?
This makes absolutely no sense. The first parse was sufficient.
Please explain in more details.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Senthamizh |
| Posted: Mon Jul 21, 2014 4:49 am Post subject: |
|
|
Apprentice
Joined: 21 Dec 2009
Posts: 47
|
Infact my incoming XML is UTF-8. As i test it in my local laptop, by default it takes the CCSID as 850.
I map some values from this XML to a text file.. The text file need to be UTF-16.
Hence the reason , i made the DFDL encodoing to UTF-16. and gave lengthunits as characters ..
Hence, here i try to map the UTF-8 to UTF-16 using the CAST functions.
Initially I tried to just set the OutputRoot.Properties.CCSID as 1200 and make a map, but it fails ..
hence, i tried to explicitly make the CAST.. |
|
| Back to top |
|
|
|
|
