ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker SupportDFDL Numbers Parsing Error

Post new topicReply to topic Goto page 1, 2  Next
DFDL Numbers Parsing Error View previous topic :: View next topic
Author Message
abooddy
PostPosted: Sat Jun 01, 2019 1:14 am Post subject: DFDL Numbers Parsing Error Reply with quote

Novice

Joined: 28 May 2019
Posts: 11
Location: Baghdad, Iraq

Hi,

We have these lines:

Code:
DECLARE B BLOB;
SET B = ASBITSTREAM(InputRoot.DFDL.ns:ISO8583_1993_Unpacked OPTIONS RootBitStream CCSID 1208);


We're getting the error:

"CTDU4011E: The DFDL serializer cannot output the character '١' in encoding 'US-ASCII' for the value of element 'MTI_Version'."

We clearly see that the "MTI_Version" field value is set to the English representation of the number one "1" not the Arabic one "١".

This issue does not happen on our brokers that run on Linux, only on Windows. We don't want to change the encoding from "US-ASCII" if we can fix it another way so that it keeps working on both Linux and Windows.

I've tried changing the CCSID of the broker from 9448 to 1208 but it didn't help. I tried changing all possible language and region configurations from Arabic but it also didn't help. Below are more information on the issue:





Can you please advice how to fix this issue?

Thank you.
Back to top
View user's profile Send private message
timber
PostPosted: Sun Jun 02, 2019 2:05 pm Post subject: Reply with quote

Guardian

Joined: 25 Aug 2015
Posts: 984

Good problem description. DFDL is designed to be latform-independent (no platform defaults) so this is hard to explain.

How is the DFDL property 'encoding' applied to the MTI_Version field? Is it defined on the field itself, or inherited from a DFDL format block? ( just quote the field definition from the DFDL XSD if you don't know)

It might be useful to see the DFDL Trace output because it would show how the DFDL parser is choosing the encoding for this field. If you enable debug-level user trace, the DFDL trace is written into the user trace output. Let me know if you need instructions on how to get a user trace. (Please quote only the relevant lines, not the entire trace!)
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Tue Jun 04, 2019 5:01 am Post subject: Reply with quote

Grand Poobah

Joined: 18 Nov 2003
Posts: 19978
Location: LI,NY

It's an unpacked number, decimal, 1 and the representation is TEXT????
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
abooddy
PostPosted: Wed Jun 05, 2019 1:41 am Post subject: Reply with quote

Novice

Joined: 28 May 2019
Posts: 11
Location: Baghdad, Iraq

Hi,

This is the definition of the element from the DFDL XSD:

Code:
<xs:element dfdl:length="1" dfdl:textNumberPattern="#0" ibmDfdlExtn:sampleValue="1" name="MTI_Version" type="xs:integer"/>


And this is the text content section:



I didn't find any encoding options defined in the XSD.
Back to top
View user's profile Send private message
timber
PostPosted: Wed Jun 05, 2019 3:36 am Post subject: Reply with quote

Guardian

Joined: 25 Aug 2015
Posts: 984

If the element does not specify its encoding then it will be inherited from the default DFDL format for the schema. You should find that either at the top or the bottom of the XSD. It will look something like this:
Code:
    <xsd:annotation>
      <xsd:appinfo source="http://www.ogf.org/dfdl/">
         <dfdl:format emptyValueDelimiterPolicy="initiator" encoding="{$dfdl:encoding}" escapeSchemeRef="" ref="fmt:CobolDataFormat" textPadKind="none" textTrimKind="none"/>
      </xsd:appinfo>
   </xsd:annotation>

I expect that it will say encoding="US-ASCII", but I would like to be sure.
Back to top
View user's profile Send private message
abooddy
PostPosted: Wed Jun 05, 2019 7:33 am Post subject: Reply with quote

Novice

Joined: 28 May 2019
Posts: 11
Location: Baghdad, Iraq

This is the block in our XSD:

Code:
<xs:annotation>
        <xs:appinfo source="http://www.ogf.org/dfdl/">
       <!-- Format annotation to apply default property values to all objects -->
            <dfdl:format ref="isofmt:ISO8583Format"/>
        </xs:appinfo>
    </xs:annotation>


Because encoding is not defined even here so it is falling back to some parameter at the OS level?
Back to top
View user's profile Send private message
timber
PostPosted: Wed Jun 05, 2019 12:49 pm Post subject: Reply with quote

Guardian

Joined: 25 Aug 2015
Posts: 984

Good guess, but wrong. DFDL never uses platform/OS defaults. If a required property is not defined somewhere then DFDL issues an error. It never constructs a default value for the property.

In this case, the encoding in this default format is being inherited from another format block. It will be defined in an IBM-supplied xsd. Search for the text 'ISO8583Format' in *.xsd and you will find it.
Back to top
View user's profile Send private message
abooddy
PostPosted: Fri Jun 07, 2019 3:50 am Post subject: Reply with quote

Novice

Joined: 28 May 2019
Posts: 11
Location: Baghdad, Iraq

Hi,

We found this:

Code:
         <dfdl:defineFormat name="ISO8583Format">
            <dfdl:format encoding="US-ASCII" ignoreCase="no" byteOrder="bigEndian"
               representation="text" textPadKind="none" textTrimKind="padChar" textBidi="no" escapeSchemeRef=""   
               textStringJustification="left" truncateSpecifiedLengthString="no" textStringPadCharacter="%SP;"
               decimalSigned="yes" textNumberRep="standard" textNumberCheckPolicy="lax" textNumberJustification="right"
               textNumberPadCharacter="%SP;" textNumberRoundingMode="roundUp" textNumberRounding="pattern"   
               textZonedSignStyle="asciiStandard"
               textStandardBase="10" textStandardDecimalSeparator="." textStandardExponentCharacter="E"
               textStandardInfinityRep="Inf" textStandardNaNRep="NaN" textStandardGroupingSeparator="," textStandardZeroRep=""
               textBooleanJustification="left" textBooleanPadCharacter="%SP;"
               calendarPatternKind="implicit" calendarPattern="yyyy-MM-dd'T'HH:mm:ss" calendarCheckPolicy="lax" calendarTimeZone="UTC"
               calendarObserveDST="yes" calendarFirstDayOfWeek="Monday" calendarDaysInFirstWeek="4" calendarCenturyStart="53" calendarLanguage="en-US"
               textCalendarJustification="left" textBooleanTrueRep="true" textBooleanFalseRep="false" textCalendarPadCharacter="%SP;"
               sequenceKind="ordered" floating="no" separator="" separatorPolicy="required" separatorPosition="infix"
               choiceLengthKind="implicit" initiatedContent="no" 
               lengthKind="explicit" prefixIncludesPrefixLength="no" lengthUnits="bytes" 
               alignment="1" alignmentUnits="bytes" leadingSkip="0" trailingSkip="0" fillByte="0"
               nilValueDelimiterPolicy="none" emptyValueDelimiterPolicy="none" 
               initiator="" terminator="" documentFinalTerminatorCanBeMissing="no" outputNewLine="%CR;%LF;"
               occursCountKind="expression"
               binaryFloatRep="ieee" binaryNumberCheckPolicy="lax" binaryDecimalVirtualPoint="0">            
            </dfdl:format>
         </dfdl:defineFormat>


Can you please tell us which attribute is causing this problem?
Back to top
View user's profile Send private message
timber
PostPosted: Fri Jun 07, 2019 3:21 pm Post subject: Reply with quote

Guardian

Joined: 25 Aug 2015
Posts: 984

OK - that simply confirms what your screenshot showed. The value for the 'encoding' property is US-ASCII.

Let's examine the facts that we know:
- The DFDL schema assigns US-ASCII as the encoding for MtiVersion
- The error message from DFDL says Cannot output the character '١' in encoding 'US-ASCII' (which is reasonable, because that Unicode character does not exist in US-ASCII)

That leaves two possibilities
1. The message tree received by the DFDL parser contains the character '١'
or
2. The message tree received by the DFDL parser contains the character '1' but DFDL has interpreted it incorrectly as '١'

I think 1. is far more likely than 2. Can you prove to me that your debugger screenshot is showing the exact message tree that DFDL receives?
Back to top
View user's profile Send private message
abooddy
PostPosted: Sat Jun 08, 2019 9:57 pm Post subject: Reply with quote

Novice

Joined: 28 May 2019
Posts: 11
Location: Baghdad, Iraq

Hi,

The input to the DFDL parser is a BLOB, its value:

Code:
31313030fcf6400108e1e400000000001400000031363532313039373230303
030303033323831383230303030303030303030323030303030303030303030
323032303030303030303030323032303036313030303030303631303030303
030323739313335313930333131313535353335323031313139303230353735
343930363939393939393030303030343237393133353939393030303939313
63820202020202020202020202042454155545920504f494e5420434c494e49
433e42454155545920504f494e5420434c494e4949513035343030313030313
130303230303337333630323530303132303236303031353032373030303032
383031323030303030343237393133353336383336383336383032323030363
031364430303030303030303032303033363830343030303132333030313336
383231303130313030313030303434303030


This value is then parsed using this:

Code:
CREATE LASTCHILD OF R DOMAIN('DFDL')
       PARSE(B CCSID InputRoot.Properties.CodedCharSetId TYPE
        '{http://www.ibm.com/dfdl/ISO8583-1993}:ISO8583_1993_Unpacked' OPTIONS RootBitStream);


Where B is the BLOB and R is is a ROW type variable.

This is R as it is in the debugger:

Code:
R
   DFDL
         ISO8583_1993_Unpacked
               MTI_Version:DECIMAL:1
               MTI_MessageClass:DECIMAL:1
               MTI_MessageFunction:DECIMAL:0
               MTI_MessageOrigin:DECIMAL:0
               PrimaryBitmap
                     Bits001to004:INTEGER:15
                     Bits005to008:INTEGER:12
                     Bits009to012:INTEGER:15
                     Bits013to016:INTEGER:6
                     Bits017to020:INTEGER:4
                     Bits021to024:INTEGER:0
                     Bits025to028:INTEGER:0
                     Bits029to032:INTEGER:1
                     Bits033to036:INTEGER:0
                     Bits037to040:INTEGER:8
                     Bits041to044:INTEGER:14
                     Bits045to048:INTEGER:1
                     Bits049to052:INTEGER:14
                     Bits053to056:INTEGER:4
                     Bits057to060:INTEGER:0
                     Bits061to064:INTEGER:0
               SecondaryBitmap
                     Bits065to068:INTEGER:0
                     Bits069to072:INTEGER:0
                     Bits073to076:INTEGER:0
                     Bits077to080:INTEGER:0
                     Bits081to084:INTEGER:0
                     Bits085to088:INTEGER:0
                     Bits089to092:INTEGER:0
                     Bits093to096:INTEGER:0
                     Bits097to100:INTEGER:1
                     Bits101to104:INTEGER:4
                     Bits105to108:INTEGER:0
                     Bits109to112:INTEGER:0
                     Bits113to116:INTEGER:0
                     Bits117to120:INTEGER:0
                     Bits121to124:INTEGER:0
                     Bits125to128:INTEGER:0
               PrimaryAccountNumber_002:CHARACTER:5210972000000328
               ProcessingCode_003:CHARACTER:182000
               AmountTransaction_004:DECIMAL:20000
               AmountReconciliation_005:DECIMAL:20200
               AmountCardHolderBilling_006:DECIMAL:20200
               ConversionRateReconciliation_009:CHARACTER:61000000
               ConversionRateCardholderBilling_010:CHARACTER:61000000
               SystemsTraceAuditNumber_011:CHARACTER:279135
               DateTimeLocalTransaction_012:TIMESTAMP:java.util.GregorianCalendar[time=1552308935000,
               areFieldsSet=true,
               areAllFieldsSet=false,
               lenient=true,zone=sun.util.calendar.ZoneInfo[id="Asia/Baghdad",
               offset=10800000,
               dstSavings=0,
               useDaylight=false,
               transitions=55,lastRule=null],
               firstDayOfWeek=7,
               minimalDaysInFirstWeek=1,
               ERA=?,
               YEAR=2019,MONTH=2,WEEK_OF_YEAR=?,WEEK_OF_MONTH=?,DAY_OF_MONTH=11,
               DAY_OF_YEAR=?,DAY_OF_WEEK=?,
               DAY_OF_WEEK_IN_MONTH=?,AM_PM=1,HOUR=3,HOUR_OF_DAY=15,MINUTE=55,SECOND=35,
               MILLISECOND=0,ZONE_OFFSET=?,DST_OFFSET=?]
               DateExpiration_014:CHARACTER:2011
               DateSettlement_015:DATE:java.util.GregorianCalendar[time=1549314000000,
               areFieldsSet=true,areAllFieldsSet=false,lenient=true,zone=sun.util.calendar.ZoneInfo[id="Asia/Baghdad",
               offset=10800000,dstSavings=0,useDaylight=false,transitions=55,lastRule=null],firstDayOfWeek=7,
               minimalDaysInFirstWeek=1,ERA=?,YEAR=2019,MONTH=1,WEEK_OF_YEAR=?,WEEK_OF_MONTH=?,DAY_OF_MONTH=5,DAY_OF_YEAR=?,
               DAY_OF_WEEK=?,DAY_OF_WEEK_IN_MONTH=?,AM_PM=0,HOUR=0,HOUR_OF_DAY=0,MINUTE=0,SECOND=0,MILLISECOND=?,
               ZONE_OFFSET=?,DST_OFFSET=?]
               MerchantType_018:CHARACTER:7549
               AcquiringInstitutionIdentificationCode_032:CHARACTER:999999
               RetrievalReferenceNumber_037:CHARACTER:000004279135
               CardAcceptorTerminalIdentification_041:CHARACTER:99900099
               CardAcceptorIdentificationCode_042:CHARACTER:168           
               CardAcceptorNameLocation_043:CHARACTER:BEAUTY POINT CLINIC>BEAUTY POINT CLINIIQ
               AdditionalDataPrivate_048:CHARACTER:001001100200373602500120260015027000028012000004279135
               CurrencyCodeTransaction_049:CHARACTER:368
               CurrencyCodeReconciliation_050:CHARACTER:368
               CurrencyCodeCardholderBilling_051:CHARACTER:368
               AmountsAdditional_054:CHARACTER:006016D000000000200368
               ReceivingInstitutionIdentificationCode_100:CHARACTER:0001
               AccountIdentification1_102:CHARACTER:00136821010100100044000


This is the InputRoot:

Code:
Message
   Properties
         MessageSet:CHARACTER:
         MessageType:CHARACTER:{}:TCPGlobalMsg
         MessageFormat:CHARACTER:
         Encoding:INTEGER:546
         CodedCharSetId:INTEGER:437
         Transactional:BOOLEAN:false
         Persistence:BOOLEAN:false
         CreationTime:TIMESTAMP:java.util.GregorianCalendar[time=1560048258445,areFieldsSet=true,
         areAllFieldsSet=false,lenient=true,zone=sun.util.calendar.ZoneInfo[id="Asia/Baghdad",
         offset=10800000,dstSavings=0,useDaylight=false,transitions=55,lastRule=null],firstDayOfWeek=7,
         minimalDaysInFirstWeek=1,ERA=?,YEAR=2019,MONTH=5,WEEK_OF_YEAR=?,WEEK_OF_MONTH=?,DAY_OF_MONTH=9,
         DAY_OF_YEAR=?,DAY_OF_WEEK=?,DAY_OF_WEEK_IN_MONTH=?,AM_PM=0,HOUR=5,HOUR_OF_DAY=5,MINUTE=44,
         SECOND=18,MILLISECOND=445,ZONE_OFFSET=?,DST_OFFSET=?]
         ExpirationTime:INTEGER:-1
         Priority:INTEGER:0
         ReplyIdentifier:BLOB:[B@aa67d062
         ReplyProtocol:CHARACTER:TCPIP
         Topic:UNKNOWN:null
         ContentType:CHARACTER:
         IdentitySourceType:CHARACTER:
         IdentitySourceToken:CHARACTER:
         IdentitySourcePassword:CHARACTER:
         IdentitySourceIssuedBy:CHARACTER:
         IdentityMappedType:CHARACTER:
         IdentityMappedToken:CHARACTER:
         IdentityMappedPassword:CHARACTER:
         IdentityMappedIssuedBy:CHARACTER:
   DFDL
         TCPGlobalMsg
               ISO8583Msg:BLOB:[B@d33646bc


Last edited by abooddy on Fri Jun 14, 2019 11:24 am; edited 1 time in total
Back to top
View user's profile Send private message
timber
PostPosted: Mon Jun 10, 2019 1:50 am Post subject: Reply with quote

Guardian

Joined: 25 Aug 2015
Posts: 984

Quote:
The input to the DFDL parser is a BLOB
True...but the error is not from the DFDL parser. It is from the DFDL serializer. To explain that another way, the error happened when DFDL was writing the message, not when it was reading it.

Next steps:
- Forget everything that you have tried so far. You were focusing on the parsing, but the problem is with the writing.
- Find out (from the error message) the exact location in your message flow where DFDL is writing the message.
- Examine the message tree that DFDL is trying to write. It will (almost certainly) contain a '١' in the MtiVersion field.

Tip: You are quoting debugger outputs. A Trace node is often a better choice (it shows more information about the message tree).
Back to top
View user's profile Send private message
abooddy
PostPosted: Thu Jun 13, 2019 12:47 pm Post subject: Reply with quote

Novice

Joined: 28 May 2019
Posts: 11
Location: Baghdad, Iraq

Hi,

I mentioned in my first reply that the line causing the issue is:

Code:
DECLARE B BLOB;
SET B = ASBITSTREAM(InputRoot.DFDL.ns:ISO8583_1993_Unpacked OPTIONS RootBitStream CCSID 1208);


I'm sure that there is no Arabic number one in the to-be-serliazed message tree. I've proved it with the following:

Code:
SET OutputRoot = InputRoot;
      DECLARE B BLOB;
      DECLARE X DECIMAL 7;
      SET OutputRoot.DFDL.ns:ISO8583_1993_Unpacked.MTI_Version = X;
      SET B = ASBITSTREAM(OutputRoot.DFDL.ns:ISO8583_1993_Unpacked OPTIONS RootBitStream CCSID 1208);


I got the error:

Code:
CTDU4011E: The DFDL serializer cannot output the character '٧' in encoding 'US-ASCII' for the value of element 'MTI_Version'.


Where '٧' is the Arabic represntation of the English number 7.

The issue seems to be in the ASBITSTREAM function, I just don't know what exactly it is. Help please.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Thu Jun 13, 2019 5:27 pm Post subject: Reply with quote

Grand Poobah

Joined: 18 Nov 2003
Posts: 19978
Location: LI,NY

Could it be that the arabic representation of 7 is a character and not a digit???
and thus does not match the definition.... working as defined...
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
timber
PostPosted: Fri Jun 14, 2019 1:08 am Post subject: Reply with quote

Guardian

Joined: 25 Aug 2015
Posts: 984

Thanks - that moves things along a little. The symptoms are definitely strange, so we should go back to basics and debug this properly. That means Trace nodes and user trace (two different things, btw).

The ASBITSTREAM function uses exactly the same code as an output node uses. Please can you change your ESQL to
Code:
    SET OutputRoot = InputRoot;
    -- DECLARE B BLOB;
    -- DECLARE X DECIMAL 7;
    -- SET OutputRoot.DFDL.ns:ISO8583_1993_Unpacked.MTI_Version = X;

    SET OutputRoot.Properties.CodedCharSetId = 1208;
    PROPAGATE TO TERMINAL 'Out1';


On terminal Out1, connect a Trace node with pattern ${Root} followed by an MQOutput node. This will show us exactly what message tree DFDL is receiving.

Finally, please enable debug-level user trace as follows:
Code:
mqsichangetrace [brokerName] -u -e [integrationServerName] -l debug -r -c 100000
mqsichangetrace . The DFDL parser will write its DFDL Trace into the user trace.

...put a message through the flow

mqsichangetrace [brokerName] -u -e [integrationServerName] -l none
mqsireadlog [brokerName] -u -e [integrationServerName] -f -o /tmp/usertrace.xml
mqsiformatlog -i c:\temp\usertrace.xml -o /tmp/usertrace.txt
The DFDL parser will write its DFDL Trace into the user trace output, and it will show exactly what encoding DFDL is using, and what data it thinks it is receiving in the MtiVersion field.
Back to top
View user's profile Send private message
rekarm01
PostPosted: Fri Jun 14, 2019 4:54 am Post subject: Re: DFDL Numbers Parsing Error Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 1357

When posting BLOBs (or other long strings of unbroken text), please add line breaks; otherwise, at least in some browsers, the horizontal scrolling for the whole thread is a nuisance. To edit existing posts, use the "Edit" button in the upper-right hand portion of the post.

For the CREATE/ASBITSTREAM functions with the RootBitStream option, if the caller does not provide values for ENCODING, CCSID, SET, TYPE, and FORMAT, then the functions use default values; they don't search the message headers.

For example, note the TYPE clause when parsing:

Code:
CREATE LASTCHILD OF R DOMAIN('DFDL')
    PARSE(B CCSID InputRoot.Properties.CodedCharSetId
      TYPE '{http://www.ibm.com/dfdl/ISO8583-1993}:ISO8583_1993_Unpacked' OPTIONS RootBitStream);

which seems to be missing, when serializing:

Code:
SET B = ASBITSTREAM(InputRoot.DFDL.ns:ISO8583_1993_Unpacked OPTIONS RootBitStream CCSID 1208);

Whether the DFDL serializer requires a valid TYPE clause here, it certainly couldn't hurt.

By the way, if the DFDL schema does not use DFDL predefined variables for the encoding, byteOrder, (and binaryFloatRep), then the CCSID and ENCODING clauses in the CREATE/ASBITSTREAM functions (or in the message headers) have no effect.

abooddy wrote:
The issue seems to be in the ASBITSTREAM function ...

Only to the extent that the ASBITSTREAM function invokes the DFDL parser to serialize the message. The DFDL serializer converts the MTIVersion value from DECIMAL to CHARACTER to BYTE ("US-ASCII"). The issue seems to occur in the conversion from DECIMAL to CHARACTER, introducing the Arabic digit; as timber suggested, a usertrace might provide more information about that.
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum IndexWebSphere Message Broker SupportDFDL Numbers Parsing Error
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.