ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker SupportDFDL Numbers Parsing Error

Post new topicReply to topic Goto page Previous  1, 2
DFDL Numbers Parsing Error View previous topic :: View next topic
Author Message
abooddy
PostPosted: Fri Jun 14, 2019 12:05 pm Post subject: Reply with quote

Novice

Joined: 28 May 2019
Posts: 13
Location: Baghdad, Iraq

Hi,

This is the TraceNode's ${Root}:

Code:
( ['GENERICROOT' : 0x291ef443b40]
  (0x01000000:Name):Properties = ( ['MQPROPERTYPARSER' : 0x291f5897e30]
    (0x03000000:NameValue):MessageSet             = NULL
    (0x03000000:NameValue):MessageType            = '{}:ISO8583_1987_Unpacked' (CHARACTER)
    (0x03000000:NameValue):MessageFormat          = NULL
    (0x03000000:NameValue):Encoding               = NULL
    (0x03000000:NameValue):CodedCharSetId         = 1208 (INTEGER)
    (0x03000000:NameValue):Transactional          = NULL
    (0x03000000:NameValue):Persistence            = NULL
    (0x03000000:NameValue):CreationTime           = NULL
    (0x03000000:NameValue):ExpirationTime         = NULL
    (0x03000000:NameValue):Priority               = NULL
    (0x03000000:NameValue):ReplyIdentifier        = NULL
    (0x03000000:NameValue):ReplyProtocol          = 'TCPIP' (CHARACTER)
    (0x03000000:NameValue):Topic                  = NULL
    (0x03000000:NameValue):ContentType            = NULL
    (0x03000000:NameValue):IdentitySourceType     = NULL
    (0x03000000:NameValue):IdentitySourceToken    = NULL
    (0x03000000:NameValue):IdentitySourcePassword = NULL
    (0x03000000:NameValue):IdentitySourceIssuedBy = NULL
    (0x03000000:NameValue):IdentityMappedType     = NULL
    (0x03000000:NameValue):IdentityMappedToken    = NULL
    (0x03000000:NameValue):IdentityMappedPassword = NULL
    (0x03000000:NameValue):IdentityMappedIssuedBy = NULL
  )
  (0x01000000:Name):DFDL       = ( ['dfdl' : 0x291f8790d60]
    (0x01000000:Name)http://www.ibm.com/dfdl/ISO8583-1993:ISO8583_1993_Unpacked = (
      (0x03000000:NameValue):MTI_Version                            = 1 (DECIMAL)
      (0x03000000:NameValue):MTI_MessageClass                       = 1 (DECIMAL)
      (0x03000000:NameValue):MTI_MessageFunction                    = '1' (CHARACTER)
      (0x03000000:NameValue):MTI_MessageOrigin                      = 0 (DECIMAL)
      (0x01000000:Name     ):PrimaryBitmap                          = (
        (0x03000000:NameValue):Bits001to004 = '15' (CHARACTER)
        (0x03000000:NameValue):Bits005to008 = '8' (CHARACTER)
        (0x03000000:NameValue):Bits009to012 = '2' (CHARACTER)
        (0x03000000:NameValue):Bits013to016 = '0' (CHARACTER)
        (0x03000000:NameValue):Bits017to020 = '0' (CHARACTER)
        (0x03000000:NameValue):Bits021to024 = '0' (CHARACTER)
        (0x03000000:NameValue):Bits025to028 = '0' (CHARACTER)
        (0x03000000:NameValue):Bits029to032 = '0' (CHARACTER)
        (0x03000000:NameValue):Bits033to036 = '0' (CHARACTER)
        (0x03000000:NameValue):Bits037to040 = '14' (CHARACTER)
        (0x03000000:NameValue):Bits041to044 = '8' (CHARACTER)
        (0x03000000:NameValue):Bits045to048 = '1' (CHARACTER)
        (0x03000000:NameValue):Bits049to052 = '4' (CHARACTER)
        (0x03000000:NameValue):Bits053to056 = '4' (CHARACTER)
        (0x03000000:NameValue):Bits057to060 = '0' (CHARACTER)
        (0x03000000:NameValue):Bits061to064 = '0' (CHARACTER)
      )
      (0x01000000:Name     ):SecondaryBitmap                        = (
        (0x03000000:NameValue):Bits065to068 = '0' (CHARACTER)
        (0x03000000:NameValue):Bits069to072 = '0' (CHARACTER)
        (0x03000000:NameValue):Bits073to076 = '0' (CHARACTER)
        (0x03000000:NameValue):Bits077to080 = '0' (CHARACTER)
        (0x03000000:NameValue):Bits081to084 = '0' (CHARACTER)
        (0x03000000:NameValue):Bits085to088 = '0' (CHARACTER)
        (0x03000000:NameValue):Bits089to092 = '0' (CHARACTER)
        (0x03000000:NameValue):Bits093to096 = '0' (CHARACTER)
        (0x03000000:NameValue):Bits097to100 = '0' (CHARACTER)
        (0x03000000:NameValue):Bits101to104 = '4' (CHARACTER)
        (0x03000000:NameValue):Bits105to108 = '0' (CHARACTER)
        (0x03000000:NameValue):Bits109to112 = '0' (CHARACTER)
        (0x03000000:NameValue):Bits113to116 = '0' (CHARACTER)
        (0x03000000:NameValue):Bits117to120 = '0' (CHARACTER)
        (0x03000000:NameValue):Bits121to124 = '0' (CHARACTER)
        (0x03000000:NameValue):Bits125to128 = '0' (CHARACTER)
      )
      (0x03000000:NameValue):PrimaryAccountNumber_002               = '5210972000000328' (CHARACTER)
      (0x03000000:NameValue):ProcessingCode_003                     = '182000' (CHARACTER)
      (0x03000000:NameValue):AmountTransaction_004                  = '000000020000' (CHARACTER)
      (0x03000000:NameValue):AmountReconciliation_005               = '000000020200' (CHARACTER)
      (0x03000000:NameValue):SystemsTraceAuditNumber_011            = '279135' (CHARACTER)
      (0x03000000:NameValue):RetrievalReferenceNumber_037           = '000004279135' (CHARACTER)
      (0x03000000:NameValue):ApprovalCode_038                       = '129601' (CHARACTER)
      (0x03000000:NameValue):ResponseCode_039                       = '988' (CHARACTER)
      (0x03000000:NameValue):CardAcceptorTerminalIdentification_041 = '99900099' (CHARACTER)
      (0x03000000:NameValue):AdditionalDataPrivate_048              = '001001100200373602500120260015027000028012000004279135' (CHARACTER)
      (0x03000000:NameValue):CurrencyCodeReconciliation_050         = '368' (CHARACTER)
      (0x03000000:NameValue):AmountsAdditional_054                  = '007016C000000000000368' (CHARACTER)
      (0x03000000:NameValue):AccountIdentification1_102             = '00136821010100100044000' (CHARACTER)
    )
  )
)


The user trace wrote a lot of logs, I'll assume that this is the interesting stuff:

Code:
UserTrace   BIP2537I: Node 'msg_TCPISO8583.Rebuild_ISO_Msg.ConvertToBLOB': Executing statement   ''BEGIN ... END;'' at ('.Rebuild_ISO_Msg_ConvertToBLOB.Main', '2.2').
UserTrace   BIP2537I: Node 'msg_TCPISO8583.Rebuild_ISO_Msg.ConvertToBLOB': Executing statement   ''SET OutputRoot = InputRoot;'' at ('.Rebuild_ISO_Msg_ConvertToBLOB.Main', '6.7').
UserTrace   BIP2539I: Node '': Evaluating expression ''InputRoot'' at ('.Rebuild_ISO_Msg_ConvertToBLOB.Main', '6.24'). This resolved to ''InputRoot''. The result was ''ROW... Root Element Type=16777216 NameSpace='' Name='Root' Value=NULL''.
UserTrace   BIP2568I: Node 'msg_TCPISO8583.Rebuild_ISO_Msg.ConvertToBLOB': Copying sub-tree from ''InputRoot'' to ''OutputRoot''.
UserTrace   BIP2537I: Node 'msg_TCPISO8583.Rebuild_ISO_Msg.ConvertToBLOB': Executing statement   ''SET OutputRoot.Properties.CodedCharSetId = 1208;'' at ('.Rebuild_ISO_Msg_ConvertToBLOB.Main', '11.6').
UserTrace   BIP2566I: Node 'msg_TCPISO8583.Rebuild_ISO_Msg.ConvertToBLOB': Assigning value       ''1208'' to field / variable ''OutputRoot.Properties.CodedCharSetId''.
UserTrace   BIP2537I: Node 'msg_TCPISO8583.Rebuild_ISO_Msg.ConvertToBLOB': Executing statement   ''PROPAGATE TO TERMINAL 'out1' FINALIZE DEFAULT DELETE DEFAULT;'' at ('.Rebuild_ISO_Msg_ConvertToBLOB.Main', '12.6').
UserTrace   BIP4016I: Message propagated to terminal ''out1'' of node 'msg_TCPISO8583.Rebuild_ISO_Msg.ConvertToBLOB' with the following message trees: 'InputLocalEnvironment, OutputRoot, InputExceptionList'.
UserTrace   BIP2539I: Node 'msg_TCPISO8583.Rebuild_ISO_Msg.Trace': Evaluating expression ''Root'' at ('', '1.3'). This resolved to ''Root''. The result was ''ROW... Root Element Type=16777216 NameSpace='' Name='Root' Value=NULL''.
UserTrace   BIP4067I: Message propagated to output terminal for trace node 'msg_TCPISO8583.Rebuild_ISO_Msg.Trace'.
 The trace node 'msg_TCPISO8583.Rebuild_ISO_Msg.Trace' has received a message and is propagating it to any nodes connected to its output terminal.
 No user action required.
UserTrace   BIP7944I: A message is being processed in node ''msg_TCPISO8583.Rebuild_ISO_Msg.MQ Output'', with the following attributes derived from the policy at '''': ''connection: SERVER, destinationQueueManagerName: APSQMGR, logLabel: msg_TCPISO8583.Rebuild_ISO_Msg.MQ Output''.
 Each message that is processed by the node might use different attributes derived from a policy. This message records the attribute values that are used for a specific message.
 No user action required.
UserTrace   BIP5841I: ''Offset: 0. Starting to process the DFDL infoset.''
UserTrace   BIP5841I: ''Offset: 0. Element 'ISO8583_1993_Unpacked' was found in the infoset.''
UserTrace   BIP5841I: ''Offset: 0. Starting to write root element 'ISO8583_1993_Unpacked'.''
UserTrace   BIP5841I: ''Offset: 0. Starting to process element 'ISO8583_1993_Unpacked'.''
UserTrace   BIP5841I: ''Offset: 0. Element 'MTI_Version' was found in the infoset.''
UserTrace   BIP5841I: ''Offset: 0. Starting to process element 'MTI_Version'.''
UserTrace   BIP5841I: ''Offset: 0. Value '1' of type xs:decimal received for element 'MTI_Version' .''
UserTrace   BIP5843E: ''The DFDL serializer cannot output the character '' in encoding 'US-ASCII' for the value of element 'MTI_Version'.''


Adding the type clause to the ASBITSTREAM function did not help, the same error occured.
Back to top
View user's profile Send private message
abooddy
PostPosted: Fri Jun 14, 2019 12:09 pm Post subject: Reply with quote

Novice

Joined: 28 May 2019
Posts: 13
Location: Baghdad, Iraq

This is how it appears in Notepad++:

Back to top
View user's profile Send private message
fjb_saper
PostPosted: Sat Jun 15, 2019 7:38 pm Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20025
Location: LI,NY

I believe there is confusion here between what should be a character representation of a digit, and the numeric representation of the digit.
Should that part not serialize as a BLOB and not as character?
At best a hex representation of the BLOB...
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
rekarm01
PostPosted: Sun Jun 16, 2019 3:13 pm Post subject: Re: DFDL Numbers Parsing Error Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 1365

abooddy wrote:
I'll assume that this is the interesting stuff:

Code:
...
UserTrace   BIP5841I: ''Offset: 0. Value '1' of type xs:decimal received for element 'MTI_Version' .''
UserTrace   BIP5843E: ''The DFDL serializer cannot output the character '١' in encoding 'US-ASCII' for the value of element 'MTI_Version'.''

The interesting stuff would probably be happening between those two statements. The initial post indicated that changing the Windows system locale didn't help, and according to timber, "DFDL never uses platform/OS defaults" anyway.

But, at the very least, changing the user locale (or codepage) may fix the substituting of arabic characters with the '<SUB>' control character, in the user trace output.

abooddy wrote:
Code:
<xs:element dfdl:length="1" dfdl:textNumberPattern="#0" ibmDfdlExtn:sampleValue="1" name="MTI_Version" type="xs:integer"/>

How does the DFDL parser convert this element to "xs:decimal", when the posted schema defines the element type as "xs:integer"? Is that from the right XSD?

It might help to provide some more detail, clarifying the order of transformations, and the schemas involved, for InputRoot ({}:TCPGlobalMsg), BLOB B, and ROW R.

Here is another (possibly irrelevant) example, using iconv on Linux to convert the Arabic characters from Unicode to IBM-9448:

Code:
$ echo -n "١٧" | iconv -f UTF-8 -t IBM-9448 | od -c -t xC
0000000   1   7
         31  37

It substitutes the given arabic digits with the corresponding ASCII digits ('1' and '7'). That works for 'IBM-9448', but not for 'WINDOWS-1256'.

However, it is not clear what that might mean, for the DFDL parser, in the broker, on a Windows platform.
Back to top
View user's profile Send private message
timber
PostPosted: Mon Jun 17, 2019 1:31 am Post subject: Reply with quote

Grand Master

Joined: 25 Aug 2015
Posts: 1002

Thanks - that's very helpful. It doesn't look like correct behaviour to me, and I wonder whether DFDL is strugging to output the DECIMAL value. What happens if you CAST MtiVersion to INTEGER before sending the message tree to DFDL?
Back to top
View user's profile Send private message
rekarm01
PostPosted: Mon Jun 17, 2019 2:34 pm Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 1365

fjb_saper wrote:
I believe there is confusion here between what should be a character representation of a digit, and the numeric representation of the digit.
Should that part not serialize as a BLOB and not as character?

Technically, everything serializes as a BLOB. But the DFDL schema governs the exact format of the resulting BLOB:

Code:
...
<dfdl:format encoding="US-ASCII"  ...>
...
<xs:element dfdl:length="1" dfdl:textNumberPattern="#0" ibmDfdlExtn:sampleValue="1" name="MTI_Version" type="xs:integer"/>
...

In this case, the parsed element is an INTEGER, and the unparsed BLOB is a "US-ASCII" byte, representing the INTEGER value.

Someone else can probably much better explain the internal workings of the DFDL parser, but there seems to be an intermediate conversion while unparsing, from INTEGER to CHARACTER, (before converting from CHARACTER to BLOB), where DFDL converts the INTEGER value to an Arabic CHARACTER, instead of an ASCII character.

A separate question is: Why are the parsed elements DECIMAL, when the posted schema defines them as INTEGER?
Back to top
View user's profile Send private message
abooddy
PostPosted: Thu Jun 20, 2019 10:24 am Post subject: Reply with quote

Novice

Joined: 28 May 2019
Posts: 13
Location: Baghdad, Iraq

Hi,

Unfortunately, I can only work on this issue on weekends, so I will available for the next two days very effectively and hopefully we manage to fix the issue by then.

Quote:
But, at the very least, changing the user locale (or codepage) may fix the substituting of arabic characters with the '<SUB>' control character, in the user trace output.


My system locale is "English (United Kingdom)" and my active code page according to chcp is 850. I see that code page 850 does not contain the Arabic numbers.

Quote:
How does the DFDL parser convert this element to "xs:decimal", when the posted schema defines the element type as "xs:integer"? Is that from the right XSD?


Yes it is from the correct and only XSD. The line:

Code:
CREATE LASTCHILD OF R DOMAIN('DFDL')
       PARSE(B CCSID InputRoot.Properties.CodedCharSetId TYPE
        '{http://www.ibm.com/dfdl/ISO8583-1993}:ISO8583_1993_Unpacked' OPTIONS RootBitStream);


Creates the element MTI_Version as a DECIMAL, not an INTEGER. This should be really weird, but I don't think it is the issue because when I tried:

Quote:
What happens if you CAST MtiVersion to INTEGER before sending the message tree to DFDL?


The same issue happened.
Back to top
View user's profile Send private message
timber
PostPosted: Thu Jun 20, 2019 11:28 pm Post subject: Reply with quote

Grand Master

Joined: 25 Aug 2015
Posts: 1002

Quote:
Why are the parsed elements DECIMAL, when the posted schema defines them as INTEGER?
That's a fair question, and one that I do know the answer to. The xsd:integer type is unconstrained (has infinite range) so it would be dangerous to convert that type to the native 64-bit integer type of ESQL. Early versions of IBM DFDL did, and the defect was caught and fixed.
If the CAST to INTEGER had fixed the problem, then I would have recommended a change from xsd:integer to xsd:long (which is defined to be 64-bit). But it didn't, so no point in making the change.
Back to top
View user's profile Send private message
timber
PostPosted: Thu Jun 20, 2019 11:40 pm Post subject: Reply with quote

Grand Master

Joined: 25 Aug 2015
Posts: 1002

I have to admit that I'm almost out of ideas. The field is a text representation of an integer in US-ASCII. The output should be a single byte with value 0x31.

There is no good reason why this should work...but maybe it's worth explicitly setting the 'encoding' property on MtiVersion to 'ISO-8859-1' (which is an alias for 7-bit US-ASCII). You can do that in the DFDL editor. It should add an attribute dfdl:encoding="ISO-8859-1" to the element definition for MtiVersion, which will override the default value.

If that does not change anything (and it probably will not) then I think it's time to ask IBM for some help. If you do open a PMR, I would advise you to reference this thread in your problem description.
Back to top
View user's profile Send private message
abooddy
PostPosted: Fri Jun 21, 2019 2:57 am Post subject: Reply with quote

Novice

Joined: 28 May 2019
Posts: 13
Location: Baghdad, Iraq

Hi,

Quote:
maybe it's worth explicitly setting the 'encoding' property on MtiVersion to 'ISO-8859-1'


Same issue occurred:

Code:
The DFDL serializer cannot output the character '١' in encoding 'ISO-8859-1' for the value of element 'MTI_Version'.


Thank you all for you support. Seems I'm forced to use the Linux brokers, which is not that bad.
Back to top
View user's profile Send private message
rekarm01
PostPosted: Fri Jun 21, 2019 10:12 am Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 1365

abooddy wrote:
My system locale is "English (United Kingdom)" and my active code page according to chcp is 850. I see that code page 850 does not contain the Arabic numbers.

Keep in mind that this would just fix the mqsiformatlog output file; it doesn't address the underlying issue. The Windows codepage for utf-8 is 65001. Try that. If that doesn't work, then another option is to look through the mqsireadlog output xml file for non-ASCII characters instead; mqsireadlog encodes its output as utf-8.

timber wrote:
The xsd:integer type is unconstrained (has infinite range) so it would be dangerous to convert that type to the native 64-bit integer type of ESQL.

That's good to know. Working as designed. ESQL DECIMAL is also finite, but that's probably less of an issue. ;-)

timber wrote:
... it's time to ask IBM for some help. If you do open a PMR, ...

IBM calls them "cases" now.

abooddy wrote:
Seems I'm forced to use the Linux brokers, which is not that bad.

Some might call that an upgrade. Still, it's probably worthwhile to open up a support case with IBM. If there's a defect with the DFDL parser, then fixing it could help future users too.
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Goto page Previous  1, 2 Page 2 of 2

MQSeries.net Forum IndexWebSphere Message Broker SupportDFDL Numbers Parsing Error
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.