MQSeries.net :: View topic - Nil for End of Stream in DFDL Model

MQSeries.net

Tech Exchange

Education

Certifications

Library

Info Center

SupportPacs

FAQÂ Â

Usergroups

RSS Feed - WebSphere MQ Support

RSS Feed - Message Broker Support

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » Nil for End of Stream in DFDL Model

Nil for End of Stream in DFDL Model

« View previous topic :: View next topic »

Author

Message

aslam_kumhar

Posted: Tue Feb 24, 2015 10:30 am Post subject: Nil for End of Stream in DFDL Model

Novice

Joined: 30 Oct 2014
Posts: 22

Hi Everyone,
We are having following issues.

1. Parsing messages with last element missing whose properties are such that occurence kind is '1' based on expression and lengthKind is 'prefixed'. The problem is that the last element is missing which is of prefixed length now i want the parser to consider it and all other subsequent values to 'nil'.
As I raised the issue in one of my other thread http://www.mqseries.net/phpBB2/viewtopic.php?p=387946#387946 so following are the dummy dfdl model with dummy message both parsable and unparsable.

Dummy DFDL Model:

Code:

<?xml version="1.0" encoding="UTF-8"?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:csv="http://www.ibm.com/dfdl/CommaSeparatedFormat" xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/" xmlns:ibmDfdlExtn="http://www.ibm.com/dfdl/extensions" xmlns:ibmSchExtn="http://www.ibm.com/schema/extensions">
<xsd:import namespace="http://www.ibm.com/dfdl/CommaSeparatedFormat" schemaLocation="IBMdefined/CommaSeparatedFormat.xsd" />
<xsd:complexType name="BitmapType">
   <xsd:sequence dfdl:separator="">
      <xsd:element dfdl:binaryNumberCheckPolicy="lax" dfdl:lengthKind="explicit" name="Bit01" type="Type_Bit" />
      <xsd:element dfdl:binaryNumberCheckPolicy="lax" dfdl:lengthKind="explicit" name="Bit02" type="Type_Bit" />
      <xsd:element dfdl:binaryNumberCheckPolicy="lax" dfdl:lengthKind="explicit" name="Bit03" type="Type_Bit" />
      <xsd:element dfdl:binaryNumberCheckPolicy="lax" dfdl:lengthKind="explicit" name="Bit04" type="Type_Bit" />
      <xsd:element dfdl:binaryNumberCheckPolicy="lax" dfdl:lengthKind="explicit" name="Bit05" type="Type_Bit" />
      <xsd:element dfdl:binaryNumberCheckPolicy="lax" dfdl:lengthKind="explicit" name="Bit06" type="Type_Bit" />
      <xsd:element dfdl:binaryNumberCheckPolicy="lax" dfdl:lengthKind="explicit" name="Bit07" type="Type_Bit" />
      <xsd:element dfdl:binaryNumberCheckPolicy="lax" dfdl:lengthKind="explicit" name="Bit08" type="Type_Bit" />
   </xsd:sequence>
</xsd:complexType>
<xsd:simpleType dfdl:alignmentUnits="bits" dfdl:binaryNumberRep="binary" dfdl:byteOrder="bigEndian" dfdl:length="1" dfdl:lengthUnits="bits" dfdl:representation="binary" name="Type_Bit">
   <xsd:restriction base="xsd:unsignedInt" />
</xsd:simpleType>
<xsd:simpleType dfdl:length="3" dfdl:lengthKind="explicit" dfdl:textNumberPattern="00" name="PrefixLength3">
   <xsd:restriction base="xsd:integer" />
</xsd:simpleType>
<xsd:simpleType dfdl:length="2" dfdl:lengthKind="explicit" dfdl:textNumberPattern="00" name="PrefixLength2">
   <xsd:restriction base="xsd:integer" />
</xsd:simpleType>
<xsd:simpleType dfdl:binaryNumberCheckPolicy="lax" dfdl:binaryNumberRep="binary" dfdl:byteOrder="bigEndian" dfdl:length="2" dfdl:lengthKind="explicit" dfdl:lengthUnits="bytes" dfdl:representation="binary" name="PrefixLength2Bytes">
   <xsd:restriction base="xsd:unsignedInt" />
</xsd:simpleType>
<xsd:annotation>
   <xsd:appinfo source="http://www.ogf.org/dfdl/">
      <dfdl:format documentFinalTerminatorCanBeMissing="yes" encoding="{$dfdl:encoding}" escapeSchemeRef="csv:CSVEscapeScheme" ref="csv:CommaSeparatedFormat" />
   </xsd:appinfo>
</xsd:annotation>
<xsd:element dfdl:lengthKind="prefixed" dfdl:lengthUnits="bytes" dfdl:prefixIncludesPrefixLength="no" dfdl:prefixLengthType="PrefixLength2Bytes" ibmSchExtn:docRoot="true" name="DummyModel">
   <xsd:complexType>
      <xsd:sequence dfdl:separator="">
         <xsd:element dfdl:length="1" dfdl:lengthKind="explicit" dfdl:lengthUnits="bytes" name="Bitmap" type="BitmapType" />
         <xsd:element dfdl:length="10" dfdl:lengthKind="explicit" dfdl:occursCount="{/DummyModel/Bitmap/Bit01}" dfdl:occursCountKind="expression" minOccurs="0" name="Field1" type="xsd:string" />
         <xsd:element dfdl:length="5" dfdl:lengthKind="explicit" dfdl:occursCount="{/DummyModel/Bitmap/Bit02}" dfdl:occursCountKind="expression" minOccurs="0" name="Field2" type="xsd:string" />
         <xsd:element dfdl:lengthKind="prefixed" dfdl:occursCount="{/DummyModel/Bitmap/Bit03}" dfdl:occursCountKind="expression" dfdl:prefixIncludesPrefixLength="no" dfdl:prefixLengthType="PrefixLength2" minOccurs="0" name="Field3" type="xsd:string" />
         <xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:lengthKind="prefixed" dfdl:nilValueDelimiterPolicy="none" dfdl:occursCount="{/DummyModel/Bitmap/Bit04}" dfdl:occursCountKind="expression" dfdl:prefixIncludesPrefixLength="no" dfdl:prefixLengthType="PrefixLength3" minOccurs="0" name="Field4" type="xsd:string" />
      </xsd:sequence>
   </xsd:complexType>
</xsd:element>
</xsd:schema>

Dummy Messages:

Code:

Parsable
----------
Original = Ã°AAAAAAAAAABBBBB04CCCC003DDD
Hex = 201CF0414141414141414141414242424242303443434343303033444444

Unparsable
-------------
Original = Ã°AAAAAAAAAABBBBB04CCCC
Hex = 2016F0414141414141414141414242424242303443434343

Now the above unparsable message can be parsed with nil or missing representation if i changed length type of Field4 from 'prefixed' to 'delimited' but doing so takes us to the following problem.

2. Now this same model is set on TCPIPServerInput node as 'Input Message Parsing' and Record Detection is 'Parsed Record Sequence'. Issue is like that the client is only allowed to create 1 socket connection with above TCPServerInput node so we created TCPServer Configurable Service with maximum connections set to 1 on 'localhost' with a arbitrary port furthermore for keeping the connection open and alive we set Advance properties of TCPIPServer node as 'Close Connection' to 'No', 'Input/Output Stream Modification' to 'Leave Unchanged'.
If we apply the fix for unparsable message and client sends us above unparsable message then the TCPIPServerNode waits for more data since this is the complete message client will send no more data so eventually TCPIPServer will timeout and flush whatever parsed data it has to next available node and terminates the connection. We only want to it to flush the data but don't terminate the connection.

Any help or suggestions will be very appreciated since we are deadlock on these issues and we have deadlines to meet.

Thanks

fjb_saper

Posted: Tue Feb 24, 2015 2:38 pm Post subject:

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

Did you define the last 2 fields as being optional? (occurs 0 to 1) with occurs count kind to implicit ?

_________________
MQ & Broker admin

aslam_kumhar

Posted: Tue Feb 24, 2015 11:08 pm Post subject:

Novice

Joined: 30 Oct 2014
Posts: 22

fjb_saper wrote:

Did you define the last 2 fields as being optional? (occurs 0 to 1) with occurs count kind to implicit ?

Yes the last 2 fields can be optional but occurrence count kind is expression not implicit, based on bitmap bits if relevant bit is on then the field has occurence of 1 if off then 0.

shanson

Posted: Wed Feb 25, 2015 2:07 am Post subject:

Partisan

Joined: 17 Oct 2003
Posts: 344
Location: IBM Hursley

Your format sounds very similar to ISO8583, which also has bits at the start of the message to indicate which fields are present. It also has many fields with prefixed lengths.

The DFDL schemas for ISO8583 on GitHub @ https://github.com/DFDLSchemas/ISO8583 show exactly what you have to do to make your model work.

Looking at your schema, it appears to be correct.

I think the issue is with your data. If you look at the hex for your un-parseable data, you have deleted one of the fields but the corresponding bit has not been set to 0 (the byte is still xF0). If you fix this I think your data will parse.

aslam_kumhar

Posted: Wed Feb 25, 2015 4:00 am Post subject:

Novice

Joined: 30 Oct 2014
Posts: 22

shanson wrote:

yeah originally i am using iso8583 format but for better understanding i have created this dummy schema with test message. Also i have deliberately not set last field data in un-parsable message, because i wanted it to be 'nil' when end of stream/data is reached on parsing, that's what i am asking. Thanks for your response.

shanson

Posted: Wed Feb 25, 2015 4:40 am Post subject:

Partisan

Joined: 17 Oct 2003
Posts: 344
Location: IBM Hursley

Firstly, if a field is missing entirely, you can not have it appear as nil. The DFDL spec makes a clear distinction between a field being missing and having a representation. A field can be treated as nil only if it has a representation (including empty string). Empty string is not the same as missing entirely.

Secondly, think what the parser is doing. It is driven entirely by the occursCount expressions. So if the bit is set, it will read the data at its current offset. So in your un-parseable example, it reads the 4th bit, it says 1 so the parser then expects to find exactly 1 occurrence of field4 (minOccurs and maxOccurs are not used - they are effectively replaced by occursCount here). So it tries to parse the prefix length - and fails because there is no data. That is correct behaviour.

Are you saying that you are expecting the data to arrive looking like this?

aslam_kumhar

Posted: Wed Feb 25, 2015 5:37 am Post subject:

Novice

Joined: 30 Oct 2014
Posts: 22

shanson wrote:

Very good explanation. Let me further clarify it the message itself is prefixed, but 4th field has occurrence has exactly 1 occurrence so parser will eventually expect data whether there are empty spaces or if in my case field is also prefixed and field contains no data then parser will expect '00' for prefixed length of 2 character if i want it to be 'nil', but the problem is that there is no more data available and will not be available, the last parsable field which is field 3 when it is parsed, prefixed message length is completed so logically parser should not expect more data that's where i want it to consider remaining fields missing (i may be confusing it with nil).

Secondly there is no way to tell parser to consider field 'nil' or 'missing' if End of Stream is reached for prefixed length but if i changed the length type from 'prefixed' to 'delimited' then parser will treat it as optional, empty or missing. Following is the parser trace for Field4 when length is set to 'delimited':

Code:

Feb 25, 2015, 6:24:02 PM info: Offset: 24. Exactly '1' occurrences of element 'Field4' will be expected because occursCountKind='expression' and the result of occursCount expression '{/DummyModel/Bitmap/Bit04}' was '1'
[dfdl = /DFDLTestApp/DummyModel.xsd, scd = #xscd(/schemaElement::DummyModel/type::0/model::sequence/schemaElement::Field4), 196]

Feb 25, 2015, 6:24:02 PM info: Offset: 24. Starting to process element 'Field4'.
[dfdl = /DFDLTestApp/DummyModel.xsd, scd = #xscd(/schemaElement::DummyModel/type::0/model::sequence/schemaElement::Field4), 57]

Feb 25, 2015, 6:24:02 PM info: Offset: 24. Element 'Field4' is empty because the content region is empty.
[dfdl = /DFDLTestApp/DummyModel.xsd, scd = #xscd(/schemaElement::DummyModel/type::0/model::sequence/schemaElement::Field4), 82]

Feb 25, 2015, 6:24:02 PM info: Offset: 24. Found delimited value: '' for element 'Field4'. The delimiter was 'End of data stream'.
[dfdl = /DFDLTestApp/DummyModel.xsd, scd = #xscd(/schemaElement::DummyModel/type::0/model::sequence/schemaElement::Field4), 108]

Feb 25, 2015, 6:24:02 PM info: Offset: 24. Element 'Field4' is optional and empty or missing. The element will not be included in the infoset.
[dfdl = /DFDLTestApp/DummyModel.xsd, scd = #xscd(/schemaElement::DummyModel/type::0/model::sequence/schemaElement::Field4), 119]

Feb 25, 2015, 6:24:02 PM info: Offset: 24. Finished processing element 'Field4'.
[dfdl = /DFDLTestApp/DummyModel.xsd, scd = #xscd(/schemaElement::DummyModel/type::0/model::sequence/schemaElement::Field4), 57]

but doing so will create another problem which is described in my first post. Thanks for your input.

aslam_kumhar

Posted: Wed Feb 25, 2015 6:40 am Post subject:

Novice

Joined: 30 Oct 2014
Posts: 22

fjb_saper wrote:

Did you define the last 2 fields as being optional? (occurs 0 to 1) with occurs count kind to implicit ?

Sorry i overlooked your suggestion. Looks like setting min/max occurrence to 0/1 and setting occurrence kind to implicit does also parse message and does not terminates connection on TCPIPServerInput node. Our IBM consultant also suggested the same approach.
Aside this, is there any way by which for prefixed lenght fields if end of stream is reached we can tell parser to consider those elements 'missing'.
Thanks

shanson

Posted: Wed Feb 25, 2015 7:35 am Post subject:

Partisan

Joined: 17 Oct 2003
Posts: 344
Location: IBM Hursley

Quote:

Looks like setting min/max occurrence to 0/1 and setting occurrence kind to implicit does also parse message and does not terminates connection on TCPIPServerInput node. Our IBM consultant also suggested the same approach.

Either the bitmap is in charge of the parsing, or it is not. The point of the bitmap is that the message can be sparse. If I have bits '01011010' then the 2nd, 4th, 5th and 7th fields are present. If you use occursCountKind 'implicit' then the parser will treat the data as 1st, 2nd, 3rd and 4th fields present. It does not know otherwise. Your example data is not a good example as the bitmap is '11110000'.

You could use 'implicit' in one use case - if you know that the message will never be sparse from field n onwards, then field n onwards can be 'implicit'. But that is not the case for ISO8583.

aslam_kumhar

Posted: Wed Feb 25, 2015 10:47 am Post subject:

Novice

Joined: 30 Oct 2014
Posts: 22

shanson wrote:

Quote:

I may have confused you. OccurenceKind = 'implicit' is only set for Field4 which is also the last field of the message, occurrenceKind of all previous fields will be get from bitmaps. Again this is just a patch solution.

Let me tell you the exact situation, we are receiving ISO8583 message from a legacy application which does comply mostly with ISO8583 standards with a little customization of its own, but one of its messages contains an anomaly. The message in question has one secondary bitmap field whose occurrence = 1 but there's no data for this field now this field is both the last field and is of prefixed length type. This anomaly is against the standard, in normal circumstances this is unacceptable but we have to make an exception since this legacy application cannot be changed. What we wanted to do was to keep ISO8583 standard and somehow tell the parser to consider that field missing or nil if End of Stream is reached.

Changing lengthKind from 'prefixed' to 'implicit' and setting min/max occurrence 0/1 does violate ISO8583 standard but it will do the job for the time being. I hope no more surprises from our legacy application will come in terms of ISO8583 standards.

Thanks again.

kimbert

Posted: Thu Feb 26, 2015 5:05 pm Post subject:

Jedi Council

Joined: 29 Jul 2003
Posts: 5542
Location: Southampton

Quote:

Changing lengthKind from 'prefixed' to 'implicit' and setting min/max occurrence 0/1 does violate ISO8583 standard but it will do the job for the time being.

I don't see how that fixes the problem. What happens when these trailing fields *are* present in the input data?

Scenario 1:
There is exactly one trailing field. The prefix length will be included in the value of the element, so the ESQL logic that processes the trailing field must be adjusted to remove the prefix length.

Scenario 2:
There are two ( or more ) trailing fields. All of them have lengthKind='delimited'. Now the situation is even worse : the first field that has lengthKind='delimited' will consume all of the data up to the end of the message and the other trailing fields will get no data at all.

Have you tested all these scenarios? Or am I missing something?
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too.

aslam_kumhar

Posted: Fri Feb 27, 2015 10:02 am Post subject:

Novice

Joined: 30 Oct 2014
Posts: 22

kimbert wrote:

Quote:

Changing lengthKind from 'prefixed' to 'implicit' and setting min/max occurrence 0/1 does violate ISO8583 standard but it will do the job for the time being.

Short answer to your questions is that there are no trailing fields that is the special case , lets just say there are trailing values but the field in question has bitmap bit on then i can add an exceptional expression to override bitmap bit which is more suitable for my special need.

Somehow the thing i wanted to ask was lost in all this confusion, I just wanted to know that Is there a way by which I can tell the parser to consider the field 'nil' for prefixed length elements if End Of Stream is reached?

Display posts from previous:

Page 1 of 1

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » Nil for End of Stream in DFDL Model

Jump to:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

Protected by Anti-Spam ACP