| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| DFDL unbounded sequence not working |
« View previous topic :: View next topic » |
| Author |
Message
|
| GeneRK4 |
 Posted: Thu Mar 27, 2014 10:00 am Post subject: DFDL unbounded sequence not working |
 |
|
Master
Joined: 08 Jul 2013
Posts: 220
|
I am trying to create DFDL for the below TDS message,
A|B|C
D|E|F
G|H|I
...
The number of records are unbounded.The fields are separated by "|".The record is separated by %CR%LF.
I created DFDL as Record Oriented structure.
I could see the structure created as below:
DFDL_Test
Sequence - Separater:%CR;
body- 1 to unbounded
Sequence - Separater:%#124;
body_element1
body_element2
body_element3
Though the Outer Sequence ,body,Inner Sequence are defined with proper Separaters,the records are getting parsed only till the last field of the first record.
After that,the below error is getting thrown :
An unexpected non-postfix separator '%CR;%WSP*;' occurs in a postfix position at offset '19' in 'sequence' group contained within element 'DFDL_Test', xpath: '/DFDL_Test[1]'.
ParsedDataRegion[SimpleContent, startOffset = 18, length = 1, scd = #xscd(/schemaElement::DFDL_Test/type::0/model::sequence/schemaElement::body/type::0/model::sequence/schemaElement::body_elem3)]
I am not sure where is this problem arising.I saw few posts regarding this in this forum,but the solution is not working though I have tried various option like Suppress,NilKind,Postfix....
Please help me to resolve this....
[/img] |
|
| Back to top |
|
 |
| kimbert |
| Posted: Thu Mar 27, 2014 3:43 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
First thought is that your message format is exactly like CSV, except that it uses pipe (|) instead of comma (,). So you might get better results using the CSV wizard.
However, "Record Oriented" is also a valid choice. If you look carefully at the error message, you will see that the error is actually occurring at offset 19, which is at the end of the third line. I would guess that you have added a carriage-return at the end of the last line. The separator %CR; is defined as 'infix' which means that it occurs *between* elements, but not after the final element.
You could have discovered this for yourself by reading the DFDL Trace. It is automatically available when you use the Test Parse facility in the toolkit.
You could change separatorPosition to 'postfix' and make the problem go away. But then your model would fail when the final CR was missing. The correct solution is to
- define a terminator of %CR;%WSP; on the root element
- set the 'DocumentFinalTerminatorCanBeMissing' property to 'yes'. Please do this in *your* xsd's format block. Don't edit the IBM-supplied xsd. See http://www.mqseries.net/phpBB2/viewtopic.php?t=64555&sid=e8f122e8c39d684b9b7901fc9f278f54 for how to do that.
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| GeneRK4 |
| Posted: Thu Mar 27, 2014 5:07 pm Post subject: |
|
|
Master
Joined: 08 Jul 2013
Posts: 220
|
| kimbert wrote: |
First thought is that your message format is exactly like CSV, except that it uses pipe (|) instead of comma (,). So you might get better results using the CSV wizard.
|
Thank you very very much 
I did all changes you have mentioned in RecordOriented DFDL.But nothing worked.
When I created csv format DFDL,and modified the xsd with relevant properties,it has worked for me very well  |
|
| Back to top |
|
|
| GeneRK4 |
| Posted: Fri Mar 28, 2014 4:59 am Post subject: |
|
|
Master
Joined: 08 Jul 2013
Posts: 220
|
Hi...
I have got a new issue in this now..
I am trying to parse the mainframe message.Each records is separated by 0D0A .But when I parsed,only the first record was parsed successfully.
While parsing the second record "0A00" (part of 0D0A terminator) is picked up as part of second record.And consecutive records are also failing...
I tried many ways like having %CR%LF%WSP* as terminator.Having only %CR%WSP* as terminator..%NL%WSP* ...but nothing is working.
Could you please advice what I am missing?
Note:This error is happening only for Mainframe message.Not for any other text message with same terminator(CRLF)
Message in BLOB received in MQInput node for reference:
0006 is the first field in the record.0d0a is the terminator.1C is the field separator in the below message:
00061cd7c8d6e3d640c1d5c440d4e4e2c9c340401cc2c1e3e3c5d9c9c5e24040401c000f44b81ce5c1d9e3c140c8c9c7c840c1c140f44ef4e24040404040401c004c1c000022800c1c002f1cf1f14bf0f64bf2f0f1f31cf0f04bf0f04bf0f01c0d0a00061cc8c5c1d3e3c8c3c1d9c5404040404040401cc6c5d4c9c3c1d9c5404040401c000f46431cc3c1d5c5e2e3c5d540f16c40c3d9c5c1d440f2f0c74040401cfffe1c000000998d1c00061cf1f14bf0f64bf2f0f1f31cf0f04bf0f04bf0f01c0d0a |
|
| Back to top |
|
|
| kimbert |
| Posted: Fri Mar 28, 2014 9:13 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| While parsing the second record "0A00" (part of 0D0A terminator) is picked up as part of second record. |
That's because your separator is defined as "%CR;". It needs to be "%CR;%LF;". Remember to include the semicolons!
| Quote: |
| I tried many ways like having %CR%LF%WSP* as terminator.Having only %CR%WSP* as terminator..%NL%WSP* ...but nothing is working. |
As I was saying...remember to include the semicolons 
| Quote: |
| Note:This error is happening only for Mainframe message.Not for any other text message with same terminator(CRLF) |
Obvious question: what is the difference between a 'Mainframe' message and the other types of message?
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| GeneRK4 |
| Posted: Fri Mar 28, 2014 5:53 pm Post subject: |
|
|
Master
Joined: 08 Jul 2013
Posts: 220
|
Hi..
I already used semicolons.Still the same problem.
Here is the complete trace :
********************* DFDL Parser Setup Starting *********************
Schema = /DFDL_Test/DFDL_Test4.xsd
**********************************************************************
********************* DFDL Parser Starting *********************
Data = /DFDL_Test/NSPSampleInput.txt
Message = DFDL_Test4 (/DFDL_Test/DFDL_Test4.xsd)
****************************************************************
Mar 29, 2014 7:20:18 AM info: Offset: 0. Parsing will start from root element 'DFDL_Test4'.
[dfdl = /DFDL_Test/DFDL_Test4.xsd, scd = #xscd(/schemaElement::DFDL_Test4), 69]
Mar 29, 2014 7:20:18 AM info: The default value of '%LF;' was assigned to variable 'outputNewLine' in namespace 'http://www.ogf.org/dfdl/dfdl-1.0/'.
[dfdl = /DFDL_Test/DFDL_Test4.xsd, scd = , 133]
Mar 29, 2014 7:20:18 AM info: Offset: 0. Starting to process element 'DFDL_Test4'.
[dfdl = /DFDL_Test/DFDL_Test4.xsd, scd = #xscd(/schemaElement::DFDL_Test4), 60]
Mar 29, 2014 7:20:18 AM info: Offset: 0. Up to '-1' occurrences of element 'record' will be expected because occursCountKind='implicit' and maxOccurs='-1'.
[dfdl = /DFDL_Test/DFDL_Test4.xsd, scd = #xscd(/schemaElement::DFDL_Test4/type::0/model::sequence/schemaElement::record), 133]
Mar 29, 2014 7:20:18 AM info: Offset: 0. Starting to process element 'record'.
[dfdl = /DFDL_Test/DFDL_Test4.xsd, scd = #xscd(/schemaElement::DFDL_Test4/type::0/model::sequence/schemaElement::record), 56]
Mar 29, 2014 7:20:18 AM info: Offset: 0. Starting to process element 'field1'.
[dfdl = /DFDL_Test/DFDL_Test4.xsd, scd = #xscd(/schemaElement::DFDL_Test4/type::0/model::sequence/schemaElement::record/type::0/model::sequence/schemaElement::field1), 56]
Mar 29, 2014 7:20:18 AM info: Offset: 0. Found delimited value: '†' for element 'field1'. The delimiter was '%FS;'.
[dfdl = /DFDL_Test/DFDL_Test4.xsd, scd = #xscd(/schemaElement::DFDL_Test4/type::0/model::sequence/schemaElement::record/type::0/model::sequence/schemaElement::field1), 94]
Mar 29, 2014 7:20:18 AM info: Offset: 1. Finished processing element 'field1'.
[dfdl = /DFDL_Test/DFDL_Test4.xsd, scd = #xscd(/schemaElement::DFDL_Test4/type::0/model::sequence/schemaElement::record/type::0/model::sequence/schemaElement::field1), 56]
Mar 29, 2014 7:20:18 AM info: Offset: 1. Found separator '%FS;' owned by the 'sequence' group contained within element 'record'.
[dfdl = /DFDL_Test/DFDL_Test4.xsd, scd = #xscd(/schemaElement::DFDL_Test4/type::0/model::sequence/schemaElement::record/type::0/model::sequence), 106]
Mar 29, 2014 7:20:18 AM info: Offset: 2. Starting to process element 'field2'.
[dfdl = /DFDL_Test/DFDL_Test4.xsd, scd = #xscd(/schemaElement::DFDL_Test4/type::0/model::sequence/schemaElement::record/type::0/model::sequence/schemaElement::field2), 56]
Mar 29, 2014 7:20:18 AM info: Offset: 2. Found delimited value: 'PHOTO AND MUSIC' for element 'field2'. The delimiter was '%FS;'.
[dfdl = /DFDL_Test/DFDL_Test4.xsd, scd = #xscd(/schemaElement::DFDL_Test4/type::0/model::sequence/schemaElement::record/type::0/model::sequence/schemaElement::field2), 108]
Mar 29, 2014 7:20:18 AM info: Offset: 19. Finished processing element 'field2'.
[dfdl = /DFDL_Test/DFDL_Test4.xsd, scd = #xscd(/schemaElement::DFDL_Test4/type::0/model::sequence/schemaElement::record/type::0/model::sequence/schemaElement::field2), 57]
Mar 29, 2014 7:20:18 AM info: Offset: 19. Found separator '%FS;' owned by the 'sequence' group contained within element 'record'.
[dfdl = /DFDL_Test/DFDL_Test4.xsd, scd = #xscd(/schemaElement::DFDL_Test4/type::0/model::sequence/schemaElement::record/type::0/model::sequence), 107]
Mar 29, 2014 7:20:18 AM info: Offset: 20. Did not find terminator for 'record'. Expected terminator list is '%CR;%LF;%WSP*;'.
[dfdl = /DFDL_Test/DFDL_Test4.xsd, scd = #xscd(/schemaElement::DFDL_Test4/type::0/model::sequence/schemaElement::record), 104]
Mar 29, 2014 7:20:18 AM error: CTDP3042E: Terminator '%CR;%LF;%WSP*;' not found at offset '20' for element '/DFDL_Test4[1]/record[1]'.
Mar 29, 2014 7:20:18 AM fatal: CTDP3042E: Terminator '%CR;%LF;%WSP*;' not found at offset '20' for element '/DFDL_Test4[1]/record[1]'.
The actual message in BLOB:
00061cd7c8d6e3d640c1d5c440d4e4e2c9c340401c0d0a00061cc8c5c1d3e3c8c3c1d9c5404040404040401c0d0a00061cc8c5c1d3e3c8c3c1d9c51c0d0a
The DFDL.xsd settings I have made ,
<?xml version="1.0" encoding="UTF-8"?><xsd:schema xmlns:csv="http://www.ibm.com/dfdl/CommaSeparatedFormat" xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/" xmlns:ibmDfdlExtn="http://www.ibm.com/dfdl/extensions" xmlns:ibmSchExtn="http://www.ibm.com/schema/extensions" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<xsd:import namespace="http://www.ibm.com/dfdl/CommaSeparatedFormat" schemaLocation="IBMdefined/CommaSeparatedFormat.xsd"/>
<xsd:annotation>
<xsd:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:format documentFinalTerminatorCanBeMissing="yes" separatorPosition="postfix" encoding="{$dfdl:encoding}" escapeSchemeRef="csv:CSVEscapeScheme" ref="csv:CommaSeparatedFormat"/>
</xsd:appinfo>
</xsd:annotation>
<xsd:element ibmSchExtn:docRoot="true" name="DFDL_Test4">
<xsd:complexType>
<xsd:sequence dfdl:separator="">
<xsd:element dfdl:occursCountKind="implicit" dfdl:terminator="%CR;%LF;%WSP*;" maxOccurs="unbounded" name="record">
<xsd:complexType>
<xsd:sequence dfdl:separator="%FS;" dfdl:separatorPolicy="suppressedAtEndLax">
<xsd:element ibmDfdlExtn:sampleValue="value1" name="field1" type="xsd:string"/>
<xsd:element ibmDfdlExtn:sampleValue="value2" name="field2" type="xsd:string"/>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
</xsd:schema>
Could you please help me to resolve this?I am not getting a clue.
For a simple text message as,
A|B|C
D|E|F
It is parsing well.There is no problem in the terminator.It is able to pick up the records with 0D0A. |
|
| Back to top |
|
|
| kimbert |
| Posted: Sat Mar 29, 2014 3:56 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
I notice that you did not answer my third question. If you want help then please
a) post tidy, accurate, readable questions and
b) answer the questions that are asked
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| GeneRK4 |
| Posted: Sat Mar 29, 2014 7:44 pm Post subject: |
|
|
Master
Joined: 08 Jul 2013
Posts: 220
|
Hi Kimbert,
Apologies..for not answering your question..
For Mainframe message ,I am setting IBM037 as the Codepage while placing message for DFDL Test.
For normal message,I am using only UTF-8.
For Mainframe message,and normal text message ,the pattern of BLOB looks similar.
Normal Test message:(Separator-26 ; Terminator-0D0A)
46264726 480D0A49 26412642 0D0A4326
4626470D 0A
Mainframe message:(Separator-1C ; Terminator-0D0A)
061CD7C8 D6E3D640 C1D5C440 D4E4C1E2
C9C34040 1C0D0A06 1CC8C5C1 D3E3C8C3
C1D9C540 40404040 40401C0D 0A061CC8
C5C1D3E3 C8C3C1D9 C51C0D0A
For normal text message,with the properties below,it is working fine for me..
dfdl:terminator="%CR;%LF;%WSP*;"
dfdl:separatorPolicy="suppressedAtEndLax"
dfdl:separator="&"
documentFinalTerminatorCanBeMissing="yes"
For Mainframe Mainframe message with the two fields "String" datatype is wqorking fine with the properties below :
separatorPosition="postfix"
dfdl:terminator="%CR;%WSP*;"
dfdl:separatorPolicy="suppressedAtEndLax"
dfdl:separator="%FS;"
documentFinalTerminatorCanBeMissing="yes"
Note:If I only use "%CR;%WSP*;" ,the terminator is detected.If I use "%CR;%LF;%WSP*;" ,its not detecting the terminator.
For mainframe message with first field as "Short(Binary datatype with length explicitly set as 1 byte)" and second field as "String",then the same setup is not working.
If I use dfdl:terminator="%CR;%WSP*;" ,then "0A" from "0D0A" is taken into next field.
fatal: CTDP3002E: Unexpected data found at offset '22' after parsing completed. Data: '0xa...'.
If I use dfdl:terminator="%CR;%LF;%WSP*;" ,then below error is thrown,
Did not find terminator for 'record'. Expected terminator list is '%CR;%LF;%WSP*;'.
I am not sure why it is not working when I setup a field as "SHORT - binary" datatype.Please suggest me.. |
|
| Back to top |
|
|
| GeneRK4 |
| Posted: Thu Apr 10, 2014 11:45 pm Post subject: |
|
|
Master
Joined: 08 Jul 2013
Posts: 220
|
For all who can get benefit from this problem solution:
We raised a PMR for this issue.After getting suggestion from IBM , we changed the record separator to explicit byte sequence %#r0d;%#r0a;
This resolved the issue |
|
| Back to top |
|
|
| kimbert |
| Posted: Fri Apr 11, 2014 2:47 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
Thanks for posting the solution.
This is not usually necessary. In this case, the data format was using ASCII CR/LF pairs and the CCSID was being picked up from the MQ message ( so it was set to EBCDIC when running on z/OS ).
A CR/LF pair does not map to 0x0D0A in EBCDIC. So the solution was to specify the line terminator using raw byte values instead of character names.
That is what the "%#r0d;%#r0a;" does.
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| GeneRK4 |
| Posted: Tue Apr 15, 2014 12:33 am Post subject: |
|
|
Master
Joined: 08 Jul 2013
Posts: 220
|
Thanks much for providing the details behind this concept...
Even IBM has not given these details in PMR..This forum is very helpful |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
