| Author |
Message
|
| wmbv7newbie |
 Posted: Tue Apr 12, 2016 12:35 pm Post subject: |
 |
|
Centurion
Joined: 13 May 2014
Posts: 121
|
Looks like my brain is not working anymore 
I modified the body in DFDL as below -
| Quote: |
---- OrderBody occurs unbounded
-------sequence occurs exactly once
----------FieldFor2 , initiator 2 (what to keep the type of this as it is not allowing me to make it unbounded if I create as local element or local group)
--------------sequence occurs exactly once (not allowed to make unbounded)
-----------------choice, initiator 4, delimited
-----------------choice, initiator 8, delimited
|
|
|
| Back to top |
|
 |
| timber |
| Posted: Thu Apr 14, 2016 1:37 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1280
|
In my last post, I recommended this ( if you want to use Parsed Record Sequence):
| Quote: |
| Create a message set that defines an optional Header, followed by a repeating sequence group ( not a repeating element ) containing the 2-record, 4-record and 8-record, followed by an optional trailer. This will allow some invalid file contents ( it will allow no header, and/or a header with every 2,4,8 group, and/or a trailer with every 2,4,8 group and/or no trailer at all ). That's unavoidable when you use Parsed Record Sequence to parse a file with a header/trailer. You could always add some logic in the message flow to check for the illegal combinations. |
But you have made the Header non-optional in your model. That simply will not work.
When you select 'Parsed Record Sequence' the file gets split into records. But what is a 'record'. How does the parser know which part of the model is supposed to be a 'record'? It does not know. It simply parses until it reaches the end of its model. So when using Parsed Record Sequence, you must take care to design your model so that it matches only the part that you want to propagate.
I can post again with specific guidance. But first I want to hear your response to these facts. |
|
| Back to top |
|
|
| wmbv7newbie |
| Posted: Thu Apr 14, 2016 7:28 am Post subject: |
|
|
Centurion
Joined: 13 May 2014
Posts: 121
|
Thanks @timber and @mqjeff. I made some significant progress on this. Going by your suggestion, I created the DFDL as below -
| Quote: |
<?xml version="1.0" encoding="UTF-8"?><xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/" xmlns:ibmDfdlExtn="http://www.ibm.com/dfdl/extensions" xmlns:ibmSchExtn="http://www.ibm.com/schema/extensions" xmlns:recSepFieldsFmt="http://www.ibm.com/dfdl/RecordSeparatedFieldFormat">
<xsd:import namespace="http://www.ibm.com/dfdl/RecordSeparatedFieldFormat" schemaLocation="IBMdefined/RecordSeparatedFieldFormat.xsd"/>
<xsd:annotation>
<xsd:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:format encoding="{$dfdl:encoding}" escapeSchemeRef="" occursCountKind="implicit" ref="recSepFieldsFmt:RecordSeparatedFieldsFormat"/>
</xsd:appinfo>
</xsd:annotation>
<xsd:element ibmSchExtn:docRoot="true" name="outerDFDL">
<xsd:complexType>
<xsd:sequence dfdl:initiator="" dfdl:separator="">
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:escapeSchemeRef="recSepFieldsFmt:RecordEscapeScheme" dfdl:initiator="0" dfdl:length="10" dfdl:lengthKind="explicit" dfdl:terminator="%LF;" minOccurs="0" name="seqheader">
<xsd:simpleType>
<xsd:restriction base="xsd:string">
<xsd:minLength value="0"/>
<xsd:maxLength value="10"/>
</xsd:restriction>
</xsd:simpleType>
</xsd:element>
<xsd:sequence dfdl:initiator="2" dfdl:separator="" dfdl:terminator="">
<xsd:element dfdl:terminator="%LF;" name="field2" type="xsd:string"/>
<xsd:element dfdl:initiator="4" dfdl:terminator="%LF;" maxOccurs="unbounded" name="field4" type="xsd:string"/>
<xsd:element dfdl:initiator="8" dfdl:terminator="%LF;" name="field8" type="xsd:string"/>
</xsd:sequence>
<xsd:element dfdl:emptyValueDelimiterPolicy="none" dfdl:escapeSchemeRef="recSepFieldsFmt:RecordEscapeScheme" dfdl:initiator="9" dfdl:length="20" dfdl:lengthKind="explicit" dfdl:terminator="%LF;" minOccurs="0" name="seqtrailer" type="xsd:string"/>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
</xsd:schema> |
This separates each record as -
1. Header, if present,
2. block of 2,4,8, present at least once
3. Trailer, if present
Since my incoming file has fixed-length data, I have also put length validation here in case any of the lines are present (only on header, trailer as of now). This is almost all I wanted to achive.
Hopefully this will pass. |
|
| Back to top |
|
|
| wmbv7newbie |
| Posted: Thu Apr 14, 2016 9:14 am Post subject: |
|
|
Centurion
Joined: 13 May 2014
Posts: 121
|
| One problem with this model is that if I add length validations to any of the 2, 4, 8 lines and any record fails, it doesn't continue processing next record. The flow throws an exception and flow terminates. Any way to add length validations here and still continue to the next record of one fails? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Apr 14, 2016 10:25 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
As I said, you need two models. One that knows *just* enough to break things into records that is used by FileInput.
The second knows the business structure of each record, and is used inside the flow.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| wmbv7newbie |
| Posted: Thu Apr 14, 2016 12:10 pm Post subject: |
|
|
Centurion
Joined: 13 May 2014
Posts: 121
|
So removed all validations from first model as it wasnt working that way. I have now put all field level validations in second model. The first model now only knows the file structure as --
1. Header, initiator 0, occurence 0 to 1
2. Body -
Line 2, initiator 2, occurence 1
Line 4, initiator 4, occurence 1 to many
Line 8, initiator 8, occurence 1
3. Trailer, initiator 9, occurence 0 to 1
I believe this is the bare minimum validation I need at initial level. A couple of problems now -
1. How to retain the initiators after propagating from File node? I need them to be able to identify the lines in second model.
2. Lets say line 2 is missing (which is mandatory in the file). The flow throws an exception and comes back to read the next sequence but continues to start from the same erroneous sequence. Any hints on how to ask it continue with next sequence in line and skip the corrupt one. At the moment, Body is a sequence group and not choice to be able to at least validate the order of 2, 4, 8 lines in the file. I know I can solve this by making all lines also optional but....
Thanks again for your time. |
|
| Back to top |
|
|
| timber |
| Posted: Thu Apr 14, 2016 2:33 pm Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1280
|
| Quote: |
| 1. How to retain the initiators after propagating from File node? I need them to be able to identify the lines in second model. |
If you examine the bytes ( not the contents of the message tree ) that get propagated by the FileInput node, you will observe that all of the data is present. You can get the propagated bytes by calling ASBITSTREAM on the propagated message tree.
| Quote: |
| 2. Lets say line 2 is missing (which is mandatory in the file). The flow throws an exception and comes back to read the next sequence but continues to start from the same erroneous sequence. Any hints on how to ask it continue with next sequence in line and skip the corrupt one. At the moment, Body is a sequence group and not choice to be able to at least validate the order of 2, 4, 8 lines in the file. I know I can solve this by making all lines also optional but.... |
Let's say that if line 2 might be missing, then you're probably busted. You *could* try to construct an even more lax model that ignores even more of the record structure (e.g. one that uses a regular expression to match everything between the start of the header line and the end of the trailer line). But then what happens if a header or a trailer is missed out? They are also 'mandatory in the file' but if line 2 might be missing then...
For these reasons, I would pick option 1 from my original response. You'll still have a hard time figuring out what to do if a header/trailer is missing, but at least you're guaranteed to advance through the file one record at a time.
If you want to work on a structured message tree, then you can concantenate well-structured groups of records into a single string and re-parse using CREATE...PARSE DOMAIN 'DFDL' TYPE <global element name of the record structure>. |
|
| Back to top |
|
|
| wmbv7newbie |
| Posted: Thu Apr 14, 2016 10:10 pm Post subject: |
|
|
Centurion
Joined: 13 May 2014
Posts: 121
|
So I have reached the final stage where everything else works as expected except for final validation of model 2.
At second stage, I am validating the Body which contains lines of 2, 4 and 8. In my esql after File node, I am creating a DFDL message as model 2 and reparsing using RCD to save only good records and skip corrupt ones. DFDL description -
| Quote: |
<?xml version="1.0" encoding="UTF-8"?><xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/" xmlns:fmt="http://www.ibm.com/dfdl/GeneralPurposeFormat" xmlns:ibmSchExtn="http://www.ibm.com/schema/extensions">
<xsd:import namespace="http://www.ibm.com/dfdl/GeneralPurposeFormat" schemaLocation="IBMdefined/GeneralPurposeFormat.xsd"/>
<xsd:element dfdl:encoding="UTF-8" dfdl:lengthKind="implicit" ibmSchExtn:docRoot="true" name="insideDFDL">
<xsd:complexType>
<xsd:sequence dfdl:encoding="UTF-8" dfdl:separator="">
<xsd:element dfdl:encoding="UTF-8" dfdl:length="66" dfdl:lengthKind="explicit" dfdl:outputNewLine="%LF;" dfdl:terminator="%LF;" name="OrderItemsRecord" type="OrderItemsRecord"/>
<xsd:element dfdl:encoding="UTF-8" dfdl:length="303" dfdl:lengthKind="explicit" dfdl:outputNewLine="%LF;" dfdl:terminator="%LF;" name="OrderCustDetailsRecord" type="OrderCustDetailsRecord"/>
<xsd:element dfdl:encoding="UTF-8" dfdl:length="13" dfdl:lengthKind="explicit" dfdl:outputNewLine="%LF;" name="OrderSummaryRecord" type="OrderSummaryRecord"/>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
<xsd:complexType name="OrderSummaryRecord">
<xsd:sequence dfdl:encoding="UTF-8" dfdl:initiator="" dfdl:outputNewLine="%LF;" dfdl:separator="" dfdl:terminator="%LF;">
<xsd:element dfdl:length="7" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="TotalLineQuantity" type="xsd:string"/>
<xsd:element dfdl:length="1" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="Blank1" type="xsd:string"/>
<xsd:element dfdl:length="4" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="TotalNoOfLines" type="xsd:string"/>
<xsd:element dfdl:length="1" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="Blank1" type="xsd:string"/>
</xsd:sequence>
</xsd:complexType>
<xsd:complexType name="OrderItemsRecord">
<xsd:sequence dfdl:encoding="UTF-8" dfdl:initiator="" dfdl:outputNewLine="%LF;" dfdl:separator="" dfdl:terminator="%LF;">
<xsd:element dfdl:length="4" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="LineNumber" type="xsd:string"/>
<xsd:element dfdl:length="10" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="ItemISBN" type="xsd:string"/>
<xsd:element dfdl:length="7" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="Quantity" type="xsd:string"/>
<xsd:element dfdl:length="1" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="QuantitySign" type="xsd:string"/>
<xsd:element dfdl:length="2" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="ServiceLevel" type="xsd:string"/>
<xsd:element dfdl:length="40" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="Text" type="xsd:string"/>
<xsd:element dfdl:length="2" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="Blank2" type="xsd:string"/>
</xsd:sequence>
</xsd:complexType>
<xsd:complexType name="OrderCustDetailsRecord">
<xsd:sequence dfdl:encoding="UTF-8" dfdl:initiator="" dfdl:outputNewLine="%LF;" dfdl:separator="" dfdl:terminator="">
<xsd:element dfdl:length="10" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="OrderNumber" type="xsd:string"/>
<xsd:element dfdl:encoding="{$dfdl:encoding}" dfdl:length="30" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="CustomerName" type="xsd:string"/>
<xsd:element dfdl:length="30" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="Address1" type="xsd:string"/>
<xsd:element dfdl:length="30" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="Address2" type="xsd:string"/>
<xsd:element dfdl:length="30" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="Address3" type="xsd:string"/>
<xsd:element dfdl:length="30" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="Address4" type="xsd:string"/>
<xsd:element dfdl:length="30" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="Address5" type="xsd:string"/>
<xsd:element dfdl:length="12" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="PostCode" type="xsd:string"/>
<xsd:element dfdl:length="4" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="CountryCode" type="xsd:string"/>
<xsd:element dfdl:length="20" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="Blank20" type="xsd:string"/>
<xsd:element dfdl:length="8" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="DespatchDate" type="xsd:string"/>
<xsd:element dfdl:length="8" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="Blank8" type="xsd:string"/>
<xsd:element dfdl:length="10" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="AccountNumber" type="xsd:string"/>
<xsd:element dfdl:length="1" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="Blank1" type="xsd:string"/>
<xsd:element dfdl:length="20" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="TelephoneNumber" type="xsd:string"/>
<xsd:element dfdl:length="30" dfdl:lengthKind="explicit" dfdl:textPadKind="padChar" name="City" type="xsd:string"/>
</xsd:sequence>
</xsd:complexType>
<xsd:annotation>
<xsd:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:format encoding="{$dfdl:encoding}" escapeSchemeRef="" ref="fmt:GeneralPurposeFormat"/>
</xsd:appinfo>
</xsd:annotation>
</xsd:schema> |
A corrupt record is behaving as expected and I am getting an exception to indicate the length or field issue but a good record is not getting parsed and the flow terminates unexpectedly.
Sample good record -
| Quote: |
22106152272ABCDE FDRster ABC Innsbruck Inst. f. Zoologie Technikerstrasse 25 6020,Innsbruck,IIHHTGF 6020 45009035 2272 Y10373136 +43-(0)-111-666 6180Innseerrd
4410218477648860000001 1
88000001 0001
|
where the tree coming out of compute node before RCD looks like -
| Quote: |
OrderItemsRecord 410218477648860000001 1
OrderCustDetailsRecord 2406152272ABCDE FDRster ABC Innsbruck Inst. f. Zoologie Technikerstrasse 25 6020,Innsbruck,IIHHTGF 6020 45009035 2272 Y10373136 +43-(0)-111-666 6180Innseerrd
OrderSummaryRecord 8000001 0001
|
|
|
| Back to top |
|
|
| timber |
| Posted: Fri Apr 15, 2016 1:28 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1280
|
| Quote: |
| a good record is not getting parsed and the flow terminates unexpectedly. |
You mean, there is no exception list? If true, that would indicate a product defect. First, double check that you are debugging this correctly. If you can't do that using the flow debugger then I strongly recommend that you take a debug-level user trace ( I can assist with the syntax if necessary ). |
|
| Back to top |
|
|
| wmbv7newbie |
| Posted: Fri Apr 15, 2016 4:12 am Post subject: |
|
|
Centurion
Joined: 13 May 2014
Posts: 121
|
I have connected both the out and failure terminals of RCD to mq but nothing comes there. I am just getting some unreadable exception with only mqrfh2 in the error queue through the logger subflow attached to beginning of flow. MQRFH2 data -
| Quote: |
00000 52 46 48 20 02 00 00 00--58 14 00 00 22 02 00 00 |RFH ....X..."...|
00010 B8 04 00 00 20 20 20 20--20 20 20 20 00 00 00 00 |�... ....|
00020 B8 04 00 00 10 14 00 00--3C 75 73 72 3E 3C 45 78 |�.......<usr><Ex|
00030 63 65 70 74 69 6F 6E 4D--65 74 61 64 61 74 61 3E |ceptionMetadata>|
00040 3C 41 70 70 6C 69 63 61--74 69 6F 6E 4E 61 6D 65 |<ApplicationName|
000E0 61 74 65 54 69 6D 65 3E--32 30 31 36 2D 30 34 2D |ateTime>2016-04-|
000F0 31 35 20 31 33 3A 30 33--3A 34 37 2E 36 30 37 38 |15 13:03:47.6078|
00100 34 32 3C 2F 44 61 74 65--54 69 6D 65 3E 3C 42 6F |42</DateTime><Bo|
00110 64 79 50 61 72 73 65 72--3E 58 4D 4C 4E 53 43 3C |dyParser>XMLNSC<|
00120 2F 42 6F 64 79 50 61 72--73 65 72 3E 3C 4D 65 73 |/BodyParser><Mes|
00130 73 61 67 65 3E 55 6E 61--62 6C 65 20 74 6F 20 65 |sage>Unable to e|
00140 78 74 72 61 63 74 20 65--78 63 65 70 74 69 6F 6E |xtract exception|
00150 20 6D 65 73 73 61 67 65--21 3C 2F 4D 65 73 73 61 | message!</Messa|
00160 67 65 3E 3C 53 65 76 65--72 69 74 79 3E 33 3C 2F |ge><Severity>3</|
00170 53 65 76 65 72 69 74 79--3E 3C 45 72 72 6F 72 43 |Severity><ErrorC|
00180 6F 64 65 3E 32 36 35 32--3C 2F 45 72 72 6F 72 43 |ode>2652</ErrorC|
002E0 67 65 50 72 6F 70 65 72--74 69 65 73 3E 3C 50 72 |geProperties><Pr|
002F0 6F 70 65 72 74 79 3E 3C--43 61 74 65 67 6F 72 79 |operty><Category|
00300 3E 50 72 6F 70 65 72 74--69 65 73 3C 2F 43 61 74 |>Properties</Cat|
00310 65 67 6F 72 79 3E 3C 4B--65 79 3E 45 6E 63 6F 64 |egory><Key>Encod|
00320 69 6E 67 3C 2F 4B 65 79--3E 3C 56 61 6C 75 65 3E |ing</Key><Value>|
00330 35 34 36 3C 2F 56 61 6C--75 65 3E 3C 2F 50 72 6F |546</Value></Pro|
00340 70 65 72 74 79 3E 3C 50--72 6F 70 65 72 74 79 3E |perty><Property>|
00350 3C 43 61 74 65 67 6F 72--79 3E 50 72 6F 70 65 72 |<Category>Proper|
00360 74 69 65 73 3C 2F 43 61--74 65 67 6F 72 79 3E 3C |ties</Category><|
00370 4B 65 79 3E 43 6F 64 65--64 43 68 61 72 53 65 74 |Key>CodedCharSet|
00380 49 64 3C 2F 4B 65 79 3E--3C 56 61 6C 75 65 3E 31 |Id</Key><Value>1|
00390 32 30 38 3C 2F 56 61 6C--75 65 3E 3C 2F 50 72 6F |208</Value></Pro|
003A0 70 65 72 74 79 3E 3C 50--72 6F 70 65 72 74 79 3E |perty><Property>|
003B0 3C 43 61 74 65 67 6F 72--79 3E 50 72 6F 70 65 72 |<Category>Proper|
003C0 74 69 65 73 3C 2F 43 61--74 65 67 6F 72 79 3E 3C |ties</Category><|
003D0 4B 65 79 3E 54 72 61 6E--73 61 63 74 69 6F 6E 61 |Key>Transactiona|
003E0 6C 3C 2F 4B 65 79 3E 3C-- |l</Key>< |
|
Nothing comes in User trace too.
Last edited by wmbv7newbie on Fri Apr 15, 2016 2:41 pm; edited 1 time in total |
|
| Back to top |
|
|
| timber |
| Posted: Fri Apr 15, 2016 4:58 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1280
|
| Can you post the user trace. It might contain a clue at to where exactly the problem is occurring. Ideally, add a Trace node at the start of your flow with Pattern set to ${Root} and Destination set to User Trace. |
|
| Back to top |
|
|
| wmbv7newbie |
| Posted: Fri Apr 15, 2016 1:36 pm Post subject: |
|
|
Centurion
Joined: 13 May 2014
Posts: 121
|
For now I am doing a substring to extract the fields from the lines due to shortage of time and since I had already validated lengths at previous stage.
Would post the trace tests over the weekend to see if this could be solved. It will make the solution decent at least . |
|
| Back to top |
|
|
|
|
