| Author |
Message
|
| giorginus80 |
 Posted: Mon Feb 02, 2009 6:28 am Post subject: Fixed length question |
 |
|
Centurion
Joined: 08 Jul 2008
Posts: 124
Location: Rome, Italy
|
Hi,
I have a fixed length file but I can't parse it in a MRM, because I got "Not all the buffer was used when reading message":
the file is like this: (100 bytes each row)
| Code: |
AA20090124
271460001E00000000003470520090121200901240102791239021010279030294F000001 0294
271460001E00000000004566620090121200901240102791239021010279030295F000002 0295
271460001E00000000004783520090121200901240102791239021010279030299F000003 0299
271730041E00000000006109020090121200901240102791239021010279030293F000001 0293
ZZ20090124000000311
|
and this is my mxsd:
| Code: |
<?xml version="1.0" encoding="UTF-8"?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:io="http://www.test.it" targetNamespace="http://www.test.it">
<xsd:complexType name="headerPosteFtpType">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsStructRep dataElementSeparation="FixedLength" groupIndicator="" groupTerminator="<CR><LF>" messageSetDefaultRep="CSV1"/>
</xsd:appinfo>
</xsd:annotation>
<xsd:sequence>
<xsd:element name="idInizio" type="xsd:string">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsElemRep length="2" messageSetDefaultRep="CSV1" precision="-1"/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
<xsd:element name="dataRiferimentoDati" type="xsd:string">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsElemRep length="8" messageSetDefaultRep="CSV1" precision="-1"/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
<xsd:element name="filler1" type="xsd:string">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsElemRep length="90" messageSetDefaultRep="CSV1" precision="-1"/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
</xsd:sequence>
</xsd:complexType>
<xsd:complexType name="versamentoFtpType">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsStructRep dataElementSeparation="FixedLength" groupTerminator="<CR><LF>" messageSetDefaultRep="CSV1"/>
</xsd:appinfo>
</xsd:annotation>
<xsd:sequence>
<xsd:element name="numeroConto" type="xsd:string">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsElemRep length="8" messageSetDefaultRep="CSV1" precision="-1"/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
<xsd:element name="cuasProvenienza" type="xsd:string">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsElemRep length="1" messageSetDefaultRep="CSV1" precision="-1"/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
<xsd:element name="valuta" type="xsd:string">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsElemRep length="1" messageSetDefaultRep="CSV1" precision="-1"/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
<xsd:element name="importo" type="xsd:string">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsElemRep length="15" messageSetDefaultRep="CSV1" precision="-1"/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
<xsd:element name="dataAccettazione" type="xsd:string">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsElemRep length="8" messageSetDefaultRep="CSV1" precision="-1"/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
<xsd:element name="dataAllibramento" type="xsd:string">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsElemRep length="8" messageSetDefaultRep="CSV1" precision="-1"/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
<xsd:element name="frazionarioUfficio" type="xsd:string">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsElemRep length="6" messageSetDefaultRep="CSV1" precision="-1"/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
<xsd:element name="tipoDocumento" type="xsd:string">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsElemRep length="3" messageSetDefaultRep="CSV1" precision="-1"/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
<xsd:element name="datiCliente" type="xsd:string">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsElemRep length="16" messageSetDefaultRep="CSV1" precision="-1"/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
<xsd:element name="idImmagineFrontale" type="xsd:string">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsElemRep length="7" messageSetDefaultRep="CSV1" precision="-1"/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
<xsd:element name="filler" type="xsd:string">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsElemRep length="7" messageSetDefaultRep="CSV1" precision="-1"/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
<xsd:element name="vcy" type="xsd:string">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsElemRep length="4" messageSetDefaultRep="CSV1" precision="-1"/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
<xsd:element name="filler2" type="xsd:string">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsElemRep length="16" messageSetDefaultRep="CSV1" precision="-1"/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
</xsd:sequence>
</xsd:complexType>
<xsd:complexType name="footerPosteFtpType">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsStructRep dataElementSeparation="FixedLength" groupIndicator="" groupTerminator="<CR><LF>" messageSetDefaultRep="CSV1"/>
</xsd:appinfo>
</xsd:annotation>
<xsd:sequence>
<xsd:element name="idFine" type="xsd:string">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsElemRep length="2" messageSetDefaultRep="CSV1" precision="-1"/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
<xsd:element name="dataRiferimentoDati" type="xsd:string">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsElemRep length="8" messageSetDefaultRep="CSV1" precision="-1"/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
<xsd:element name="numeroVersamenti" type="xsd:string">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsElemRep length="9" messageSetDefaultRep="CSV1" precision="-1"/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
<xsd:element name="filler3" type="xsd:string">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsElemRep length="81" messageSetDefaultRep="CSV1" precision="-1"/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
</xsd:sequence>
</xsd:complexType>
<xsd:complexType name="versamentiType">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsStructRep dataElementSeparation="UseDataPattern" groupTerminator="" messageSetDefaultRep="CSV1"/>
</xsd:appinfo>
</xsd:annotation>
<xsd:sequence>
<xsd:element name="header" type="io:headerPosteFtpType">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsElemRep dataPattern="^[AA][A-Za-z0-9]+" messageSetDefaultRep="CSV1" precision="-1"/>
<tdsInclRep dataPattern="^[AA][A-Za-z0-9]+" messageSetDefaultRep="CSV1"/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
<xsd:element maxOccurs="unbounded" minOccurs="0" name="versamento" type="io:versamentoFtpType">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsElemRep dataPattern="[0-9]{8}[A-Za-z0-9]+" messageSetDefaultRep="CSV1" precision="-1"/>
<tdsInclRep dataPattern="[0-9]{8}[A-Za-z0-9]+" messageSetDefaultRep="CSV1" repeatingElementDelimiter=""/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
<xsd:element name="footer" type="io:footerPosteFtpType">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsElemRep dataPattern="^[ZZ][A-Za-z0-9]+" messageSetDefaultRep="CSV1" precision="-1"/>
<tdsInclRep dataPattern="^[ZZ][A-Za-z0-9]+" messageSetDefaultRep="CSV1"/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
</xsd:sequence>
</xsd:complexType>
<xsd:complexType name="posteFtpType">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsStructRep dataElementSeparation="FixedLength" groupTerminator="" messageSetDefaultRep="CSV1"/>
</xsd:appinfo>
</xsd:annotation>
<xsd:sequence>
<xsd:element maxOccurs="1" minOccurs="1" name="versamenti" type="io:versamentiType">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<tdsElemRep dataPattern="" messageSetDefaultRep="CSV1" precision="-1"/>
<tdsInclRep dataPattern="" messageSetDefaultRep="CSV1" repeatingElementDelimiter=""/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
</xsd:sequence>
</xsd:complexType>
<xsd:element name="PosteFTP" type="io:posteFtpType">
<xsd:annotation>
<xsd:appinfo source="WMQI_APPINFO">
<MRMessage messageDefinition="/0/PosteFTP;XSDElementDeclaration$MRObject"/>
</xsd:appinfo>
</xsd:annotation>
</xsd:element>
</xsd:schema>
|
I can't parse my fixed length file. The file is fixed each row (header and footer included) by 100bytes. I can recognize header by the first 2 words AA and the footer by ZZ I used the data pattern in the mxsd. So how can I do? Is there any way or my mxsd is wrong? Am I missing something? |
|
| Back to top |
|
 |
| Vitor |
| Posted: Mon Feb 02, 2009 6:35 am Post subject: Re: Fixed length question |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| giorginus80 wrote: |
| Is there any way or my mxsd is wrong? Am I missing something? |
One of us is. Are you entirely certain the definition the non-header data describes 100 bytes? I come up a bit short when I add the lengths together (which might just mean I need to switch from coffee to water earlier than usual).
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| giorginus80 |
| Posted: Mon Feb 02, 2009 6:39 am Post subject: Re: Fixed length question |
|
|
Centurion
Joined: 08 Jul 2008
Posts: 124
Location: Rome, Italy
|
| Vitor wrote: |
| giorginus80 wrote: |
| Is there any way or my mxsd is wrong? Am I missing something? |
One of us is. Are you entirely certain the definition the non-header data describes 100 bytes? I come up a bit short when I add the lengths together (which might just mean I need to switch from coffee to water earlier than usual). |
Yes sure, each row is delimited by <CR><LF> and the length is 100. But may be I don't know how use the data pattern! |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Feb 02, 2009 6:45 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
So, if your data is delimited, why are you not trying to use a delimiter?
And given that there is no Comma Seperated Values, why are you using the CSV sample as the base for your model?
This is straight forward Fixed Length messaging. You do not require any use of Data Patterns at all, and only need to use Tagged-Fixed Length messaging if you do not have a count of how many instances of the body record will occur in the header record.
A usual cause of "Not all buffer was used in processing the message" is because you have managed not to model your repeating structure very well. |
|
| Back to top |
|
|
| giorginus80 |
| Posted: Mon Feb 02, 2009 7:02 am Post subject: |
|
|
Centurion
Joined: 08 Jul 2008
Posts: 124
Location: Rome, Italy
|
| mqjeff wrote: |
So, if your data is delimited, why are you not trying to use a delimiter?
And given that there is no Comma Seperated Values, why are you using the CSV sample as the base for your model?
This is straight forward Fixed Length messaging. You do not require any use of Data Patterns at all, and only need to use Tagged-Fixed Length messaging if you do not have a count of how many instances of the body record will occur in the header record.
A usual cause of "Not all buffer was used in processing the message" is because you have managed not to model your repeating structure very well. |
I mean there is <CR><LF> at the end of each row. Internally the row is length fixed. |
|
| Back to top |
|
|
| kimbert |
| Posted: Mon Feb 02, 2009 7:06 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
Hi,
This is a fairly straightforward problem, if you approach it in the right way.
Your input message is composed of fixed-length records which are identified by tags. In other words, it is a 'Tagged Fixed Length' message.
You need to
- Forget about Data Patterns. They should only be used when other techniques don't work
- On the outermost complex type, set the Data Element Separation to 'Tagged Fixed Length'
- On the outermost complex type, set the Group Indicator to the line ending character(s). Probably <CR>, <LF> or <CR><LF>
- On the element for each fixed-length structure, set the Tag.
If it doesn't work, take a user trace. From v6.1, TDS has been pretty good at describing what it is doing in user trace. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Feb 02, 2009 7:08 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| giorginus80 wrote: |
| I mean there is <CR><LF> at the end of each row. Internally the row is length fixed. |
Granted, and I've added up the lengths again so confirming I do need to lay off the coffee.
So why are you not reading the file a row at a time, using the <CR><LF> as a delimiter (or getting the FileInput node to cut it up) and using the front 2 bytes to identify the structure? If you need to?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Feb 02, 2009 7:08 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| giorginus80 wrote: |
| I mean there is <CR><LF> at the end of each row. Internally the row is length fixed. |
Yes.
That's a delimiter.
It delimits the end of one row from the start of another.
Again, if the header record contains a number that tells you exactly how many times the body record repeats, you can even ignore the delimiter as a delimiter and just include it as a two-character string. |
|
| Back to top |
|
|
| giorginus80 |
| Posted: Mon Feb 02, 2009 8:03 am Post subject: |
|
|
Centurion
Joined: 08 Jul 2008
Posts: 124
Location: Rome, Italy
|
That's ok, if the body has a delimiter...but in this case I have a delimiter AA for Header and ZZ for footer, but the internal body has no delimiter
For example
| Code: |
AA20090124
342560001E00000000003470520090121200901240102791239021010279030294F000001 0294
271460001E00000000004566620090121200901240102791239021010279030295F000002 0295
738960001E00000000004783520090121200901240102791239021010279030299F000003 0299
290230041E00000000006109020090121200901240102791239021010279030293F000001 0293
ZZ20090124000000311
|
Last edited by giorginus80 on Mon Feb 02, 2009 8:11 am; edited 1 time in total |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Feb 02, 2009 8:09 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| giorginus80 wrote: |
| but the internal body has no delimiter |
I thought you said each row ended with a <CR><LF>. That's certainly what it looks like from your screen print.
And technically the AA & ZZ are prefixes, not delimiters.
But even then, you know the length of the internal rows.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| giorginus80 |
| Posted: Mon Feb 02, 2009 8:15 am Post subject: |
|
|
Centurion
Joined: 08 Jul 2008
Posts: 124
Location: Rome, Italy
|
| Vitor wrote: |
| giorginus80 wrote: |
| but the internal body has no delimiter |
I thought you said each row ended with a <CR><LF>. That's certainly what it looks like from your screen print.
And technically the AA & ZZ are prefixes, not delimiters.
But even then, you know the length of the internal rows. |
I mean at the end of each row there is a curriage return (CR)
The file is like:
| Code: |
AA20090124
342560001E00000000003470520090121200901240102791239021010279030294F000001 0294
271460001E00000000004566620090121200901240102791239021010279030295F000002 0295
738960001E00000000004783520090121200901240102791239021010279030299F000003 0299
290230041E00000000006109020090121200901240102791239021010279030293F000001 0293
ZZ20090124000000311
|
If there was a tag in the body lines it was ok, I can detect header and footer, but I have problem detecting the body. For this I tried with data pattern, but with no result  |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Feb 02, 2009 8:17 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| giorginus80 wrote: |
| I can detect header and footer, but I have problem detecting the body. |
No you don't
| Bharat_123 wrote: |
| For this I tried with data pattern, but with no result |
Why do you persist with this? More than one person has told you that it's unnecessary. Which it is.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| giorginus80 |
| Posted: Mon Feb 02, 2009 8:18 am Post subject: |
|
|
Centurion
Joined: 08 Jul 2008
Posts: 124
Location: Rome, Italy
|
| Vitor wrote: |
| giorginus80 wrote: |
| I can detect header and footer, but I have problem detecting the body. |
No you don't
| Bharat_123 wrote: |
| For this I tried with data pattern, but with no result |
Why do you persist with this? More than one person has told you that it's unnecessary. Which it is. |
It's ok but how can I detect body ? |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Feb 02, 2009 8:22 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| giorginus80 wrote: |
| It's ok but how can I detect body ? |
| kimbert wrote: |
You need to
- Forget about Data Patterns. They should only be used when other techniques don't work
- On the outermost complex type, set the Data Element Separation to 'Tagged Fixed Length'
- On the outermost complex type, set the Group Indicator to the line ending character(s). Probably <CR>, <LF> or <CR><LF>
- On the element for each fixed-length structure, set the Tag.
|
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Feb 02, 2009 8:22 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
The header is a fixed length piece of data that occurs once.
It is followed by a set of fixed length records that repeat.
If the header tells you how many repeats of the body there are, then you can build the model to know that.
If the header does not tell you how many repeats of the body there are, then you can use the tag on the trailer to tell you that the trailer has occurred instead of another repeat of the body. |
|
| Back to top |
|
|
|
|
