| Author |
Message
|
| Thomas2ab |
 Posted: Mon Jun 09, 2014 5:52 pm Post subject: DFDL Position |
 |
|
Acolyte
Joined: 07 Mar 2014
Posts: 51
|
Hello everyone,
I am starting to use DFDL within WMB 8.0.0.1 and I was wondering if it could work based on positional rules?
Let me explain:
My input message is gonna be one line (or more) without explicit delimiter. It is going to be a huge line of almost 1000 characters and the rules are like:
- From the 1st character to the 10th it represents the company name
- From the 11th character to the 15th it represents the company ID
- From the 50st character to the 60st it represents the company creation date (as you can see the line could be fill with blank and not every character has a meaning or even exist).
etc...
until almost 1000 characters.
I would like to create a xsd and parse it in order to then map it to a xml file, but from what I saw it seems to only work with a specific delimiter like ';' or '|' or something else like this. In my case it is all positional.
Thank you for your help,
Thomas |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Mon Jun 09, 2014 8:22 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
If it makes it easier for you, you could create a c structure or a cobol copy book and import from there... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| kimbert |
| Posted: Tue Jun 10, 2014 2:49 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| from what I saw it seems to only work with a specific delimiter like ';' or '|' or something else like this. |
No - that is not correct. You probably got that impression because you created your DFDL model using one of the wizards (which do assume that you are dealing with delimited data).
DFDL can read and write almost any data format. Text or binary. Fixed length or delimited. Tagged or positional. For fixed-length data, set lengthKind='explicit' and set the length on each field.
You will probably want to set lengthUnits='characters' ( not 'bytes' ). Make sure that you get the character encoding correct - it defaults to UTF-8 which is not likely to be the correct choice for your format.
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| Thomas2ab |
| Posted: Tue Jun 10, 2014 3:57 am Post subject: |
|
|
Acolyte
Joined: 07 Mar 2014
Posts: 51
|
Thank you kimbert.
So you suggest me to create the schema manually instead of using the wizard?
would you have an example of a xsd (or at least a part of it) that delimit data the way I'm willing to please?
Thank you for your help,
Regards |
|
| Back to top |
|
|
| kimbert |
| Posted: Tue Jun 10, 2014 7:13 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
IBM does not provide a wizard for modelling fixed-length data, so you can either
a) create the model from scratch ( select 'Other text or binary' and then select 'Create an empty DFDL file' ).
or
b) create a COBOL copybook that specifies what the data looks like, and then import that.
No need for me to do anything for you. Not yet, anyway.
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| Thomas2ab |
| Posted: Tue Jun 10, 2014 10:19 am Post subject: |
|
|
Acolyte
Joined: 07 Mar 2014
Posts: 51
|
Thanks kimbert, you are right.
Last question (for now), I never used COBOL, do you think it will then be much more complicated for me to create a COBOL copybook or not necessarily?
Thanks,
Regards, |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Jun 10, 2014 10:33 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Thomas2ab wrote: |
Thanks kimbert, you are right.
Last question (for now), I never used COBOL, do you think it will then be much more complicated for me to create a COBOL copybook or not necessarily?
Thanks,
Regards, |
Have you ever created a C structure? There is also a wizard to import C structures.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Thomas2ab |
| Posted: Tue Jun 10, 2014 11:00 am Post subject: |
|
|
Acolyte
Joined: 07 Mar 2014
Posts: 51
|
Yes I did, it might be easier.
But actually I found some tutorials about COBOL copybook that doesn't seem that complicated.
But is there a way in a C structure to define like 50 characters that would be 'spaces'?
Furthermore, in my case I have like the first 4 fields that are separated by ';' and then the 6th one is that long line I talked about in my first message.
That would look like:
Value1;Value2;Value3;Value4;Company 2598 56 NYC 69885 LegalEntity 5699663252141ACB
Somthing like that...Would that be possible to combine it in a COBOL copybook or C structure then?
Thank you very much for your help,
Regards, |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Jun 10, 2014 11:21 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Kimbert can confirm. I would think that you'd model that with 2 distinct DFDL models, one for the delimited part and one for the fixed part and then combine into a message.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| kimbert |
| Posted: Tue Jun 10, 2014 12:42 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
Urk! Words. They really get in the way when discussing parsing rules.
In your first post, you said:
| Quote: |
I am starting to use DFDL within WMB 8.0.0.1 and I was wondering if it could work based on positional rules?
...
My input message is gonna be one line (or more) without explicit delimiter. |
But your latest post shows an example that includes delimiters on the first few fields.
This is not a problem - DFDL is supremely flexible about lengths. fjb_saper will be amazed to learn that in DFDL you can mix delimited fields with fixed-length fields in the same structure. Just set lengthKind to 'delimited' or 'explicit' on each field, according to the need.
Now to your example:
| Code: |
| Value1;Value2;Value3;Value4;Company 2598 56 NYC 69885 LegalEntity 5699663252141ACB |
You need a model that looks like this:
| Code: |
element name='singleLineMessage'
sequence separator=";" separatorSuppressionPolicy="suppressedAtEndLax"
element name='field1' lengthKind='delimited'
element name='field2' lengthKind='delimited'
element name='field3' lengthKind='delimited'
element name='field4' lengthKind='delimited'
element name='field5' lengthKind='delimited'
element name='field6' lengthKind='delimited'
element name='fixedLengthRecord' lengthKind='implicit'
sequence separator=""
element name='fixed1' lengthKind='explicit' length='7'
element name='fixed2' lengthKind='explicit' length='5'
element name='fixed3' lengthKind='explicit' length='2'
element name='fixed4' lengthKind='explicit' length='10'
...
|
Before you rely on that too much, you may like to check something. The 'fixed length' record at the end of your example does not contain very many padding spaces. Are you sure that the fields are really fixed-length, and are not delimited by ( one or more ) spaces?
Again, not a problem for DFDL either way, but you need to describe the data format accurately if you want the DFDL parser to interpret it correctly.
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| Thomas2ab |
| Posted: Tue Jun 10, 2014 12:49 pm Post subject: |
|
|
Acolyte
Joined: 07 Mar 2014
Posts: 51
|
Thanks Kimbert.
Actually at the beginning I wanted to check if it was possible to do DFDL with positionnal rule and I didn't want to already mix the problem with delimitated values.
Regarding my sample, actually it won't be exactly like this, there will have much more spaces. I just wanted to give kind of an easy reading sample just to give an overview of the situation.
I will try this and let you know.
Thanks!!!!!! |
|
| Back to top |
|
|
| Thomas2ab |
| Posted: Tue Jun 10, 2014 2:32 pm Post subject: |
|
|
Acolyte
Joined: 07 Mar 2014
Posts: 51
|
Hello Kimbert,
So I tried and I got this XSD:
| Code: |
<?xml version="1.0" encoding="UTF-8"?>
<xsd:schema xmlns:csv="http://www.ibm.com/dfdl/CommaSeparatedFormat"
xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<xsd:import namespace="http://www.ibm.com/dfdl/CommaSeparatedFormat"
schemaLocation="IBMdefined/CommaSeparatedFormat.xsd" />
<xsd:annotation>
<xsd:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:format documentFinalTerminatorCanBeMissing="yes"
encoding="{$dfdl:encoding}" escapeSchemeRef=""
ref="csv:CommaSeparatedFormat" />
</xsd:appinfo>
</xsd:annotation>
<xsd:element name="singleLineMessage">
<xsd:complexType>
<xsd:sequence dfdl:separator=";" dfdl:separatorPolicy="suppressedAtEndLax">
<xsd:element name="field1" type="xsd:string"
dfdl:lengthKind='delimited' />
<xsd:element name="field2" type="xsd:string"
dfdl:lengthKind='delimited' />
<xsd:element name="field3" type="xsd:string"
dfdl:lengthKind='delimited' />
<xsd:element name="field4" type="xsd:string"
dfdl:lengthKind='delimited' />
<xsd:element name="fixedLengthRecord">
<xsd:complexType>
<xsd:sequence dfdl:separator="" dfdl:lengthKind='implicit'>
<xsd:element name="fixedA" type="xsd:string"
dfdl:lengthKind='explicit' dfdl:length='7' />
<xsd:element name="fixedB" type="xsd:string"
dfdl:lengthKind='explicit' dfdl:length='5' />
<xsd:element name="fixedC" type="xsd:string"
dfdl:lengthKind='explicit' dfdl:length='10' />
</xsd:sequence>
</xsd:complexType>
</xsd:element>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
</xsd:schema>
|
But it is not generating only one line and it does not seem to validate the length of the fixed element.
Am I forgeting something?
Thank you,
Regards, |
|
| Back to top |
|
|
| kimbert |
| Posted: Wed Jun 11, 2014 5:04 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
You need to read the DFDL trace and see what the DFDL parser or serializer has done with your data. If you cannot understand what you are seeing then please post examples and actual outputs and required outputs. Words are simply not precise enough.
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| Thomas2ab |
| Posted: Wed Jun 11, 2014 8:36 am Post subject: |
|
|
Acolyte
Joined: 07 Mar 2014
Posts: 51
|
Hi kimbert.
You are right. I actually tried with a sample input xml message and it worked.
The only thing is that all the fixed fields arebeing set one to each other without space.
How would I set in my message definition that there would have like a bloc of 50 spaces? Or even better, that for example the element name='fixed3' starts at the 154th position within the fixedLengthRecord element?
Is there a way to set a start position? Or shoulf I add space elements in order to reach the desired position?
Thanks!!
Regards, |
|
| Back to top |
|
|
| kimbert |
| Posted: Wed Jun 11, 2014 1:57 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
The only thing is that all the fixed fields are being set one to each other without space.
How would I set in my message definition that there would have like a bloc of 50 spaces? |
Set lengthKind='explicit' and length=50. Then set the padKind and padCharacter properties. You should probably set trimKind as well, so that the parser removes the padding after reading a field.
| Quote: |
Or even better, that for example the element name='fixed3' starts at the 154th position within the fixedLengthRecord element?
Is there a way to set a start position? |
No - the DFDL parser will parse every field up to and including the one that you ask for. It finds the position of the Nth field by parsing the previous N-1 fields. You could, of course, merge a whole sequence of fields into a single fixed-length field by manually adding up the lengths. But then you would not be able to access the individual fields (obviously).
| Quote: |
| Or should I add space elements in order to reach the desired position? |
Definitely not. How would you calculate the length of those space elements? You need padding/trimming not artificial 'space elements'.
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
|
|
