| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| Can we handle fields variation count using DFDL ??? |
« View previous topic :: View next topic » |
| Author |
Message
|
| Poli091987 |
 Posted: Tue Mar 22, 2016 1:58 am Post subject: |
 |
|
Novice
Joined: 10 Dec 2012
Posts: 16
|
You have a DFDL model for a CSV record --- Yes
The model has 37 fields ( YES or NO ). -- Yes
ALL fields in the model have minOccurs = 1, maxOccurs =1 --- No --- We have have minoccurs = 0 and maxoccurs = 1 |
|
| Back to top |
|
 |
| Poli091987 |
| Posted: Tue Mar 22, 2016 2:01 am Post subject: |
|
|
Novice
Joined: 10 Dec 2012
Posts: 16
|
Can you explain clearly what you mean by "a field does not come up"? --- If no value is present then they should be sending atleast Space in the message separated by comma.
Then answer the following:
You have a DFDL model for a CSV record --- Yes
The model has 37 fields --- Yes
ALL fields in the model have minOccurs = 1, maxOccurs =1 --- No --- minoccurs =0 and maxoccurs =1 |
|
| Back to top |
|
|
| timber |
| Posted: Tue Mar 22, 2016 2:32 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1280
|
First, let's get some simple facts out in the open.
1. If your data format *can* be parsed, then DFDL can parse it.
2. ...and write it, if necessary.
3. Missing one or more optional fields from the end of a line is standard CSV behaviour.
DFDL automatically supports that, if you create your model using the CSV wizard. You have stated that minOccurs=0 for all of your fields, so that sounds correct.
Next we need to understand exactly what your problem is. So you need to
a) explain with examples what your data looks like. To be fair you have already done that, but we need more examples.
b) Quote the error message(s) that IIB is giving you. We cannot see your screen!
Finally, please explain very carefully what you mean by this statement:
| Quote: |
we are not sure which field will be coming and which will be missing.
That is the worst part of it. |
Do you mean that any of the 37 fields can be zero-length ( represented as two commas one after the other)?
That should not be a problem. Empty fields will not be put into the message tree, but that's not a problem. Or is it? |
|
| Back to top |
|
|
| timber |
| Posted: Tue Mar 22, 2016 2:34 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1280
|
I do not agree with this statement:
| Quote: |
| If no value is present then they should be sending atleast Space in the message separated by comma. |
There is no good reason to force the sending application to do this. You can design your DFDL model to cope with the data in whatever form it comes in to IIB. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Mar 22, 2016 4:44 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
| timber wrote: |
Finally, please explain very carefully what you mean by this statement:
| Quote: |
we are not sure which field will be coming and which will be missing.
That is the worst part of it. |
Do you mean that any of the 37 fields can be zero-length ( represented as two commas one after the other)?
That should not be a problem. Empty fields will not be put into the message tree, but that's not a problem. Or is it? |
Er... I don't believe this is what the OP is saying. He is talking about optional fields, not nillable ones. If you have the required number of delimiters and nothing else, this would mean the fields are present but nill. What I am hearing from the OP is that you have a random number of delimiters and you never know which of the fields is going to be present...
This means, if you don't have a field initiator that allows you to distinguish between fields, or some other distinctive rule (like each field has a unique length, etc, etc...), the model cannot be parsed by a machine...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| maurito |
| Posted: Tue Mar 22, 2016 4:56 am Post subject: |
|
|
Partisan
Joined: 17 Apr 2014
Posts: 358
|
| fjb_saper wrote: |
| timber wrote: |
Finally, please explain very carefully what you mean by this statement:
| Quote: |
we are not sure which field will be coming and which will be missing.
That is the worst part of it. |
Do you mean that any of the 37 fields can be zero-length ( represented as two commas one after the other)?
That should not be a problem. Empty fields will not be put into the message tree, but that's not a problem. Or is it? |
Er... I don't believe this is what the OP is saying. He is talking about optional fields, not nillable ones. If you have the required number of delimiters and nothing else, this would mean the fields are present but nill. What I am hearing from the OP is that you have a random number of delimiters and you never know which of the fields is going to be present...
This means, if you don't have a field initiator that allows you to distinguish between fields, or some other distinctive rule (like each field has a unique length, etc, etc...), the model cannot be parsed by a machine... |
The OP might be saying that, but I doubt that that is what he means. That is not how CSVs work unless they have initiators for each field, which according to the examples given, they do not.
But it is difficult to know what the problem is when first the OP says the last field is mandatory, then says all fields have minOccurs =0 , maxOccurs = 1.
so, is it Mandatory or not ?. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Mar 22, 2016 5:11 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
It's a shame there isn't an example record of the OP's data posted.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| maurito |
| Posted: Tue Mar 22, 2016 5:19 am Post subject: |
|
|
Partisan
Joined: 17 Apr 2014
Posts: 358
|
| mqjeff wrote: |
| It's a shame there isn't an example record of the OP's data posted. |
Yes, there are. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Mar 22, 2016 6:36 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| maurito wrote: |
| mqjeff wrote: |
| It's a shame there isn't an example record of the OP's data posted. |
Yes, there are. |
You mean those strings in the first post, that have lots of ",," and etc.?
What's this about missing vs. nil fields?
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| paintpot |
| Posted: Tue Mar 22, 2016 6:39 am Post subject: |
|
|
Centurion
Joined: 19 Sep 2005
Posts: 112
Location: UK
|
the optional field appears to be around 10 fields from the end of the record:
| Code: |
| 000000000.00,,,000000000.00,,.00,0.00,12.39,000000000.00,218.85,20150619, |
vs.
| Code: |
| 000000000.00,,,,000000000.00,,.00,0.00,12.39,000000000.00,218.85,20150619, |
It's the 3 or 4 empty tags between the large zero fields... |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Mar 22, 2016 6:58 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Which isn't how CSV works.
If the comma isn't there, then which field is missing? The 2nd, the 3rd, or the 4th?
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| paintpot |
| Posted: Tue Mar 22, 2016 7:04 am Post subject: |
|
|
Centurion
Joined: 19 Sep 2005
Posts: 112
Location: UK
|
| the comma is there - there are 3 in the first record (start at the start of the subset I copied), and 4 in the second version |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Mar 22, 2016 7:08 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| paintpot wrote: |
| the comma is there - there are 3 in the first record (start at the start of the subset I copied), and 4 in the second version |
So the three commas indicate there are two fields that aren't there - the 2nd and 3rd fields.
The four commas indicate that there are three fields that aren't there- the 2nd, 3rd and 4th fields.
If there are always supposed to be three fields in that space, and there are only three commas, which field is left out entirely? The field that's supposed to be the 2nd field? The field that's supposed to be the 3rd field? Or the field that's supposed to be the 4th field?
How can you tell? How do you know that it's always the same field that is left out entirely?
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| paintpot |
| Posted: Tue Mar 22, 2016 7:31 am Post subject: |
|
|
Centurion
Joined: 19 Sep 2005
Posts: 112
Location: UK
|
| mqjeff wrote: |
| paintpot wrote: |
| the comma is there - there are 3 in the first record (start at the start of the subset I copied), and 4 in the second version |
How can you tell? How do you know that it's always the same field that is left out entirely? |
One for the OP - just pointing out that he has supplied two different sample files 
I would presume it is consistent, or he is doomed, in which case he just needs an optional field - his starting point was 36 or 37 fields per row. |
|
| Back to top |
|
|
| paintpot |
| Posted: Tue Mar 22, 2016 7:37 am Post subject: |
|
|
Centurion
Joined: 19 Sep 2005
Posts: 112
Location: UK
|
| Poli091987 wrote: |
As i said The Last Field may come (or) not and please find the difference below :
36 Fields
PI.HEADER:2000197067,20150619,3,NAZ,DNY,,20150619,,2000197067,218.85,000000000.00,218.85,000000000.00,000000000.00,000000000.00,0.00,,000000000.00,0.00,000000000.00,218.85,,,,000000000.00,,,000000000.00,,.00,0.00,12.39,000000000.00,218.85,20150619,
37 Fields
37 Fields
PI.HEADER:2000197067,20150619,3,NAZ,DNY,,20150619,,2000197067,218.85,000000000.00,218.85,000000000.00,000000000.00,000000000.00,0.00,,000000000.00,0.00,000000000.00,218.85,,,,000000000.00,,,,000000000.00,,.00,0.00,12.39,000000000.00,218.85,20150619 |
this version includes the change difference I spotted before, plus an extra comma at the end - so clear as mud... so the above is two fields different from each other (maybe a handcrafted issue?)  |
|
| Back to top |
|
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
