| Author |
Message
|
| martinrydman |
 Posted: Fri Jul 02, 2004 7:37 am Post subject: Modelling undeterminde fixed-length records structure |
 |
|
Centurion
Joined: 30 Jan 2004
Posts: 139
Location: Gothenburg, Sweden
|
Hi all,
I've tried in vain to model a structure made up of an variable number of fixed-lenght records, either in CWF or TDS.
I've come to the conclusion that the CWF parser doesn't allow unknown number of repetitions at all, and that the TDS parser gets confused if there's no delimiters between records. My solution thus far is to model the message with delimiters, and then remove them from the resulting bitstream. Not a very elegant solution, but it works 
I've noted that I can read such a structure, but not write it out to a queue. Darned strange, if you ask me 
Can any of you guys confirm/correct this conclusion?
TIA!
/Martin |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Fri Jul 02, 2004 7:46 am Post subject: Re: Modelling undeterminde fixed-length records structure |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| martinrydman wrote: |
| I've come to the conclusion that the CWF parser doesn't allow unknown number of repetitions at all, |
This is very very very true.
| martinrydman wrote: |
| and that the TDS parser gets confused if there's no delimiters between records. |
I don't think that's as true. You should be able to do strict fixed-width modelling for an unknown number of repeats with TDS - PROVIDED your messages are strictly homogenous (one record type only) or your records are distinguishable by content (one starts with an integer, the other starts with a string).
When you get into situations where you have a header record, followed by an unknown number of repeated body records, followed by a trailer... then you start needing tags or delimiters.
But regardless, TDS is a more complicated modelling paradigm than CWF, and has a higher learning curve.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| djeripo |
| Posted: Fri Jul 02, 2004 7:59 am Post subject: |
|
|
Master
Joined: 25 Jan 2004
Posts: 225
|
So,You are trying to model a variable number of repeating FixedLength Records .
How did you try in TDS to model a repeating element ?
How does you message look?
Do you get a one continuous stream of message without CR&LF ? |
|
| Back to top |
|
|
| martinrydman |
| Posted: Fri Jul 02, 2004 8:26 am Post subject: |
|
|
Centurion
Joined: 30 Jan 2004
Posts: 139
Location: Gothenburg, Sweden
|
thanks for quick replies!
I model it like this (bottom-up):
Compound type t_IPT:
-sequence, closed
-fixed length
-contains a number of fixed length strings and decimals
Componud type t_Msg:
-sequence, closed
-fixed length
-contains 1 to 2000000000 repetitions of IPT (element of type t_IPT)
Note that I'm not allowed to set maxOccurs to -1!
Message is of type t_Msg.
The message itself is a stream of bytes, in multiples of the compound length of the elements of t_IPT.
Note again that I can parse this in the MQInput node, it is when I construct an identical structure and try to write it with MQOutput that my troubles start, the error being that the parser puts the DataFlow engine in what appears to be an infinite loop.
I had this problem back in the good old 2.1 days, and after PMR:ing it to IBM, they vaguely confirmed this to be a problem. I had sort of hoped that it had gone away in 5.0, and indeed yours suggestions would indicate this.
Any further input greatly appreciated!
/Martin |
|
| Back to top |
|
|
| djeripo |
| Posted: Mon Jul 05, 2004 1:21 am Post subject: |
|
|
Master
Joined: 25 Jan 2004
Posts: 225
|
Martin,
I did a test on my computer . Even I too ran into the same problem which you mentioned . May be we cannot model variable number of fixed length messages with DES as Fixed Length.
I tried transforming TDS to CWF . TDS parser was successfully parsing but was not able to write on to the OutputQueue .I am running on WMQI2.1 & CSD06 . May be experts on this forum have to comment on this strange behaviour.
I was doing TDS to CWF transformation for a quite a some time .
For me every message comes in a new line something like,
PLCYNE12345
PLCYCO80139
........
and the number of messages are variable .
I set ,
DES as 'All Elements Delimited',
Delimiter <CR><LF>
Repeating Element Delimiter <CR><LF>
I know this would be hardly of any help for you .
Thanks |
|
| Back to top |
|
|
| martinrydman |
| Posted: Mon Jul 05, 2004 2:31 am Post subject: |
|
|
Centurion
Joined: 30 Jan 2004
Posts: 139
Location: Gothenburg, Sweden
|
Hi,
thanks for your input, and good to know others experience the same problem. As I mentioned, I usually solve this by inserting CRLF after each record, parse it out as a bitstream (using BITSREAM) and the use REPLACE to get rid of CRLF. Used to do the same in 2,1, but REPLACE is not available, so a loop using POSITION and OVERLAY used to do the trick.
/Martin |
|
| Back to top |
|
|
| wooda |
| Posted: Mon Jul 05, 2004 4:50 am Post subject: |
|
|
Master
Joined: 21 Nov 2003
Posts: 265
Location: UK
|
Martin -
I believe that writing out a variable number of repeats inside TDS fixed length is not currently supported. The TDS writer expects a fixed number of repeats.
However I think I have a solution which will work for you.
If you make your top level compound type (t_Msg) tagged fixed length and not fixed length. Set the tag length to 0.
Add a new group inside t_Msg. Make it fixed length and set the min/max occurs to 1/-1 respectively.
Move the element IPT inside the new group
Change the min/max occurs of the IPT element to 1/1 respectively.
The ESQL required for this model should be the same as if the element itself was repeating.
- Alex |
|
| Back to top |
|
|
| shanson |
| Posted: Mon Jul 05, 2004 4:53 am Post subject: |
|
|
Partisan
Joined: 17 Oct 2003
Posts: 344
Location: IBM Hursley
|
For CWF, an undetermined number of repeats is theoretically possible if the repeating item is the last thing in the message.
For TDS Fixed Length, an undetermined number of repeats is theoretically possible if the repeating item is either the last thing in the message or there is an identifiable end point such as a Group Terminator.
I say 'theoretically' because V5 does not support this today. But we are aware of the requirement and are treating it as a high priority  |
|
| Back to top |
|
|
| djeripo |
| Posted: Mon Jul 05, 2004 9:33 am Post subject: |
|
|
Master
Joined: 25 Jan 2004
Posts: 225
|
Alex,
| Quote: |
| If you make your top level compound type (t_Msg) tagged fixed length and not fixed length. Set the tag length to 0. |
I tried setting tag length to '0' but its not allowing me .
| Quote: |
| Add a new group inside t_Msg. Make it fixed length and set the min/max occurs to 1/-1 respectively |
What actually did you mean adding a new group ? An element of new type ? Could you please elaborate .
Thanks |
|
| Back to top |
|
|
| wooda |
| Posted: Tue Jul 06, 2004 1:07 am Post subject: |
|
|
Master
Joined: 21 Nov 2003
Posts: 265
Location: UK
|
djeripo wrote
| Quote: |
| I tried setting tag length to '0' but its not allowing me . |
I am able to set the tag length to 0 the version I am using (WBIMB fixpack3) However if you can't don't worry because it isn't actually required.
djeripo also wrote;
| Quote: |
| What actually did you mean adding a new group ? An element of new type ? Could you please elaborate . |
The original model looked like;
| Code: |
Msg (t_Msg) (type: sequence, Fixed Length)
-> IPT (t_IPT) (element: max/min occurs 1/2000000000)
(type: sequence, Fixed Length) |
The new model looks like;
| Code: |
Msg (t_Msg) (type: sequence, Tagged Fixed Length)
-> newGroup (group: max/min occurs 1/-1, sequence, Fixed Length)
->IPT (t_IPT) (element: max/min occurs 1/1)
(type: sequence, Fixed Length) |
So now we have inserted newGroup which repeats an unknown number of times which contains the original element which now has max occurs=1.
The top level is no longer Fixed Length so the group can have max occurs=-1 and is able to handle variable number of repeats of newGroup on output. |
|
| Back to top |
|
|
|
|
