| Author |
Message
|
| llaros |
 Posted: Tue Jan 29, 2008 4:09 am Post subject: XML inheritance using xsi:type with TDS parser |
 |
|
Apprentice
Joined: 22 Jan 2008
Posts: 37
|
Hello
I use MRM domain, broker 6.0.0.3 on AIX 5.3.
I am trying to change message format from XML to TDS (AllElementsDelimited) due to performance problems.
Now, messages are created using subtypes inheritance where xsi:type attribute is specified.
Example message:
<getCustomerVehicle>
<person>
<firsName>Jim</firsName>
<lastName>Smith</lastName>
</person>
<vehicle xsi:type="car">
<productionYear>2006</productionYear>
<maxSpeed>200</maxSpeed>
</vehicle>
</getCustomerVehicle>
There isn't anything described in doc that it is possible or not to use subtypes in TDS so I assume it is  . My idea to do that was to use distinct delimiter per subtype. For example:
vehicle - complex type group indicator: '{T0:' - this type has no child elements.
car - complex type group indicator: '{T1:' - derived by extension, base type vehicle
lorry - complex type group indicator: '{T2:' - derived by extension, base type vehicle
submarine - complex type group indicator: '{T3:' - derived by extension, base type vehicle
Other delimiters:
delimiter ';'
group iterminator '}'
When I ran test on a flow which should have changed the message from XML1 to TDS1 I've got a result like:
{{Jim;Smith}{T0:}}
but I wanted
{{Jim;Smith}{T1:2006;200}}
I think that there is a problem that a broker cast the T1 on T0 when writing the output stream. But it isn't correct! The real type which was carried by this message is T1.
Do You think I should raise a PMR?
Maybe There is another way to use subtypes in TDS? (But I'd like it to be a dynamic - I don't want to create new messages definitions ex. getCustomerCar, getCustomerSubmarine ect.)
P.S. It is not possible to use xsd:choice because many services have been already implemented.
Any help would be appreciated. |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Tue Jan 29, 2008 4:26 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Okay, first of all... TDS is likely going to be slower than the XMLNSC parser.
Second of all... you do need to use choice for this ...
Thirdly, the output message that you get is determined solely by how you build it - so if you're getting the "wrong" output then you're doing the "wrong" thing in your ESQL/JavaCompute node...
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| llaros |
| Posted: Tue Jan 29, 2008 5:42 am Post subject: |
|
|
Apprentice
Joined: 22 Jan 2008
Posts: 37
|
Ok.
1. I didn't mean the speed problems but the problems with size of messages. The TDS message is much more packed than XML. I hope we agree on that matter.
2. Why should I use choice if there is a possibility to use inheritance (XSD W3C) what is working with format XML (MRM not XMLNSC) domain.
3. My ESQL Code is performing transformation from XML1->TDS1:
| Code: |
CREATE COMPUTE MODULE transformTDS2REB
CREATE FUNCTION Main() RETURNS BOOLEAN
BEGIN
SET OutputRoot = InputRoot;
SET OutputRoot.Properties.MessageFormat = 'TDS1';
SET OutputRoot.MQRFH2.mcd.Fmt = 'TDS1';
PROPAGATE TO TERMINAL 'out';
RETURN FALSE;
END;
END MODULE;
|
Please tell me what is wrong with this code. I will be gratefull. |
|
| Back to top |
|
|
| kimbert |
| Posted: Tue Jan 29, 2008 5:58 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| Why should I use choice if there is a possibility to use inheritance |
It is not a good idea to use advanced XML Schema constructs with non-XML formats ( TDS / CWF ). WMB does make every effort to 'do what the user expects' but sometimes it is far from obvious what the right answer is. I'm not sure why your ( very innovative ) technique did not work, but I would prefer to see a simpler solution utilizing choice if that is possible.
Or are you requesting support for binary XML in WMB?  |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Jan 29, 2008 6:05 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| kimbert wrote: |
| Or are you requesting support for binary XML in WMB? |
I'm happy to request DFDL support in WMB... that would cover Binary XML, wouldn't it?
Although perhaps I should be careful what I ask for.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| llaros |
| Posted: Tue Jan 29, 2008 6:40 am Post subject: |
|
|
Apprentice
Joined: 22 Jan 2008
Posts: 37
|
I know that it would be much easier to use xsd:choice but it is not possible because of many other components which were already implemented.
The question is if there is a possibility to solve that problem:
If Broker parse (MRM domain XML format) message with xsi:type attribute it builds its logical message tree and after parsing broker must know type of every element in the tree.
This is from documentation wbimb6:
Type substitution allows the type of the element to be defined within the instance document, using the xsi:type attribute on the element, so that the element's real type is not known until the element has been partly parsed. This mechanism can also be blocked or allowed based on the derivation method of the types involved.
So after parsing broker should know that the type of element vehicle (from the message I've written above) is Car not Vehicle. So when creating a delimited stream it should use delimiters which were defined for the type Car not Vehicle! I do not demand from broker to do very complex work, do I? I do not want broker to write xsi:type in TDS just to follow the rules which were specified during configuring mset .
I think that the problem might be solved but there is some dissonance in broker. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Jan 29, 2008 7:01 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
So, to review: You created your message description based on an XML schema. You then went in and added the TDS physical layer, and set all the necessary properties on the same types in the message description.
And you have validated that all your input message data is read using MRM-XML, and has not parsed any data as "self-defining"?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| llaros |
| Posted: Tue Jan 29, 2008 8:08 am Post subject: |
|
|
Apprentice
Joined: 22 Jan 2008
Posts: 37
|
Yes Jeff.
I use MRM so I needed to define MRM tree. Up to now my flows and connected systems works on XML format but I'd like to change it to TDS. So As You said I added TDS physical layer and sent a message to the flow which was to translate XML format to TDS format. After reading XML stream there is a proper MRM tree (I see it during debug), but after writing the output stream broker uses the TDS properties of the ancesor element (Vehicle) not the subelement (Car). Moreover broker do not write to output stream the contents of the elemet (Car) because this content is not defined in (Vehicle), but this content (productionYear, maxSpeed) is in message tree during processing the message.[/img] |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Jan 29, 2008 8:11 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Turn on user trace (debug) and look for reports that the CAR element has been parsed as a self-defining element.
Your MRM-XML may not be matching your XML data, and so your Car element may not be of the MRM Car type.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| JLRowe |
| Posted: Tue Jan 29, 2008 9:16 am Post subject: |
|
|
Yatiri
Joined: 25 May 2002
Posts: 664
Location: South East London
|
| llaros wrote: |
Yes Jeff.
I use MRM so I needed to define MRM tree. Up to now my flows and connected systems works on XML format but I'd like to change it to TDS. So As You said I added TDS physical layer and sent a message to the flow which was to translate XML format to TDS format. After reading XML stream there is a proper MRM tree (I see it during debug), but after writing the output stream broker uses the TDS properties of the ancesor element (Vehicle) not the subelement (Car). Moreover broker do not write to output stream the contents of the elemet (Car) because this content is not defined in (Vehicle), but this content (productionYear, maxSpeed) is in message tree during processing the message.[/img] |
but you have no equivalent to xsi:type in TDS, so how can the parser work out which type should be expected?
this may work in an XML wire format, but it obviously doesn't in TDS.
I do know that TDS does work with choice, so either switch to using choice or forget about it. |
|
| Back to top |
|
|
| kimbert |
| Posted: Wed Jan 30, 2008 5:00 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
OK - time for some facts:
- llaros is quite correct in believing that this *can* work.
- I don't know (yet) why it does not work
- I believe it is best practice to limit TDS to simple schema constructs
- I understand that llaros has valid reasons for wanting this to work
- I will *try* to find some time to try this out, and see why it is not working. I cannot promise that it will be soon.
I still think EXI ( http://www.w3.org/TR/2007/WD-exi-primer-20071219/ ) offers a better solution if it ever becomes a W3C standard. Or there may be existing XML compression tools around which could be used. |
|
| Back to top |
|
|
| llaros |
| Posted: Wed Jan 30, 2008 6:37 am Post subject: |
|
|
Apprentice
Joined: 22 Jan 2008
Posts: 37
|
Thanks for your reply Kimbert! It is really hopeful.
I'm looking forward for your results.
Tell me if I should raise PMR or provide You any help on that matter. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Jan 30, 2008 6:42 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
First thing to understand is that any help you get here is voluntary. So don't hold your breath that kimbert or anyone else will come back with a solution in any given timeframe.
Second thing is to confirm that your Input Message is parsing out the way you think it is, properly.
Do this by enabling User Trace at Debug level, and reading the output.
If you feel like you should open a PMR on this, then go ahead. You're welcome to refer to this thread in the PMR, too. I'd be surprised if support doesn't ask for a debug trace of your flow, though... 
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| llaros |
| Posted: Thu Jan 31, 2008 7:42 am Post subject: |
|
|
Apprentice
Joined: 22 Jan 2008
Posts: 37
|
Jeff my MRM tree is correct. I see it in Debug perspective. Here is a trace (debug level)
xsi:type is not defined in the message set and that's why the parser signals WARNING.
| Code: |
2008-01-31 16:33:36.286643 3244 UserTrace BIP6060I: Parser type ''Properties'' created on behalf of node 'tranformations.reb2tds.REB.IN' to handle portion of incoming message of length 0 bytes beginning at offset '0'.
2008-01-31 16:33:36.286739 3244 UserTrace BIP6061I: Parser type ''MQMD'' created on behalf of node 'tranformations.reb2tds.REB.IN' to handle portion of incoming message of length '364' bytes beginning at offset '0'. Parser type selected based on value ''MQHMD'' from previous parser.
2008-01-31 16:33:36.287479 3244 UserTrace BIP6061I: Parser type ''MQRFH2'' created on behalf of node 'tranformations.reb2tds.REB.IN' to handle portion of incoming message of length '396' bytes beginning at offset '364'. Parser type selected based on value ''MQHRF2'' from previous parser.
2008-01-31 16:33:36.292732 3244 UserTrace BIP6061I: Parser type ''MRM'' created on behalf of node 'tranformations.reb2tds.REB.IN' to handle portion of incoming message of length '1551' bytes beginning at offset '760'. Parser type selected based on value ''MRM'' from previous parser.
2008-01-31 16:33:36.292781 3244 UserTrace BIP2632I: Message received and propagated to 'out' terminal of MQ input node 'tranformations.reb2tds.REB.IN'.
2008-01-31 16:34:02.045723 3244 UserTrace BIP2537I: Node 'tranformations.reb2tds.reb2tds': Executing statement ''BEGIN ... END;'' at ('tranformations.reb2tds_Compute.Main', '2.2').
2008-01-31 16:34:02.045833 3244 UserTrace BIP2537I: Node 'tranformations.reb2tds.reb2tds': Executing statement ''SET OutputRoot = InputRoot;'' at ('tranformations.reb2tds_Compute.Main', '4.3').
2008-01-31 16:34:02.046303 3244 UserTrace BIP2539I: Node 'tranformations.reb2tds.reb2tds': Evaluating expression ''InputRoot'' at ('tranformations.reb2tds_Compute.Main', '4.20'). This resolved to ''InputRoot''. The result was ''ROW... Root Element Type=16777216 NameSpace='' Name='Root' Value=NULL''.
2008-01-31 16:34:02.046442 3244 UserTrace BIP2568I: Node 'tranformations.reb2tds.reb2tds': Copying sub-tree from ''InputRoot'' to ''OutputRoot''.
2008-01-31 16:34:02.051075 3244 UserTrace BIP2537I: Node 'tranformations.reb2tds.reb2tds': Executing statement ''SET OutputRoot.Properties.MessageFormat = 'TDS1';'' at ('tranformations.reb2tds_Compute.Main', '5.3').
2008-01-31 16:34:02.051075 3244 UserTrace BIP2566I: Node 'tranformations.reb2tds.reb2tds': Assigning value '''TDS1''' to field / variable ''OutputRoot.Properties.MessageFormat''.
2008-01-31 16:34:02.051075 3244 UserTrace BIP2537I: Node 'tranformations.reb2tds.reb2tds': Executing statement ''SET OutputRoot.MQRFH2.mcd.Fmt = 'TDS1';'' at ('tranformations.reb2tds_Compute.Main', '6.3').
2008-01-31 16:34:02.051075 3244 UserTrace BIP2566I: Node 'tranformations.reb2tds.reb2tds': Assigning value '''TDS1''' to field / variable ''OutputRoot.MQRFH2.mcd.Fmt''.
2008-01-31 16:34:02.051987 3244 UserTrace BIP2537I: Node 'tranformations.reb2tds.reb2tds': Executing statement ''PROPAGATE TO TERMINAL 'out' FINALIZE DEFAULT DELETE DEFAULT;'' at ('tranformations.reb2tds_Compute.Main', '7.3').
2008-01-31 16:34:02.052009 3244 UserTrace BIP4008I: Message propagated to '''out''' terminal of node 'tranformations.reb2tds.reb2tds'.
2008-01-31 16:34:23.922140 3244 UserTrace BIP5605W: ***WARNING*** Open or open-defined elemenet ''type'' has occurred inside a type or group of Data Element separation ''ALL ELEMENTS DELIMITED''.
This Data Element Separation type cannot contain open or open-defined elements.
The data has been discarded, and the message will be written without it.
Confirm that the element matches that in the model, and that it is correctly described in the flow.
2008-01-31 16:34:23.922151 3244 UserTrace BIP5605W: ***WARNING*** Open or open-defined elemenet ''type'' has occurred inside a type or group of Data Element separation ''ALL ELEMENTS DELIMITED''.
This Data Element Separation type cannot contain open or open-defined elements.
The data has been discarded, and the message will be written without it.
Confirm that the element matches that in the model, and that it is correctly described in the flow.
2008-01-31 16:34:23.938766 3244 UserTrace BIP2638I: The MQ output node 'tranformations.reb2tds.TDS.OUT' attempted to write a message to queue ''TDS.OUT'' connected to queue manager ''''. The MQCC was '0' and the MQRC was '0'.
2008-01-31 16:34:23.938766 3244 UserTrace BIP2622I: Message successfully output by output node 'tranformations.reb2tds.TDS.OUT' to queue ''TDS.OUT'' on queue manager ''''.
2008-01-31 16:34:23.938768 3244 UserTrace BIP2537I: Node 'tranformations.reb2tds.reb2tds': Executing statement ''RETURN FALSE;'' at ('tranformations.reb2tds_Compute.Main', '9.3').
|
|
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Jan 31, 2008 7:50 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
The Debug perspective does not generally provide you enough information to determine that the logical message tree is exactly correct.
I'm confused about how you expect the model to use the xsi:type information when you haven't defined it in the model.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
|
|
