| Author |
Message
|
| Vitor |
 Posted: Wed Feb 16, 2011 1:57 pm Post subject: XML datatype matching |
 |
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
WMB Version: 6.1.0.8
OS: Windows
Pain Level: Quite high
Please bear with me as I describe my issue:
The input to my flows is an XML document produced from 3rd party software. I'm trying to improve the quality by validating against the schema. After some effort, I've been given an XSD which is claimed to be the one this XML comes from. XMLSpy doesn't think so for some of the fields:
| Code: |
Value '' is not allowed for element <NumberOfVehiclesExtraHeavyTrucks>.
Reason: it does not satisfy any of the defined patterns (see below)
'[\-+]?[0-9]+'
Details
cvc-pattern-valid: '' is not matched by the regular expressions of type definition 'xs:integer'.
cvc-datatype-valid.1.1: For type definition 'int' the string '' is not pattern valid.
cvc-simple-type.1: For type definition 'int' the string '' is not valid.
cvc-complex-type.2.2: Content of element <NumberOfVehiclesExtraHeavyTrucks> must be valid with respect to its type definition 'columnInteger'.
cvc-elt.5.2.1: The element <NumberOfVehiclesExtraHeavyTrucks> is not valid with respect to the actual type definition 'columnInteger'. |
The element is defined thus:
| Code: |
| <xsd:element name="NumberOfVehiclesExtraHeavyTrucks" type="columnInteger" minOccurs="0"/> |
and the type thus:
| Code: |
<xsd:complexType name="columnInteger">
<xsd:simpleContent>
<xsd:extension base="int">
<xsd:attributeGroup ref="attgrpColumn"/>
</xsd:extension>
</xsd:simpleContent>
</xsd:complexType> |
What we get in the XML is this:
| Code: |
| <NumberOfVehiclesExtraHeavyTrucks/> |
Now it's possible I've finally cracked, but that tag is present but empty isn't it? So if it was entirely absent it would be fine because of the minOccurs, but as it's present it can't be empty but must be an integer? Yes? No? Does that tag actually match it's schema definition and I need to update to a newer version of XMLSpy?
(It's been a long day)
Moving on and accepting that I've got more chance of being elected King Of England (or introducing automated builds) than I have of getting the XML changed is there anything I can do inside a message set to convince WMB that this XML matches the schema in the face of the evidence? Both the schema and message set are under my control, but I'd like to keep it mostly to the vendor supplied one to reduce maintenance when a new schema is released.
(Not that they'll release a new schema soon just to spite me. They wouldn't be that petty. No, really.)
Any help, advice or spirtual comfort welcomed.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Wed Feb 16, 2011 2:06 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Quote: |
| Code: |
| <NumberOfVehiclesExtraHeavyTrucks/> |
Now it's possible I've finally cracked, but that tag is present but empty isn't it? So if it was entirely absent it would be fine because of the minOccurs, but as it's present it can't be empty but must be an integer? Yes? No? Does that tag actually match it's schema definition and I need to update to a newer version of XMLSpy? |
Not quite. What you have is a null value. However by the definition of your element it is not nillable.
| Code: |
| <xsd:element name="NumberOfVehiclesExtraHeavyTrucks" type="columnInteger" minOccurs="0"/> |
This is missing the nillable="true" attribute to allow it to have the null value you are reporting. Also and depending on your XML setup a null value may need to be expressed differently, more like:
| Code: |
| <NumberOfVehiclesExtraHeavyTrucks xsi:nil="true"/> |
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Feb 16, 2011 3:47 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| fjb_saper wrote: |
| Have fun |
I always do 
And thank you so much for the clarification. That these people are sending a null value in the XML where the schema says the tag isn't nillable.
Wonderful.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Feb 16, 2011 5:06 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| What was that again, about the job of broker and accepting bad data? |
|
| Back to top |
|
|
| mgk |
| Posted: Thu Feb 17, 2011 1:49 am Post subject: |
|
|
Padawan
Joined: 31 Jul 2003
Posts: 1642
|
| Quote: |
| What was that again, about the job of broker and accepting bad data? |
It should accept this, unless you try to validate it 
_________________
MGK
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Feb 17, 2011 5:04 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mgk wrote: |
| Quote: |
| What was that again, about the job of broker and accepting bad data? |
It should accept this, unless you try to validate it |
It does, it does. I'm caught in a trap of my own making.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Feb 17, 2011 5:12 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| What was that again, about the job of broker and accepting bad data? |
There's no question of accepting this, and we have rafts of ESQL to detect and fix problems.
My intention here is to improve things by demonstrating how dire the XML is. I've been whining for a while that the XML being provided is rubbish (my current favourite is a percentage field that came through as a string '15%' rather than the decimal we were sort of expecting) and am looking for some statistics. By validiating against the schema I'd hoped to produce some figures like:
15% of XML had tag names not in the schema
25% of XML had string values in integer or decimals
15% of XML had mandatory tags (which the schema shows with a minOccurs 1) missing
and so forth.
There's no question of not accepting the XML and doing the best we can with it. There's almost no question of getting the XML corrected but with some figures to demonstrate why we need so much code I might get longer development windows or (I dare to dream) more staff.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| kimbert |
| Posted: Thu Feb 17, 2011 5:23 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
Here are some options:
a) Ask for the 3rd party app to be fixed ( unlikely )
b) Edit the schema so that it it validates less strictly. For this particular error, that would mean changing the data type to xs:string, I think.
c) Switch off validation
b) leaves open the possibility that you will have to edit the schema multiple times, never being 100% sure whether another problem with the 3rd party app will surface. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Feb 17, 2011 5:50 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| kimbert wrote: |
| a) Ask for the 3rd party app to be fixed ( unlikely ) |
Well nobody connected with the Palace has called to advice me that the monachy is now an elected position and asking for my candidate papers so any fix remains unlikely
| kimbert wrote: |
| b) Edit the schema so that it it validates less strictly. For this particular error, that would mean changing the data type to xs:string, I think. |
This would remove a lot of the benefit I was looking to obtain.
| kimbert wrote: |
| c) Switch off validation |
Or in my specific instance not bother switching it on. Just sit back and deal with this rubbish.
| kimbert wrote: |
| b) leaves open the possibility that you will have to edit the schema multiple times, never being 100% sure whether another problem with the 3rd party app will surface. |
Well quite. As it stands we code into the ESQL for all possibilities of data huff up (though the decimal as a string was a surprise) and I suspect this is where matters will remain.
I thank all those who took the time.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| joebuckeye |
| Posted: Thu Feb 17, 2011 6:06 am Post subject: |
|
|
Partisan
Joined: 24 Aug 2007
Posts: 365
Location: Columbus, OH
|
| Quote: |
| I'm trying to improve the quality by validating against the schema. After some effort, I've been given an XSD which is claimed to be the one this XML comes from. |
I'm shocked to learn that someone would send another group an XSD that was supposedly used to build an XML that doesn't validate said XML.
Can you ask the 3rd party vendor if they validate their own XML with the schema? And if they claim to could you ask them to provide that schema instead of the one you were previously given?
Schema validation is nice in theory but I've seen too many instances where it is turned on (in broker or datapower) and then messages are failed for not being validated according to the schema given to us. It is sometimes a wake up call to the message provider's management that their developers are not using their own schemas for validation purposes.
We've basically found that many groups start their dev work based on a schema and then during the testing phase changes are made to the code to get it to work and the schema is never updated to reflect these changes. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Feb 17, 2011 6:17 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| joebuckeye wrote: |
| Can you ask the 3rd party vendor if they validate their own XML with the schema? And if they claim to could you ask them to provide that schema instead of the one you were previously given? |
Apparently this is the schema they use. The fact that it doesn't match the XML explains the problems I had in obtaining it.
| joebuckeye wrote: |
| It is sometimes a wake up call to the message provider's management that their developers are not using their own schemas for validation purposes. |
I feel the vendor's management are wide awake & on their 3rd cup of coffee. I was actually surprised when an XSD turned up; there's been a lot of stonewalling from the vendor management and now I think I know why.
| joebuckeye wrote: |
| We've basically found that many groups start their dev work based on a schema and then during the testing phase changes are made to the code to get it to work and the schema is never updated to reflect these changes. |
Something much like this has probably happened here. But since the vendor clearly doesn't use it internally it's not going to get changed any more than the XML's going to be fixed.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Feb 17, 2011 6:50 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Vitor wrote: |
Well quite. As it stands we code into the ESQL for all possibilities of data huff up (though the decimal as a string was a surprise) and I suspect this is where matters will remain.
I thank all those who took the time. |
Can't imagine why the decimal as string should come as a surprise. XML represents all numeric values as string. As such if the schema is violated (wrong namespace or lack thereof or wrong tag name...) the field will be self-defined an the content a string. It will be up to you to parse it correctly.
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Feb 17, 2011 6:56 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| fjb_saper wrote: |
| Can't imagine why the decimal as string should come as a surprise. |
Because for the last few months it was a string with "0.15" in it & in many business cases it still is.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Feb 17, 2011 7:03 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Vitor wrote: |
| fjb_saper wrote: |
| Can't imagine why the decimal as string should come as a surprise. |
Because for the last few months it was a string with "0.15" in it & in many business cases it still is. |
Sorry now I understand you meant "0.15" vs "15%"
Well I imagine the difference comes from using a different format for rendering the value to the XML field...  developers
Tell the data provider to use the XMLBeans package instead of writing their own values into fields. This should enforce some of the formats for you, and put the burden where it should have been from the start.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Feb 17, 2011 7:11 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| fjb_saper wrote: |
| Tell the data provider to use the XMLBeans package instead of writing their own values into fields. |
If I could tell the providers and their developers something that they would listen to and do, it's more likely to involve a road trip with Thelma & Louise style ending. Ideally in a vehicle full of gas cans with a chain smoker. Previous complaints about the data have not been met with the phrase, "well it sucks to be you" but that's the general principle.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
