| Author |
Message
|
| ghoshly |
 Posted: Mon Nov 16, 2015 2:49 pm Post subject: Lets try to remove mis understanding |
 |
|
Partisan
Joined: 10 Jan 2008
Posts: 325
|
Hello All,
I just want to make sure there is no mis communication. let me try to depict once again.
Source System -> (JSON) -> ESB (Receive) convert to XML & Validate against XSD -> Transform (XSLT) -> Call Target System's SOAP web service.
XSD in ESB layer against which I am trying to validate the JSON message does not has any namespace. I couldn't validate with data type decimal here.
I'll try changing ESB xsd to FLOAT here and see the outcome.
Even if I do not validate the incoming message and transform and send to target system, it sends SOAP fault as mentioned earlier. |
|
| Back to top |
|
 |
| ghoshly |
| Posted: Mon Nov 16, 2015 3:07 pm Post subject: |
|
|
Partisan
Joined: 10 Jan 2008
Posts: 325
|
I changed the field details in the mxsd as xsd:float but it still fails in validation.
Input Data field -
"TotalCost": 350.750,
Error Text - cvc-datatype-valid.1.2: The value "3.5075E+2" is not a valid value for the "#Anonymous" datatype. |
|
| Back to top |
|
|
| timber |
| Posted: Tue Nov 17, 2015 1:24 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1280
|
I think this is what is happening. Pleaes correct me if I am wrong.
- The JSON parser is seeing something that looks like a number. It unconditionally creates a FLOAT ( and not a DECIMAL ) in the message tree.
- In your Compute node, you copy the JSON message tree to OutputRoot.XMLNSC. The numbers are still FLOATs, not DECIMALs.
- The XMLNSC parser writes the message tree to an XML document. The numbers are rendered as FLOATs according to the xs:float lexical representation: http://www.w3.org/TR/2012/REC-xmlschema11-2-20120405/datatypes.html#float
- But the lexical representation of an xs:float is never a valid representation of an xs:decimal: http://www.w3.org/TR/2012/REC-xmlschema11-2-20120405/datatypes.html#decimal so this triggers a validation error.
There may be a secondary problem, because you still get an error when you change the schema type to xs:float. Without seeing the xsd it is hard to comment on that. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Nov 17, 2015 5:24 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Are you positive the XSD has no namespace?
Can you post the top levels of the xsd? (up to the root element).
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| timber |
| Posted: Tue Nov 17, 2015 6:32 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1280
|
| @mqjeff : I agree with you/Vitor that a WSDL without a namespace is not best practice. But I think the error in the SOAP fault would be different if the namespace was missing/incorrect. I believe the SOAP parser is locating the correct root element in the schemas, and is starting to validate, but is then being tripped up by the float/decimal mismatch. |
|
| Back to top |
|
|
| ghoshly |
| Posted: Tue Nov 17, 2015 1:59 pm Post subject: You are so bang on Kimbert |
|
|
Partisan
Joined: 10 Jan 2008
Posts: 325
|
Hello Kimbert, You are always very very correct.
I have removed the ESB XSD and recreated the XSD with small number of fields as those numeric fields as float. It validates as float now in ESB and with the representation, when it tries to reach the end system (after necessary transformation) it denies the validation as it finds not a decimal format.
Do you think there would be any straight forward solution to this problem?
Please help.
I am thinking to try to multiply by 10 or 100 or 1000 based on the value after + sign. Though this does not seem to be a good solution. |
|
| Back to top |
|
|
| ghoshly |
| Posted: Tue Nov 17, 2015 2:16 pm Post subject: Handle NaN |
|
|
Partisan
Joined: 10 Jan 2008
Posts: 325
|
Why JSON parser is assigning FLOAT type to the values? Based on the lower range of values?
I think we should be able to cast those FLOAT fields to DECIMAL and see how the outcome is. I would rather love to do that within a XSLT as per our current flow design, otherwise we need to add custom ESQL after regular transformation is done by XSLT.
I could see some outcome as NaN (notANumber) because there are some arithmatic operation as well in the transformation logic. Would you please suggest the best way to handle NaN if the target side is expecting decimal.
Thanks in advance.... |
|
| Back to top |
|
|
| timber |
| Posted: Wed Nov 18, 2015 12:22 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1280
|
| Quote: |
| Why JSON parser is assigning FLOAT type to the values? |
Not sure - I * think* this is an unconditional feature of the JSON parser.
If stoney is watching this thread, he may be able to suggest a way to switch off that behaviour ( the message tree would contain CHARACTER data, which would probably be OK because the JSON text representation of the number is a valid xs:decimal ).
| Quote: |
| I would rather love to do that within a XSLT as per our current flow design, otherwise we need to add custom ESQL after regular transformation is done by XSLT. |
It could work. The JSON message tree will be serialized into an XML document. Your XSLT would need to detect every xs:float and change it to the equivalent xs:decimal lexical representation.
| Quote: |
| I could see some outcome as NaN (notANumber) because there are some arithmatic operation as well in the transformation logic. Would you please suggest the best way to handle NaN if the target side is expecting decimal. |
I see an opportunity! If you can convince the receiver that they really need to model this number as xs:float....problem solved! |
|
| Back to top |
|
|
| stoney |
| Posted: Wed Nov 18, 2015 4:35 am Post subject: |
|
|
Centurion
Joined: 03 Apr 2013
Posts: 140
|
| timber wrote: |
| Quote: |
| Why JSON parser is assigning FLOAT type to the values? |
Not sure - I * think* this is an unconditional feature of the JSON parser. |
The JSON parser will parse integers into INTEGER tree values, and it will parse decimal/float values into FLOAT tree values.
If the number being parsed doesn't fit into those types - for example, a number bigger than the limit of a 64-bit signed integer - only then does the JSON parser parse the number into a DECIMAL tree value.
| Quote: |
| If stoney is watching this thread, he may be able to suggest a way to switch off that behaviour ( the message tree would contain CHARACTER data, which would probably be OK because the JSON text representation of the number is a valid xs:decimal ). |
There's no way to turn on parse numbers as strings.
However, you can force the JSON parser to parse all precision values as DECIMAL tree values.
| Code: |
| MQSI_JSON_NUMBER_PRECISION_TYPE=DECIMAL |
|
|
| Back to top |
|
|
| timber |
| Posted: Wed Nov 18, 2015 4:58 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1280
|
I assume that MQSI_JSON_NUMBER_PRECISION_TYPE is an environment variable, and will affect all JSON parsers in all execution groups?
Decimal numbers are at least as common as floats in business software ( because floats are not suitable for storing monetary values ). I think this merits a proper parser option, available on the input node.
I *would* suggest that support for Swagger would fix this. But Swagger, inexplicably, does not support any kind of decimal data type. Which reduces its usefuless for modelling APIs in financial institutions. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Nov 18, 2015 5:52 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I mean, you could also just do a CAST on the value...
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| ghoshly |
| Posted: Wed Nov 18, 2015 8:48 am Post subject: Tried CAST, but |
|
|
Partisan
Joined: 10 Jan 2008
Posts: 325
|
We tried to CAST to DECIMAL after the necessary transformation in XSLT, it resolves the issue temporarily.
We would try with environment variable later and update the behavior accordingly. |
|
| Back to top |
|
|
| stoney |
| Posted: Wed Nov 18, 2015 9:10 am Post subject: |
|
|
Centurion
Joined: 03 Apr 2013
Posts: 140
|
| timber wrote: |
| I assume that MQSI_JSON_NUMBER_PRECISION_TYPE is an environment variable, and will affect all JSON parsers in all execution groups? |
Correct, unless you use per-execution group profiles to add the environment variable.
| timber wrote: |
| Decimal numbers are at least as common as floats in business software ( because floats are not suitable for storing monetary values ). I think this merits a proper parser option, available on the input node. |
Agreed. Feel free to raise an RFE 
| Quote: |
| I *would* suggest that support for Swagger would fix this. But Swagger, inexplicably, does not support any kind of decimal data type. Which reduces its usefuless for modelling APIs in financial institutions. |
I wonder if that's due to the lack of support in JavaScript. All numbers in JavaScript are double precision floating point numbers, and there's no built-in support for decimal numbers. |
|
| Back to top |
|
|
| ghoshly |
| Posted: Fri Nov 20, 2015 4:40 pm Post subject: Difference between Windows and AIX |
|
|
Partisan
Joined: 10 Jan 2008
Posts: 325
|
Hello Stoney,
Your suggested environment variable is working in Windows machine but not in AIX. Can you suggest if anything more specific would be required in AIX? In AIX, we are still getting as FLOAT
| Code: |
(0x03000000:NameValue):Quantity = 1.0575E+2 (FLOAT)
|
Thanks in advance. |
|
| Back to top |
|
|
| ghoshly |
| Posted: Sat Nov 21, 2015 10:20 am Post subject: |
|
|
Partisan
Joined: 10 Jan 2008
Posts: 325
|
We even tried after restarting the broker. Broker user profile is also show showing the MQSI variable, but the JSON parser is not working as expected in AIX environment.
Do we need to restart the server? Please help how we can debug the reason of not working. |
|
| Back to top |
|
|
|
|
