| Author |
Message
|
| rk1891 |
 Posted: Wed Jul 24, 2019 2:08 pm Post subject: JSON message validation |
 |
|
Apprentice
Joined: 12 Oct 2011
Posts: 42
|
Hello all,
I'm facing some strange situation with JSON message parsing.
I've a REST request Node that is invoking a REST API to get a JSON response. On this node I've selected the message domain as BLOB as I want to pass in the message as-is to downstream ESQL compute node in another flow.
In the compute node I'm converting the BLOB to JSON and accessing the JSON response using JSON.Data.'xxx'.Everything works as expected when I get a valid JSON message as the response.
However, I stumbled on a scenario where in the REST service I'm calling returned a message with an invalid JSON structure and my compute node failed to parse the message.
I see that it's failing as soon as it hits JSON.Data xpath.
Finally my question is:
Is there anyway to check if the response is a valid conforming JSON message so I could do that detailed transformation only when it is valid otherwise I can simply convert the incoming BLOB to CHAR and pass on the JSON parsing error message to downstream.
I tried different ways but none of them seem to work, the below are couple of tries -
| Code: |
FIELDNAME(InputRoot.JSON.Data) IS NOT NULL
|
| Quote: |
This one gets through as there is some data under JSON
| Code: |
EXISTS(InputRoot.JSON.[])
|
|
[/code]
Any help is appreciated! |
|
| Back to top |
|
 |
| timber |
| Posted: Thu Jul 25, 2019 1:56 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
| Quote: |
| On this node I've selected the message domain as BLOB as I want to pass in the message as-is to downstream ESQL compute node in another flow. |
I don't see any advantage in that approach. Is there a specific reason why you did not simply set the domain to JSON on the request node?
| Quote: |
| the REST service I'm calling returned a message with an invalid JSON structure and my compute node failed to parse the message. |
Important question: did the REST service return invalid JSON or valid JSON with unexpected/missing data. If the returned JSON is invalid then you will get a parser exception from the JSON parser. If the returned JSON is valid but does not conform to the API specs then you have a different problem.
Currently, IIB does not offer JSON validation. That's because the validation rules were not properly standardised when Swagger 2.0 was released. If you really want to validate against the Swagger, there are Java class libraries that will do the job for you. |
|
| Back to top |
|
|
| rk1891 |
| Posted: Thu Jul 25, 2019 6:25 am Post subject: |
|
|
Apprentice
Joined: 12 Oct 2011
Posts: 42
|
| timber wrote: |
| Quote: |
| On this node I've selected the message domain as BLOB as I want to pass in the message as-is to downstream ESQL compute node in another flow. |
I don't see any advantage in that approach. Is there a specific reason why you did not simply set the domain to JSON on the request node? |
This as per the current architectural pattern we adopted. We've the adapter flows that does all the communication part of work and pass the info to Transformation flows where the actual transformation occurs. Even if I change the domain as JSON, don't think it failed on the Request node ie the message is NOT sent to Failure terminal.The response is sent through Out terminal and the failure is triggered only when I touch the contents under that JSON message tree in the subsequent compute node.
| timber wrote: |
| Quote: |
| the REST service I'm calling returned a message with an invalid JSON structure and my compute node failed to parse the message. |
Important question: did the REST service return invalid JSON or valid JSON with unexpected/missing data. If the returned JSON is invalid then you will get a parser exception from the JSON parser. If the returned JSON is valid but does not conform to the API specs then you have a different problem.
Currently, IIB does not offer JSON validation. That's because the validation rules were not properly standardised when Swagger 2.0 was released. If you really want to validate against the Swagger, there are Java class libraries that will do the job for you.
|
I kind of see this question coming. It's an invalid JSON(or not well-formed in XML semantics). I do have a Swagger YAML for this service which I'm using on the Request node by referring it from a static library. I'm not planning to validate the response against the Swagger document or such as I've already learnt that it's not possible.
I could simply connect the failure terminal and catch this error on the Compute node where the transformation happens. But is there any error code/ static phrase that I could just simply look for JSON parsing errors ?
My requirement is that for any non well-formed JSON messages, I just want to save those off to another error queue and return a meaningful message to the user. |
|
| Back to top |
|
|
| rk1891 |
| Posted: Thu Jul 25, 2019 10:37 am Post subject: |
|
|
Apprentice
Joined: 12 Oct 2011
Posts: 42
|
Another interesting finding -
When the REST service responds with a non 200-299 status code, for example 401- Unauthorized, the Request node is sending the below JSON message in BLOB message tree even though I selected JSON as domain..
| Code: |
{
"message":"Unauthorized"
} |
|
|
| Back to top |
|
|
| abhi_thri |
| Posted: Thu Jul 25, 2019 9:56 pm Post subject: |
|
|
Knight
Joined: 17 Jul 2017
Posts: 516
Location: UK
|
| rk1891 wrote: |
| This as per the current architectural pattern we adopted. We've the adapter flows that does all the communication part of work and pass the info to Transformation flows where the actual transformation occurs. Even if I change the domain as JSON, don't think it failed on the Request node ie the message is NOT sent to Failure terminal.The response is sent through Out terminal and the failure is triggered only when I touch the contents under that JSON message tree in the subsequent compute node. |
This is because the by default the Response message parsing is set to 'OnDemand', try changing it to 'Immediate' and you should see the parsing failure at this node itself.
| rk1891 wrote: |
I could simply connect the failure terminal and catch this error on the Compute node where the transformation happens. But is there any error code/ static phrase that I could just simply look for JSON parsing errors ?
My requirement is that for any non well-formed JSON messages, I just want to save those off to another error queue and return a meaningful message to the user. |
Once the parse timing is changed to Immediate at the Request node you can then write logic at the failure terminal to traverse the ExpectionList and retrieve the validation error message. |
|
| Back to top |
|
|
| rk1891 |
| Posted: Fri Jul 26, 2019 6:23 am Post subject: |
|
|
Apprentice
Joined: 12 Oct 2011
Posts: 42
|
| abhi_thri wrote: |
| rk1891 wrote: |
| This as per the current architectural pattern we adopted. We've the adapter flows that does all the communication part of work and pass the info to Transformation flows where the actual transformation occurs. Even if I change the domain as JSON, don't think it failed on the Request node ie the message is NOT sent to Failure terminal.The response is sent through Out terminal and the failure is triggered only when I touch the contents under that JSON message tree in the subsequent compute node. |
This is because the by default the Response message parsing is set to 'OnDemand', try changing it to 'Immediate' and you should see the parsing failure at this node itself.
| rk1891 wrote: |
I could simply connect the failure terminal and catch this error on the Compute node where the transformation happens. But is there any error code/ static phrase that I could just simply look for JSON parsing errors ?
My requirement is that for any non well-formed JSON messages, I just want to save those off to another error queue and return a meaningful message to the user. |
Once the parse timing is changed to Immediate at the Request node you can then write logic at the failure terminal to traverse the ExpectionList and retrieve the validation error message. |
When I change it to Immediate, it does send to failure terminal but I lost the response content.
There are the settings I'm using on the Request node.
ResponseBodyLocation - $ResultRoot/JSON
OutputBodyLocation - $OutputBody
I change the ParseTiming back to OnDemand so I can try to get the contents out of the JSON tree as a string in the compute node but not been successful so far.
Is there anyway I could get the invalid JSON message as String ? |
|
| Back to top |
|
|
| rk1891 |
| Posted: Fri Jul 26, 2019 10:58 am Post subject: |
|
|
Apprentice
Joined: 12 Oct 2011
Posts: 42
|
Another question -
Even though I select ResponseBodyLocation as $ResultRoot/BLOB and the domain as BLOB on the Rest Request node, Why is the response is coming under this tree ?
| Quote: |
JSON
UnknownParserName
BLOB 7b200a20202276656869636c65496e73706
|
where as I'm expecting like below -
| Quote: |
BLOB
UnknownParserName
BLOB 7b200a20202276656869636c65496e73706
|
|
|
| Back to top |
|
|
| rk1891 |
| Posted: Mon Jul 29, 2019 11:46 am Post subject: |
|
|
Apprentice
Joined: 12 Oct 2011
Posts: 42
|
Can anyone please explain this or how we avoid this?
I almost dealt with all kinds of scenarios so far, this seems to be the final straw I need to get resolved. For this also I've the solution but feels like it's bandage on top of a original problem.
| rk1891 wrote: |
Another question -
Even though I select ResponseBodyLocation as $ResultRoot/BLOB and the domain as BLOB on the Rest Request node, Why is the response is coming under this tree ?
| Quote: |
JSON
UnknownParserName
BLOB 7b200a20202276656869636c65496e73706
|
where as I'm expecting like below -
| Quote: |
BLOB
UnknownParserName
BLOB 7b200a20202276656869636c65496e73706
|
|
|
|
| Back to top |
|
|
| abhi_thri |
| Posted: Tue Jul 30, 2019 1:22 am Post subject: |
|
|
Knight
Joined: 17 Jul 2017
Posts: 516
Location: UK
|
| hi...is this because you've got 'Automatic Content-Type handling' enabled at the RESTRequest node's 'Response Message Parsing' tab |
|
| Back to top |
|
|
| rk1891 |
| Posted: Tue Jul 30, 2019 5:16 am Post subject: |
|
|
Apprentice
Joined: 12 Oct 2011
Posts: 42
|
No, I do not have it enabled.
| abhi_thri wrote: |
| hi...is this because you've got 'Automatic Content-Type handling' enabled at the RESTRequest node's 'Response Message Parsing' tab |
|
|
| Back to top |
|
|
|
|
