ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker SupportJSON message validation

Post new topicReply to topic
JSON message validation View previous topic :: View next topic
Author Message
rk1891
PostPosted: Wed Jul 24, 2019 2:08 pm Post subject: JSON message validation Reply with quote

Apprentice

Joined: 12 Oct 2011
Posts: 37

Hello all,

I'm facing some strange situation with JSON message parsing.

I've a REST request Node that is invoking a REST API to get a JSON response. On this node I've selected the message domain as BLOB as I want to pass in the message as-is to downstream ESQL compute node in another flow.
In the compute node I'm converting the BLOB to JSON and accessing the JSON response using JSON.Data.'xxx'.Everything works as expected when I get a valid JSON message as the response.

However, I stumbled on a scenario where in the REST service I'm calling returned a message with an invalid JSON structure and my compute node failed to parse the message.
I see that it's failing as soon as it hits JSON.Data xpath.

Finally my question is:
Is there anyway to check if the response is a valid conforming JSON message so I could do that detailed transformation only when it is valid otherwise I can simply convert the incoming BLOB to CHAR and pass on the JSON parsing error message to downstream.

I tried different ways but none of them seem to work, the below are couple of tries -
Code:

FIELDNAME(InputRoot.JSON.Data) IS NOT NULL

Quote:
This one gets through as there is some data under JSON
Code:

EXISTS(InputRoot.JSON.[])


[/code]

Any help is appreciated!
Back to top
View user's profile Send private message
timber
PostPosted: Thu Jul 25, 2019 1:56 am Post subject: Reply with quote

Grand Master

Joined: 25 Aug 2015
Posts: 1012

Quote:
On this node I've selected the message domain as BLOB as I want to pass in the message as-is to downstream ESQL compute node in another flow.
I don't see any advantage in that approach. Is there a specific reason why you did not simply set the domain to JSON on the request node?

Quote:
the REST service I'm calling returned a message with an invalid JSON structure and my compute node failed to parse the message.
Important question: did the REST service return invalid JSON or valid JSON with unexpected/missing data. If the returned JSON is invalid then you will get a parser exception from the JSON parser. If the returned JSON is valid but does not conform to the API specs then you have a different problem.
Currently, IIB does not offer JSON validation. That's because the validation rules were not properly standardised when Swagger 2.0 was released. If you really want to validate against the Swagger, there are Java class libraries that will do the job for you.
Back to top
View user's profile Send private message
rk1891
PostPosted: Thu Jul 25, 2019 6:25 am Post subject: Reply with quote

Apprentice

Joined: 12 Oct 2011
Posts: 37

timber wrote:
Quote:
On this node I've selected the message domain as BLOB as I want to pass in the message as-is to downstream ESQL compute node in another flow.
I don't see any advantage in that approach. Is there a specific reason why you did not simply set the domain to JSON on the request node?

This as per the current architectural pattern we adopted. We've the adapter flows that does all the communication part of work and pass the info to Transformation flows where the actual transformation occurs. Even if I change the domain as JSON, don't think it failed on the Request node ie the message is NOT sent to Failure terminal.The response is sent through Out terminal and the failure is triggered only when I touch the contents under that JSON message tree in the subsequent compute node.


timber wrote:

Quote:
the REST service I'm calling returned a message with an invalid JSON structure and my compute node failed to parse the message.
Important question: did the REST service return invalid JSON or valid JSON with unexpected/missing data. If the returned JSON is invalid then you will get a parser exception from the JSON parser. If the returned JSON is valid but does not conform to the API specs then you have a different problem.
Currently, IIB does not offer JSON validation. That's because the validation rules were not properly standardised when Swagger 2.0 was released. If you really want to validate against the Swagger, there are Java class libraries that will do the job for you.

I kind of see this question coming. It's an invalid JSON(or not well-formed in XML semantics). I do have a Swagger YAML for this service which I'm using on the Request node by referring it from a static library. I'm not planning to validate the response against the Swagger document or such as I've already learnt that it's not possible.

I could simply connect the failure terminal and catch this error on the Compute node where the transformation happens. But is there any error code/ static phrase that I could just simply look for JSON parsing errors ?
My requirement is that for any non well-formed JSON messages, I just want to save those off to another error queue and return a meaningful message to the user.
Back to top
View user's profile Send private message
rk1891
PostPosted: Thu Jul 25, 2019 10:37 am Post subject: Reply with quote

Apprentice

Joined: 12 Oct 2011
Posts: 37

Another interesting finding -
When the REST service responds with a non 200-299 status code, for example 401- Unauthorized, the Request node is sending the below JSON message in BLOB message tree even though I selected JSON as domain..
Code:
{
"message":"Unauthorized"
}
Back to top
View user's profile Send private message
abhi_thri
PostPosted: Thu Jul 25, 2019 9:56 pm Post subject: Reply with quote

Master

Joined: 17 Jul 2017
Posts: 295
Location: UK

rk1891 wrote:
This as per the current architectural pattern we adopted. We've the adapter flows that does all the communication part of work and pass the info to Transformation flows where the actual transformation occurs. Even if I change the domain as JSON, don't think it failed on the Request node ie the message is NOT sent to Failure terminal.The response is sent through Out terminal and the failure is triggered only when I touch the contents under that JSON message tree in the subsequent compute node.


This is because the by default the Response message parsing is set to 'OnDemand', try changing it to 'Immediate' and you should see the parsing failure at this node itself.


rk1891 wrote:

I could simply connect the failure terminal and catch this error on the Compute node where the transformation happens. But is there any error code/ static phrase that I could just simply look for JSON parsing errors ?
My requirement is that for any non well-formed JSON messages, I just want to save those off to another error queue and return a meaningful message to the user.


Once the parse timing is changed to Immediate at the Request node you can then write logic at the failure terminal to traverse the ExpectionList and retrieve the validation error message.
Back to top
View user's profile Send private message
rk1891
PostPosted: Fri Jul 26, 2019 6:23 am Post subject: Reply with quote

Apprentice

Joined: 12 Oct 2011
Posts: 37

abhi_thri wrote:
rk1891 wrote:
This as per the current architectural pattern we adopted. We've the adapter flows that does all the communication part of work and pass the info to Transformation flows where the actual transformation occurs. Even if I change the domain as JSON, don't think it failed on the Request node ie the message is NOT sent to Failure terminal.The response is sent through Out terminal and the failure is triggered only when I touch the contents under that JSON message tree in the subsequent compute node.


This is because the by default the Response message parsing is set to 'OnDemand', try changing it to 'Immediate' and you should see the parsing failure at this node itself.


rk1891 wrote:

I could simply connect the failure terminal and catch this error on the Compute node where the transformation happens. But is there any error code/ static phrase that I could just simply look for JSON parsing errors ?
My requirement is that for any non well-formed JSON messages, I just want to save those off to another error queue and return a meaningful message to the user.


Once the parse timing is changed to Immediate at the Request node you can then write logic at the failure terminal to traverse the ExpectionList and retrieve the validation error message.


When I change it to Immediate, it does send to failure terminal but I lost the response content.

There are the settings I'm using on the Request node.
    ResponseBodyLocation - $ResultRoot/JSON
    OutputBodyLocation - $OutputBody

I change the ParseTiming back to OnDemand so I can try to get the contents out of the JSON tree as a string in the compute node but not been successful so far.

Is there anyway I could get the invalid JSON message as String ?
Back to top
View user's profile Send private message
rk1891
PostPosted: Fri Jul 26, 2019 10:58 am Post subject: Reply with quote

Apprentice

Joined: 12 Oct 2011
Posts: 37

Another question -

Even though I select ResponseBodyLocation as $ResultRoot/BLOB and the domain as BLOB on the Rest Request node, Why is the response is coming under this tree ?
Quote:

JSON
UnknownParserName
BLOB 7b200a20202276656869636c65496e73706

where as I'm expecting like below -
Quote:

BLOB
UnknownParserName
BLOB 7b200a20202276656869636c65496e73706
Back to top
View user's profile Send private message
rk1891
PostPosted: Mon Jul 29, 2019 11:46 am Post subject: Reply with quote

Apprentice

Joined: 12 Oct 2011
Posts: 37

Can anyone please explain this or how we avoid this?

I almost dealt with all kinds of scenarios so far, this seems to be the final straw I need to get resolved. For this also I've the solution but feels like it's bandage on top of a original problem.

rk1891 wrote:
Another question -

Even though I select ResponseBodyLocation as $ResultRoot/BLOB and the domain as BLOB on the Rest Request node, Why is the response is coming under this tree ?
Quote:

JSON
UnknownParserName
BLOB 7b200a20202276656869636c65496e73706

where as I'm expecting like below -
Quote:

BLOB
UnknownParserName
BLOB 7b200a20202276656869636c65496e73706
Back to top
View user's profile Send private message
abhi_thri
PostPosted: Tue Jul 30, 2019 1:22 am Post subject: Reply with quote

Master

Joined: 17 Jul 2017
Posts: 295
Location: UK

hi...is this because you've got 'Automatic Content-Type handling' enabled at the RESTRequest node's 'Response Message Parsing' tab
Back to top
View user's profile Send private message
rk1891
PostPosted: Tue Jul 30, 2019 5:16 am Post subject: Reply with quote

Apprentice

Joined: 12 Oct 2011
Posts: 37

No, I do not have it enabled.
abhi_thri wrote:
hi...is this because you've got 'Automatic Content-Type handling' enabled at the RESTRequest node's 'Response Message Parsing' tab
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexWebSphere Message Broker SupportJSON message validation
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.