| Author |
Message
|
| ruimadaleno |
 Posted: Mon Feb 13, 2017 3:46 am Post subject: Validating JSON payload |
 |
|
Master
Joined: 08 May 2014
Posts: 274
|
Hi all,
running iib v10.0.0.6 and building a proof of concept (POC) on building and running REST/Json services.
the goal here is to understand the technology, define models, check potential problems, define security,etc
One of the questions we are fighting with is the payload validation 
in a soap/xml service we have a contract (a wsdl) where the data type, format and restrictions of every element in the payload is defined.
in the REST/Json world we do not have this kind of contract, so, how do we validate the payload ? how do we validade that the json "CustomeName" element must be a string , max 100 char ?
All this validations must be codified and maintened in ESQL ? other options ?
_________________
Best regards
Rui Madaleno |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Mon Feb 13, 2017 5:15 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
This is a tricky one. JSON is by its very nature lightweight and flexible.
There are JSON Schemas. Google for them. There are .NET validators and I'm sure that there are others. No doubt with a little work you could craft a Java interface to them that is callable from IIB.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| timber |
| Posted: Mon Feb 13, 2017 5:55 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1280
|
| Quote: |
| There are JSON Schemas |
This is true. The problem is, there is no *real* standard for how to validate a JSON document against a JSON schema.
Say what you like about the readability of the XML Schema specification (and most of us do) but it is a properly reviewed and tested specification, and it has produced inter-operable implementations from a wide variety of teams. None of this is true of JSON schema. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Feb 13, 2017 6:02 am Post subject: Re: Validating JSON payload |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ruimadaleno wrote: |
| in the REST/Json world we do not have this kind of contract, so, how do we validate the payload ? how do we validade that the json "CustomeName" element must be a string , max 100 char ? |
Well you do have SWAGGER & YAML for this kind of role, and I believe IIBv10 supports SWAGGER but I agree with my worthy associate; a REST API is typically not associated with the kind of contract that a SOAP service is. The languages I mention are more means of providing information about your endpoint than they are about validation. You can enforce that "CustomerName" is a string but it would be anti-pattern to enforce a length.
The better question is why do you want such tight restrictions on REST APIs? Even if you have good business reasons, they're not going to be popular with the community into which you're attempting to inject them and obtain a revenue source.
Do not fall into the design trap of thinking REST APIs are like SOAP services with a different payload language. They're not intended to be that, even if a number of people think they are.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Feb 13, 2017 6:30 am Post subject: Re: Validating JSON payload |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| Vitor wrote: |
Do not fall into the design trap of thinking REST APIs are like SOAP services with a different payload language. They're not intended to be that, even if a number of people think they are. |
Never a truer set of words written on this forum.
I had this sort of battle many times in a previous job. The JSON believers insisted on the flexibility (over XML) and refused to take responsibility for their data being incorrectly structured.
In the end, they relented and went with XML.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| ruimadaleno |
| Posted: Mon Feb 13, 2017 7:16 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
Let us consider two scenarios:
1) In the end of the day data is stored in a relational/tightly structured data store. Typically a relational database, So somebody needs to validate data before it gets recorded in database tables.
In the limit the database itself will validate the data (disclaimer: i am assuming that the database model is "correct/complete" , there exists PK, FK and other constraints) and when the rest/json service tries to persist the data in datastore it get's errors like "PK - duplicate values", "referential integrity violated", "too large for column xxx".
In this scenario we have two options:
a) code ESQL to validate json - more time to develop, higher maintenance.
b) waste network resources letting the data travel to database and return with error.
2) In the end of the day data is stored in a non-relational/loose structured data store. Typically a noSQL database. In this case , the datastore it self can handle null elements, flexible schemas,etc.
some thougths:
So, maybe scenario 2) is the more correct to apply this "style" of services.
As for scenario 1) we need to talk about compromise/agreement and not contract 
on the server side there is a compromise of deliver a well documented service. on the client side there is a compromise of generating a payload that contains sufficient and valid data to execute business function. Any violation on this "compromise" will result on errors. both parts acknowledge this terms and there is no responsabity on the json validation by the rest service.
_________________
Best regards
Rui Madaleno |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Feb 13, 2017 8:19 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
I would put it to your that your scenarios are not the best examples.
There's no reason why you can't have a highly structured data store behind a REST API. A better question is why you would have direct access to that database from such an API rather than a more conventional SOAP XML. A update operation exposed by REST is more likely to run through some intermediate business process first - this is the root of my point about REST APIs fitting for different use cases than SOAP WS.
I certainly don't see a scenario where REST will cause us to all be using NoSQL in 5 years, and for DB2/Oracle/etc. to become obsolete.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| ruimadaleno |
| Posted: Mon Feb 13, 2017 9:54 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
| Quote: |
There's no reason why you can't have a highly structured data store behind a REST API.
|
I fully agree with you. The question here is: if you are inserting some rows in a oracle/db2 table, who is responsible for data validation ? the client/app ? the service (as we don't have any "standard" to validate JSON - Lots of ESQL code) ? let the request travel all the way to the database just to throw a "cannot insert null" error ?
_________________
Best regards
Rui Madaleno |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Feb 13, 2017 10:13 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| ruimadaleno wrote: |
I fully agree with you. The question here is: if you are inserting some rows in a oracle/db2 table, who is responsible for data validation ? the client/app ? the service (as we don't have any "standard" to validate JSON - Lots of ESQL code) ? let the request travel all the way to the database just to throw a "cannot insert null" error ? |
My worthless 2p on the subject is that the IIB flow should do the best it can to make sure that the data is correct before any DB update is attempted. Otherwise, you WILL get into the case of GIGO (Garbage In, Garbage Out)
Crap/bad data in the database is really hard to get rid of.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Feb 13, 2017 10:17 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ruimadaleno wrote: |
| The question here is: if you are inserting some rows in a oracle/db2 table, who is responsible for data validation ? the client/app ? |
Well, my personal opinion (and other opinions may be better or at least equally valid) is that your database will be a System Of Record. The REST API is a System Of Engagement (called a Presentation Layer back in the day) which would never be allowed to insert data by any means into an SOR. That insert would be performed by a System Of Integration.
Consider this scenario: you've exposed a REST API that allows you manipulate an asset, like a server or a printer. The process to add a new asset to the asset SOR is much more than just inserting rows into a data source. That's an Engagement, which will be performing a lot more validation than just checking all the fields are valid data types and isn't an obvious use case for REST.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Feb 13, 2017 10:21 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| smdavies99 wrote: |
Otherwise, you WILL get into the case of GIGO (Garbage In, Garbage Out)
Crap/bad data in the database is really hard to get rid of. |
Exactly so. Which is why you shouldn't use the same size hammer for all problems.
| smdavies99 wrote: |
| My worthless 2p on the subject is that the IIB flow should do the best it can to make sure that the data is correct before any DB update is attempted. |
Again, I put it to you that the changes to the data source are part of a far more rigid design contract than the REST methodology is intended to support. A REST resource is intended to be a small, light interface and (crucially) should be associated with a resource not an operation. So db operations are best hosted elsewhere and there's a place for both.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Feb 13, 2017 11:53 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| Vitor wrote: |
Again, I put it to you that the changes to the data source are part of a far more rigid design contract than the REST methodology is intended to support. A REST resource is intended to be a small, light interface and (crucially) should be associated with a resource not an operation. So db operations are best hosted elsewhere and there's a place for both. |
Quite right but...
A business has decided to use REST because is it a COOL thing and some $10000 a day consultant has said that it is the way of the future. The PHB's bought it lock stock and COST implications be botherted.
The DB probably exists so the poor IIB developer is piggiy in the middle and has to make the best of a bad job.
Warnings will be issued and ignored and opps the whole thing turns into a veritable crapfest all because the wrong decision was made at a very early stage.
They chose a 10kg sledge hammer and expected it to be capable of doing repousse work on a find silver pot.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Feb 13, 2017 12:17 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| smdavies99 wrote: |
A business has decided to use REST because is it a COOL thing and some $10000 a day consultant has said that it is the way of the future. The PHB's bought it lock stock and COST implications be botherted.
The DB probably exists so the poor IIB developer is piggiy in the middle and has to make the best of a bad job.
Warnings will be issued and ignored and opps the whole thing turns into a veritable crapfest all because the wrong decision was made at a very early stage.
They chose a 10kg sledge hammer and expected it to be capable of doing repousse work on a find silver pot. |
Bad day at the office dear? 
Everything you say is true, if more skeptical & negative than even I usually go. I've said before that we few, we happy few, we band of integrators, should push forward with doing things right even in the face of political opposition (or political disinterest).
Things will settle down after the first few crapfests have dampened the PHB's enthusiasm for cool, and the consultant is off evangelizing The Next Big Thing.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Feb 13, 2017 1:06 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| Vitor wrote: |
Bad day at the office dear?
|
Yes, the Central Heating boiler went Phut!
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| ruimadaleno |
| Posted: Tue Feb 14, 2017 9:02 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
| Vitor wrote: |
| ruimadaleno wrote: |
| The question here is: if you are inserting some rows in a oracle/db2 table, who is responsible for data validation ? the client/app ? |
Well, my personal opinion (and other opinions may be better or at least equally valid) is that your database will be a System Of Record. The REST API is a System Of Engagement (called a Presentation Layer back in the day) which would never be allowed to insert data by any means into an SOR. That insert would be performed by a System Of Integration.
Consider this scenario: you've exposed a REST API that allows you manipulate an asset, like a server or a printer. The process to add a new asset to the asset SOR is much more than just inserting rows into a data source. That's an Engagement, which will be performing a lot more validation than just checking all the fields are valid data types and isn't an obvious use case for REST. |
so in this scenario you would build something like:
Consumer Apps --> REST Service --> Integration Service --> insert into datasource
Rest service performs no data validation
Integration service is other service maybe deployed in iib in soap/xml style with contract and data validation
Integration service records data to datasource and maybe to other systems (as you said, creating an asset can be much more than just insert an record i n an SOR)
In this high level architecture the REST service is simply a façade to the integration service that converts JSON (or other format) to XML and delivers it to the integration service ?
_________________
Best regards
Rui Madaleno |
|
| Back to top |
|
|
|
|
