| Author |
Message
|
| sumit |
 Posted: Wed Jan 27, 2016 8:29 am Post subject: Handling Special characters in JSON |
 |
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
Hi,
OS - Windows
IIB 9001
Scenario - Calling REST service hosted at Azure. Payload has JSON message and a file.
I am using ESQL to generate a multipart http request message and calling the REST service. In the JSON payload, some of the fields have either destination URI or fully-qualified location for a file; and hence has '/' (forward slash) in it.
For eg., one of the fields is like
{ "Site_URL":"http://abc.com"}
Now, forward slash being a special character, the JSON Parser at IIB replaces it with '\/' i.e. now the url value becomes-
"http:\/\/abc.com"
We have captured the request going to the REST service using Fiddler and WireShark for requests going from IIB and from another C# service (the other service is working fine) . The only difference 'appears' to be this change in data.
I looked into this forum, infocenter and at other places but didn't find any info which suggest how I can send the exact same URL.
I must mention, I am not sure if the request is failing because of this error. However, we know a C# class is calling the same REST service and sending the proper request and it is working.
At Azure end, the error is coming because the service is not able to parse the data properly.
_________________
Regards
Sumit |
|
| Back to top |
|
 |
| sumit |
| Posted: Wed Jan 27, 2016 8:31 am Post subject: |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
Just to be clear, WireShark is running on my machine and I am using it to capture the outgoing request message.
_________________
Regards
Sumit |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jan 27, 2016 8:42 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You shouldn't need to escape characters in JSON like this?
What does a usertrace show in the message tree as it approaches the output node?
What CCSID are you using? Are you using the JSON parser for the JSON message?
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| adubya |
| Posted: Wed Jan 27, 2016 8:45 am Post subject: |
|
|
Partisan
Joined: 25 Aug 2011
Posts: 377
Location: GU12, UK
|
| I'm not sure what the problem is ? IIB is correctly escaping the forward slash in the JSON payload, so what's going wrong ? |
|
| Back to top |
|
|
| sumit |
| Posted: Wed Jan 27, 2016 9:17 am Post subject: |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
| mqjeff wrote: |
| What does a usertrace show in the message tree as it approaches the output node? |
I am bundling the JSON payload and the content from input file in a blob while calling the REST service. The content from user trace is-
| Code: |
( ['MQROOT' : 0x2befccd0]
(0x01000000:Name):HTTPRequestHeader = ( ['WSREQHDR' : 0x2c3ead50]
(0x03000000:NameValue):Content-Type = 'multipart/form-data; boundary=POCUniqueBoundry' (CHARACTER)
(0x03000000:NameValue):Accept = '*/*' (CHARACTER)
)
(0x01000000:Name):BLOB = ( ['none' : 0x2c3eaa40]
(0x03000000:NameValue):BLOB = <big blob> (BLOB)
)
) |
which converts to-
| Code: |
--POCUniqueBoundry
Content-Type: text/plain; charset=utf-8
Content-Disposition: form-data
{"Properties":[{"Type":"text","Value":"test","Name":"doctype1"}],"AzureConnectionString":"DefaultEndpointsProtocol=https;AccountName=someaccountname;AccountKey=somekey","AzureBlobName":"811YZKNQ","AzureContainer":"uploads","SP_Site_URL":"<https:\/\/my.abc.com\/sites\/jwestow>","SharedFile":"NULL","SP_Document_URL":"<\/sites\/jwestow\/Documents\/Xingtest.exe>","SP_Document_Library_URL":"\/sites\/jwestow\/Documents","SP_Document_Library_Name":"Documents","FailureNotificationEmail":"sumit.sharma@xyz.com","SuccessNotificationEmail":"sumit.sharma@xyz.com","NumRetries":10}
--POCUniqueBoundry
Content-Disposition: form-data; name=file; filename=zscaler.xlsx; filename*=utf-8'zscaler.xlsx
<content of the file>--POCUniqueBoundry
|
| mqjeff wrote: |
| What CCSID are you using? |
CCSID set on my qmgr is 437.
| mqjeff wrote: |
| Are you using the JSON parser for the JSON message? |
Yes. JSON part of the ESQL code looks like-
| Code: |
CREATE FIELD OutputRoot.JSON.Data.Properties IDENTITY(JSON.Array)Properties;
SET OutputRoot.JSON.Data.Properties.Item[1].Type = 'text';
SET OutputRoot.JSON.Data.Properties.Item[1].Value = 'test';
SET OutputRoot.JSON.Data.Properties.Item[1].Name = 'doctype1';
SET OutputRoot.JSON.Data.AzureConnectionString = 'DefaultEndpointsProtocol=https;AccountName=someaccountname;AccountKey=somekey';
SET OutputRoot.JSON.Data.AzureBlobName = '811YZKNQ';
SET OutputRoot.JSON.Data.AzureContainer = 'uploads';
-- SET OutputRoot.JSON.Data.UploadDoc = 'true';
SET OutputRoot.JSON.Data.SP_Site_URL = eSP_Site_URL;
SET OutputRoot.JSON.Data.SharedFile = 'NULL';
SET OutputRoot.JSON.Data.SP_Document_URL = eSP_Document_URL|| '/' || eFileName;
SET OutputRoot.JSON.Data.SP_Document_Library_URL = eSP_Document_URL;
SET OutputRoot.JSON.Data.SP_Document_Library_Name = cLibName;
SET OutputRoot.JSON.Data.FailureNotificationEmail = eFailureEmail;
SET OutputRoot.JSON.Data.SuccessNotificationEmail = eSuccessEmail;
SET OutputRoot.JSON.Data.NumRetries = 10; |
All the variables you see here are UDPs.
And then bundling the JSON in blob-
| Code: |
SET blPayload = ASBITSTREAM(OutputRoot.JSON.Data, InputProperties.Encoding, InputProperties.CodedCharSetId);
|
this variable is then attached to the multipart request.[/code]
_________________
Regards
Sumit |
|
| Back to top |
|
|
| sumit |
| Posted: Wed Jan 27, 2016 9:26 am Post subject: |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
| adubya wrote: |
| I'm not sure what the problem is ? IIB is correctly escaping the forward slash in the JSON payload, so what's going wrong ? |
I agree. However, I am not sure why the service is failing.
From Fiddler, here is how a good service request looks like (this was from C# application)-
| Code: |
POST <http://abc.com/upload/api/files HTTP/1.1>
Content-Type: multipart/form-data; boundary="10540b71-eff1-4409-ab96-d370d27f8230"
Host: abc.com
Content-Length: 1320
Expect: 100-continue
Connection: Keep-Alive
--10540b71-eff1-4409-ab96-d370d27f8230
Content-Type: text/plain; charset=utf-8
Content-Disposition: form-data
{"Properties":[{"Type":"text","Value":"test","Name":"doctype1"}],"AzureConnectionString":"DefaultEndpointsProtocol=https;AccountName=someaccountname;
AccountKey=somekey","AzureBlobName":"811YZKNQ","AzureContainer":"uploads","SP_Site_URL":"https://abc.com/sites/jwestow",
"SharedFile":null,"SP_Document_URL":"/sites/jwestow/Documents/Xingtest.exe","SP_Document_Library_URL":"/sites/jwestow/Documents",
"SP_Document_Library_Name":"Documents","FailureNotificationEmail":"milo@xyz.com","SuccessNotificationEmail":"milo@xyz.com","NumRetries":"10"}
--10540b71-eff1-4409-ab96-d370d27f8230
Content-Disposition: form-data; name=file; filename=zscaler.xlsx; filename*=utf-8''zscaler.xlsx
<file content>--10540b71-eff1-4409-ab96-d370d27f8230-- |
and here is how my IIB request looks like, from Wireshark
| Code: |
POST /upload/api/files HTTP/1.0
Content-Length: 1293
Content-Type: multipart/form-data; boundary=POCUniqueBoundry
Accept: */*
Host: abc.com
SOAPAction: ""
--POCUniqueBoundry
Content-Type: text/plain; charset=utf-8
Content-Disposition: form-data
{"Properties":[{"Type":"text","Value":"test","Name":"doctype1"}],"AzureConnectionString":"DefaultEndpointsProtocol=https;AccountName=someaccountname;AccountKey=somekey",
"AzureBlobName":"811YZKNQ","AzureContainer":"uploads","SP_Site_URL":"<https:\/\/abc.com\/sites\/jwestow>","SharedFile":"NULL","SP_Document_URL":"<\/sites\/jwestow\/Documents\/Xingtest.exe>","SP_Document_Library_URL":"\/sites\/jwestow\/Documents",
"SP_Document_Library_Name":"Documents","FailureNotificationEmail":"sumit.sharma@xyz.com","SuccessNotificationEmail":"sumit.sharma@xyz.com","NumRetries":10}
--POCUniqueBoundry
Content-Disposition: form-data; name=file; filename=zscaler.xlsx; filename*=utf-8'zscaler.xlsx
<file content>--POCUniqueBoundry |
_________________
Regards
Sumit |
|
| Back to top |
|
|
| sumit |
| Posted: Wed Jan 27, 2016 9:27 am Post subject: |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
Error I am getting is
| Code: |
| 500: Internal Server Error |
_________________
Regards
Sumit |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jan 27, 2016 9:59 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Can you find out why you'd have
as part of your transmission?? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| sumit |
| Posted: Wed Jan 27, 2016 12:16 pm Post subject: |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
| fjb_saper wrote: |
Can you find out why you'd have
as part of your transmission?? |
The SOAPAction was going as part of http header because I had asked IIB to generate the HTTP header. The property 'Generate Default HTTP headers from input' was checked.
I have unchecked it and now the SOAPAction is gone.
The HTTP request header now looks like-
| Code: |
POST /upload/api/files HTTP/1.0
Content-Type: multipart/form-data; boundary=POCUniqueBoundry
Accept: */*
Content-Length: 1289
--POCUniqueBoundry
Content-Type: text/plain; charset=utf-8
Content-Disposition: form-data.... |
and then the remaining portion, as I've already shared.
_________________
Regards
Sumit |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jan 27, 2016 12:20 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
What's the parsing error that the Azure service throws?
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| sumit |
| Posted: Wed Jan 27, 2016 12:57 pm Post subject: |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
I have requested for the logs. I will share as soon as I get it.
_________________
Regards
Sumit |
|
| Back to top |
|
|
| sumit |
| Posted: Wed Jan 27, 2016 2:18 pm Post subject: |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
It finally worked!
The final boundary in my message was looking like
However, the final boundary in the successful message was like
| Code: |
| --10540b71-eff1-4409-ab96-d370d27f8230[b]--[/b] |
Those extra hyphens in the end was causing all the problem.
I finally make my final boundary to look like-
| Code: |
| --POCUniqueBoundry-- |
and it worked!
However, the request was then failing at the REST Service provider. For now, we put a dirty fix which will look for combination of backward slash-forward slash '\/' and will replace it with forward slash '/' so the URL embedded in JSON message remains valid.
I did some more reading and understood that in other languages, one can chose not-to-escape the forward slash character. Still finding out how we can do this in IIB.
https://code.google.com/archive/p/json-simple/issues/8
_________________
Regards
Sumit |
|
| Back to top |
|
|
| timber |
| Posted: Thu Jan 28, 2016 12:15 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
|
| Back to top |
|
|
| stoney |
| Posted: Thu Jan 28, 2016 1:54 am Post subject: |
|
|
Centurion
Joined: 03 Apr 2013
Posts: 140
|
| Quote: |
| For now, we put a dirty fix which will look for combination of backward slash-forward slash '\/' and will replace it with forward slash '/' so the URL embedded in JSON message remains valid. |
You shouldn't need to do this.
If you use a JSON parser that is compliant with the specification, it should remove all of the escape sequences from the JSON strings in the JSON message, and replace them with the escaped characters.
This should happen before the JSON parser presents those strings to the application.
The application should never see the escape sequences - \\, \/, \", \f, \n, etc.
| Quote: |
| ...unless stoney comes along and tells us how to do it using existing facilities. |
No facilities currently exist I'm afraid! |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jan 28, 2016 6:10 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| stoney wrote: |
You shouldn't need to do this.
If you use a JSON parser that is compliant with the specification, it should remove all of the escape sequences from the JSON strings in the JSON message, and replace them with the escaped characters.
This should happen before the JSON parser presents those strings to the application.
The application should never see the escape sequences - \\, \/, \", \f, \n, etc. |
So (for the  among us), are you saying that this is analogous to an XML parser ensuring that an application processes a string "Fish & Chips" even though the XML tag value was "Fish & Chips" and the OP's problem is principally that whatever's parsing in Azure is failing to do this?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
