| Author |
Message
|
| anveshita |
 Posted: Fri Aug 06, 2021 7:45 am Post subject: JSON Message conversion issue |
 |
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
My apologies if this is already addressed. I tried to search and could not find it.
Our Z/OS system formats a JSON message in string form and puts the message on a cluster queue. This message is consumed by datapower MQ handler. DP is failing with invalid JSON. I looked at the message and it was the issue with [ (square brackets) which are coming to DP as invalid characters.
As a workaround, we have set the cluster receiver channel CONVERT to YES and this resolved the issue and the square brackets are coming out properly. Before standardizing it as a solution, I would like to know what is the best way to handle such situations. Please let me know. |
|
| Back to top |
|
 |
| PeterPotkay |
| Posted: Fri Aug 06, 2021 1:24 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
Enabling Convert on the channel, especially a Cluster Receiver channel, should not be needed. Do not go down this path.
Confirm the MQ QM Object in DataPower that parents the MQ Front Side Handler that DataPower uses to pull from this queue has "Convert Input" enabled. That will tell the MQ Queue Manager that DPWR is connected to to convert the message from the EBCDIC code page the z/OS app is producing it into the code page DataPower is running before handing it to DataPower.
It is also a requirement that the z/OS producing this message is shipping it with the MQMD CCSID set to the correct value that describes that payload.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| anveshita |
| Posted: Tue Aug 10, 2021 5:29 am Post subject: |
|
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
| PeterPotkay wrote: |
Enabling Convert on the channel, especially a Cluster Receiver channel, should not be needed. Do not go down this path.
Confirm the MQ QM Object in DataPower that parents the MQ Front Side Handler that DataPower uses to pull from this queue has "Convert Input" enabled. That will tell the MQ Queue Manager that DPWR is connected to to convert the message from the EBCDIC code page the z/OS app is producing it into the code page DataPower is running before handing it to DataPower.
It is also a requirement that the z/OS producing this message is shipping it with the MQMD CCSID set to the correct value that describes that payload. |
Thank you. On MQMD CCSID, Z/OS is producing the JSON as a string ( probably hardcoding). If so what is the CCSID they should be using. Any ideas? Probably that could be the issue. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Aug 10, 2021 8:14 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
| anveshita wrote: |
| PeterPotkay wrote: |
It is also a requirement that the z/OS producing this message is shipping it with the MQMD CCSID set to the correct value that describes that payload. |
Thank you. On MQMD CCSID, Z/OS is producing the JSON as a string ( probably hardcoding). If so what is the CCSID they should be using. Any ideas? Probably that could be the issue. |
Doesn't matter much what CCSID they are using (37 or 500). The important thing is that it must match the content of the message and the CCSID set on the message. 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| anveshita |
| Posted: Tue Aug 10, 2021 8:25 am Post subject: |
|
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
| fjb_saper wrote: |
| anveshita wrote: |
| PeterPotkay wrote: |
It is also a requirement that the z/OS producing this message is shipping it with the MQMD CCSID set to the correct value that describes that payload. |
Thank you. On MQMD CCSID, Z/OS is producing the JSON as a string ( probably hardcoding). If so what is the CCSID they should be using. Any ideas? Probably that could be the issue. |
Doesn't matter much what CCSID they are using (37 or 500). The important thing is that it must match the content of the message and the CCSID set on the message. |
I have confirmed that Z/OS is using CCSID (037). What Z/OS sending is a JSON message built as a string on the Z/OS to DataPower via MQ. I have confirmed the datapower has CONVERT on the front side handler. I am confused bacause Peter's suggestion was to look for correct MQMD CCSID, so wondering what is the correct CCSID to use if the string that is sent is a JSON message. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Aug 10, 2021 8:30 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9399
Location: US: west coast, almost. Otherwise, enroute.
|
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Aug 10, 2021 10:23 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
JSON or no JSON, it makes no difference. The important thing is that it is a text message (MQFMT_STRING). Now the mainframe may write the text in CCSID 37 (EBCDIC US) or CCSID 500 ( EBCDIC international) or in ASCII which could be any of CCSIDs (437, 819, 1208 ...) the important thing is that the message CCSID is set to the CCSID of the content before the put.
So if you are writing the message in CCSID 500 and your platform is CCSID 37 and the message carries the CCSID as 37, you may have a difference in the way you encoded the "{" character...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| hughson |
| Posted: Tue Aug 10, 2021 10:25 am Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1916
Location: Bay of Plenty, New Zealand
|
| anveshita wrote: |
| I am confused bacause Peter's suggestion was to look for correct MQMD CCSID, so wondering what is the correct CCSID to use if the string that is sent is a JSON message. |
| anveshita wrote: |
| I looked at the message and it was the issue with [ (square brackets) which are coming to DP as invalid characters. |
To discover the correct CCSID to use, it is helpful to review the problem characters. Square brackets are part of the variant set of characters in EBCDIC. This means that they differ between EBCDIC code pages. In contrast, the invariant set of characters (e.g. 0-9, a-z, A-Z and a few others) are the same in all EBCDICs.
- If your '[' (open square bracket) is represented by hex code point 0x4A, use CCSID 500.
- If your '[' is represented by hex code point 0xBA, use CCSID 37.
- If your '[' is represented by hex code point 0xAD, use CCSID 1047.
If the square brackets are coming from string literals in a compiled program, then I would guess CCSID 1047 which is the compiler CCSID (by default).
So have a look at the data in your message and figure out what hex value is being used for your square bracket characters and then you can set the CCSID in the MQMD to go with that.
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| anveshita |
| Posted: Tue Aug 10, 2021 1:18 pm Post subject: |
|
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
Thank you Bruce and FJB_SAPER
@hugson
| Quote: |
[*]If your '[' (open square bracket) is represented by hex code point 0x4A, use CCSID 500.
[*]If your '[' is represented by hex code point 0xBA, use CCSID 37.
[*]If your '[' is represented by hex code point 0xAD, use CCSID 1047.
[/list]
|
I have dig few feet deep 
1. Captured the message put by Z/OS on RFHUTIL and found that
is converted to
with Hex code
. Also confirmed that Z/OS is using code page "037". When I check on RFH Util I find the followng for the message: Under MQMD tab CCSID: 819, under RFH tab code page is:37 and CCSD:1208. Z/os is putting the message as MQFMT_String.
2. On the MQ front side handler of the datapower, on the queue manager object CONVERT is set to YES. The code page used by datapower is 819
3. When we enbaled CONVERT to yes on cluster receiver channel, the characters are getting converted properly and the datapower is able to process.
4. When we changed the flag to CONVERT to NO on cluster receiver channel. the characters are ntot getting converted.
I have reviewed the following http://www.mqseries.net/phpBB2/viewtopic.php?t=48061. It appears the solution is to change the default behavior of the DP Front sidehandler to not to convert. But it does not make sense to me.
It appears like Z/OS is putting the string in the right format, but for some reason the datapower is not able to see the right format ( conversion).
I am bit lost here and any help is appreciated[/code] |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Tue Aug 10, 2021 3:23 pm Post subject: |
|
|
Jedi
Joined: 25 Mar 2003
Posts: 2495
Location: Melbourne, Australia
|
Given that CONVERT(YES) on the channel is successfully converting the message, and you have confirmed the message ccsid is 37, and the square bracket is the correct hex code BA, it appears the issue is with conversion on DP.
Can you dump a message using amqsbcg? This will show all the MQMD fields and the raw unconverted RFH and data. There could be a lurking ccsid issue that is upsetting DP.
_________________
Glenn |
|
| Back to top |
|
|
| anveshita |
| Posted: Thu Aug 12, 2021 8:40 am Post subject: |
|
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
Thanks, gbaddeley.
The DP is converting JSON to JSONX to parse some data. When we send a JSON without sub -arrays the DP is not complaining. When we send large JSON with sub-arrays DP is failing with
| Code: |
| Unable to parse JSON and generate JSONx: Incomplete markup or missing document element at offset 39 of |
|
|
| Back to top |
|
|
| gbaddeley |
| Posted: Thu Aug 12, 2021 3:45 pm Post subject: |
|
|
Jedi
Joined: 25 Mar 2003
Posts: 2495
Location: Melbourne, Australia
|
| anveshita wrote: |
Thanks, gbaddeley.
The DP is converting JSON to JSONX to parse some data. When we send a JSON without sub -arrays the DP is not complaining. When we send large JSON with sub-arrays DP is failing with
| Code: |
| Unable to parse JSON and generate JSONx: Incomplete markup or missing document element at offset 39 of |
|
Assuming the JSON sub-arrays are valid, I suggest that you raise a service request with IBM.
_________________
Glenn |
|
| Back to top |
|
|
| hughson |
| Posted: Thu Aug 12, 2021 11:57 pm Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1916
Location: Bay of Plenty, New Zealand
|
| anveshita wrote: |
| the message put by Z/OS ... with Hex code
... using code page "037". Z/os is putting the message as MQFMT_String. |
This sounds like your message is indeed being correctly generated by the z/OS application.
| anveshita wrote: |
| When we enbaled CONVERT to yes on cluster receiver channel, the characters are getting converted properly and the datapower is able to process. |
This also suggests that your messages is correctly generated.
The difference between using CONVERT(YES) on a channel, or using CONVERT(NO) is where the data conversion gets done.
By setting CONVERT(YES) on your cluster-receiver, you are actually causing CONVERT(YES) to be set on your automatically defined cluster-sender. This means that the MQGET that is done to retrieve the message from the XMITQ, is done using MQGMO_CONVERT and the message is converted into the code page of the target queue manager at the other end of the channel. It would appear that this conversion is successfully done.
By changing to CONVERT(NO), the MQGET that is done to retrieve the message from the XMITQ is done without MQGMO_CONVERT and the message arrives at the destination queue manager still in codepage EBCDIC 037. When the application, in this case DataPower, gets the message from the queue, the data conversion now needs to be done. It is my understanding that there are still two places where data conversion could take place. Either DataPower could do an MQGET using MQGMO_CONVERT and have the SVRCONN on the queue manager actually do the data conversion, or the DataPower could not ask for MQGMO_CONVERT and do the data conversion actually on the DataPower box itself.
It seems that the configuration to tell the queue manager to do the data conversion is in the DataPower "IBM MQ queue manager" in a field called "Convert Input" (see DataPower Gateways > Configuring an IBM MQ queue manager). Is that what you mean when you say:-
| anveshita wrote: |
| On the MQ front side handler of the datapower, on the queue manager object CONVERT is set to YES |
Alternatively, the DataPower box itself will do the data conversion after the MQGET call is finished. I think if you switch off "Convert Input" then this might be what happens.
If you suspect that the SVRCONN is where the data conversion is being performed incorrectly, you could try using an alternative simple application that utilises MQGMO_CONVERT to see if it can correctly convert the message. amqsgetc is one such example. However, I think that your RFHUtil investigations probably also already prove this.
If you suspect that the Datapower box itself is where the data conversion is being performed incorrectly, you could try changing over to have the data conversion done on the queue manager.
Either way, I would suggest contacting IBM and raising a case, as it sounds like one of the IBM products, either the queue manager hosting the cluster queue, or the DataPower box, is failing to correctly convert from EBCDIC 037 to ASCII 819. That sounds like quite a common combination to me!
I should say here, that I am not a DataPower expert, only an MQ expert, so I don't know the DP configuration well and may well have missed some other nuance about how to control when DP does the data conversion itself.
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| anveshita |
| Posted: Fri Aug 13, 2021 2:31 am Post subject: |
|
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
| Code: |
| It seems that the configuration to tell the queue manager to do the data conversion is in the DataPower "IBM MQ queue manager" in a field called "Convert Input" (see DataPower Gateways > Configuring an IBM MQ queue manager). Is that what you mean when you say:- |
Thanks Hudson. Yes that is correct.
Working with the team to open a PMR and see how it goes. In the mean time trying different options and will keep you posted, |
|
| Back to top |
|
|
| anveshita |
| Posted: Mon Aug 16, 2021 2:15 pm Post subject: |
|
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
| gbaddeley wrote: |
Given that CONVERT(YES) on the channel is successfully converting the message, and you have confirmed the message ccsid is 37, and the square bracket is the correct hex code BA, it appears the issue is with conversion on DP.
Can you dump a message using amqsbcg? This will show all the MQMD fields and the raw unconverted RFH and data. There could be a lurking ccsid issue that is upsetting DP. |
We ran the amqscbcg. For this test, the convert flag is YES st on cluster receiver channel is on. I am unable to share the data part but can share the header with some info masked.
| Code: |
****Message descriptor****
StrucId : 'MD ' Version : 2
Report : 0 MsgType : 8
Expiry : -1 Feedback : 0
Encoding : 546 CodedCharSetId : 819
Format : 'MQHRF2 '
Priority : 0 Persistence : 0
MsgId : X'F1F8F2F2F8F0F0F0F0F0F2F5F6F0F8404040404040404040'
CorrelId : X'000000000000000000000000000000000000000000000000'
BackoutCount : 0
ReplyToQ : 'xxx.xxx.xxxx.xxxx '
ReplyToQMgr : 'xxxx '
** Identity Context
UserIdentifier : 'xxxxx '
AccountingToken :
X'0000000000000000000000000000000000000000000000000000000000000000'
ApplIdentityData : ' '
** Origin Context
PutApplType : '2'
PutApplName : 'xxxxxxx '
PutDate : '2021xxxx' PutTime : 'xxxxxx'
ApplOriginData : ' '
GroupId : X'000000000000000000000000000000000000000000000000'
MsgSeqNumber : '1'
Offset : '0'
MsgFlags : '0'
OriginalLength : '-1'
|
Please see Encoding : 546 CodedCharSetId : 819 from the amqsbcg program output. The Z/os is using "037" though. We ran both with channel conversion ON and OFF on the request cluster channel. But did not see any difference in the amqsbcg output of the encoding. So I am bit confused. |
|
| Back to top |
|
|
|
|
