| Author |
Message
|
| ghoshly |
 Posted: Fri Sep 26, 2014 4:30 am Post subject: Handle multi byte character through SOAP message in UTF-8 |
 |
|
Partisan
Joined: 10 Jan 2008
Posts: 326
|
Hello,
I am requesting your help to understand the reason behind the failure to handle multi byte unicode character through SOAP message in UTF-8.
Websphere Message Broker V 8.0.0.4
Websphere MQ 7.5.0.1
Queue Manager Default CCSID - 819
Our message flow looks like SOAP Input node - SOAP Extract - Multiple compute - MQ Output node.
Significant code snippet mentioned below. Please help to understand the reason of failure.
| Code: |
DECLARE intCCSID INTEGER InputRoot.Properties.CodedCharSetId;
DECLARE intEncoding INTEGER InputRoot.Properties.Encoding;
CALL CopyMessageHeaders();
SET OutputRoot.HTTPInputHeader = NULL;
-- No MQMD header so create domain
CREATE LASTCHILD OF OutputRoot DOMAIN ('MQMD') NAME 'MQMD';
SET OutputRoot.MQMD.Version = MQMD_CURRENT_VERSION;
SET OutputRoot.MQMD.ApplIdentityData = SQL.BrokerName;
SET OutputRoot.MQMD.CodedCharSetId = intCCSID;
SET OutputRoot.MQMD.Encoding = intEncoding;
/*Create MQRFH2 header if one doesnt exist*/
IF NOT EXISTS(InputRoot.MQRFH2[]) THEN
SET OutputRoot.MQMD.Format = 'MQHRF2';
SET OutputRoot.MQRFH2.(MQRFH2.Field)Version = 2;
SET OutputRoot.MQRFH2.(MQRFH2.Field)Format = InputRoot.MQMD.Format;
SET OutputRoot.MQRFH2.(MQRFH2.Field)NameValueCCSID = 1208;
SET OutputRoot.MQRFH2.usr.OriginalHTTPInputHeader = InputRoot.HTTPInputHeader;
SET OutputRoot.MQRFH2.usr.MainMessageFlowName = MessageFlowLabel;
END IF;
|
When I took message broker trace, we get as below -
| Quote: |
RecoverableException BIP2136E: Source character ''201d'' in field
The source character is an invalid code point within the given codepage.
Correct the application or message flow that generated the message to ensure that all data within the message can be represented in the target codepage. |
Character in question is ” i.e. U+201D Right Double Quotation Mark
Not sure why Message Broker is trying to use Queue Manager default character set when we are specifying UTF-8 to be used. intCCSID field is getting the value as 1208 |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Fri Sep 26, 2014 4:49 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
In the RFH2 you are setting the CCSID for the name, value pair section.
I did not see you set a CCSID for the next section (payload).
Neither did you set a CCSID in the Properties.
On the MQMD you are setting the CCSID of the incoming Properties, and this may not be 1208...
So your flow is probably right to protest that the codepoint is not defined in the output CCSID...
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Sep 26, 2014 4:51 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
There are several things wrong with your code.
For one...
| Code: |
SET OutputRoot.MQRFH2.(MQRFH2.Field)Format = InputRoot.MQMD.Format;
|
Where does the InputRoot.MQMD come from?
You created the MQMD properly. Why didn't you do the same for the RFH2? (ok, that's a nitpick) but do you really think that you will have an RFH2 header in your message that has just arrived via a SOAP node?
This sounds like you have been doing some cut/paste from other flows.
OR even think about using the properties fields available within MQ.
But frankly, if I were developing this, then I'd start by introducing some trace nodes and use UserTrace s othat I can be sure that I am doing what I think I am doing.
But hey, what do I know, I'm just an almost extinct 
you also have to ask yourself, is the input message properly constructed if it contains multibyte characters? If the data in the SOAP message froper XML with all the delimiters etc for whatever multibyte CCSID you are using?
etc, etc, etc
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| ghoshly |
| Posted: Fri Sep 26, 2014 5:21 am Post subject: response |
|
|
Partisan
Joined: 10 Jan 2008
Posts: 326
|
I have already noticed the errorneous code to create format field within MQRFH2 but at this point I didn't removed it before presenting here, because this is what is running in environment.
We certainly do not have RFH2 in input coming from SOAP node, but we are creating RFH2 header in output to store usr fields within.
The mentioned quote in my post is from user trace only.
| Quote: |
| On the MQMD you are setting the CCSID of the incoming Properties, and this may not be 1208... |
We have copied all message headers, hence OutputRoot.Properties also has the CCSID value, and the same is also mentioned within OutputRoot.MQMD as we have verified the value of the variable intCCSID (1208) |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Sep 26, 2014 5:52 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Ok so you think you have go the various headers all hunky-dory but what about this...
| smdavies99 wrote: |
you also have to ask yourself, is the input message properly constructed if it contains multibyte characters? If the data in the SOAP message froper XML with all the delimiters etc for whatever multibyte CCSID you are using?
|
Is the data actually correct? What does it look like?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| ghoshly |
| Posted: Fri Sep 26, 2014 8:13 am Post subject: How to confirm ? |
|
|
Partisan
Joined: 10 Jan 2008
Posts: 326
|
I have put a Trace node after the SOAP Input node and there we are getting charset UTF-8 in the HTTPInputHeader, so as 1208 in the InputRoot.Properties.CodedcharsetId
Even if we are mapping InputRoot.Properties.CodedcharsetId to OutputRoot.Properties.CodedcharsetId, or copy message headers, is there a chance that it gets overridden or I need set by using hard coded 1208 value?
Sounds definitely weird. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Sep 26, 2014 8:22 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
As you are creating an RFH2 header, you did not show in the code where on the RFH2 you are setting the CCSID for the next section (i.e. payload).
You are setting the ccsid for the properties (copy from input), for the RFH2 on MQMD header and for the RF2 name, value pairs.... but not for the payload... as per your code example....
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Sep 26, 2014 8:25 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
But..... My point has not been answered.
Is the data really sent to you as UTF-8 really UTF-8? Could it have been written differently?
Does every characters hex value match the UTF-8 Character Set?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| ghoshly |
| Posted: Fri Sep 26, 2014 11:21 am Post subject: sorry |
|
|
Partisan
Joined: 10 Jan 2008
Posts: 326
|
fjb,
Now I got your point, my idea was when we set CCSID as name value pair in RFH2 it will serve the purpose for payload. My lack of knowledge... I'll correct that next time and see if there is some silver line.
Davies,
Certainly we couldn't confirm that fully. We have seen the HTTPInput heaser, as I said lastly.
After that, we put a File Output node instead of trace node, after the SOAP Input node to capture the payload. Not quite sure what encoding / character set it (File output node) is using while writing the content in a file. Once we try to open the file in text editor, it first tries to open using ANSI and then ANSI with BOM, so little confused.
I'll take your remarks and try again next day.. today completely drained. |
|
| Back to top |
|
|
| ghoshly |
| Posted: Mon Sep 29, 2014 12:24 am Post subject: Code Updated - Not sure how it works |
|
|
Partisan
Joined: 10 Jan 2008
Posts: 326
|
Please see the updated code below
| Code: |
-- Code update to handle Multi-byte characters
DECLARE intCCSID INTEGER InputRoot.Properties.CodedCharSetId;
DECLARE intEncoding INTEGER InputRoot.Properties.Encoding;
SET OutputRoot.Properties = InputRoot.Properties;
-- Create MQMD
CREATE LASTCHILD OF OutputRoot DOMAIN ('MQMD') NAME 'MQMD';
SET OutputRoot.MQMD.Version = MQMD_CURRENT_VERSION;
SET OutputRoot.MQMD.ApplIdentityData = SQL.BrokerName;
SET OutputRoot.MQMD.CodedCharSetId = intCCSID;
SET OutputRoot.MQMD.Encoding = intEncoding;
/*Create MQRFH2 header if it doesnt exist*/
IF NOT EXISTS(InputRoot.MQRFH2[]) THEN
SET OutputRoot.MQMD.Format = MQFMT_RF_HEADER_2; -- 'MQHRF2';
SET OutputRoot.MQRFH2.(MQRFH2.Field)Version = MQRFH_VERSION_2;
SET OutputRoot.MQRFH2.(MQRFH2.Field)CodedCharSetId = intCCSID;
SET OutputRoot.MQRFH2.(MQRFH2.Field)Encoding = intEncoding;
SET OutputRoot.MQRFH2.(MQRFH2.Field)Format = MQFMT_STRING;
SET OutputRoot.MQRFH2.(MQRFH2.Field)NameValueCCSID = 1208;
SET OutputRoot.MQRFH2.usr.OriginalHTTPInputHeader = InputRoot.HTTPInputHeader;
SET OutputRoot.MQRFH2.usr.MainMessageFlowName = MessageFlowLabel;
END IF; |
However this does not solves the issue and I am getting the same exception. I took trace before MQOutput node and the result shows with the values set.
| Code: |
##############################
TimeStamp : 2014-09-29 13:38:13.853544
( ['SOAPRoot' : 0xd02cac8]
(0x01000000:Name ):Properties = ( ['MQPROPERTYPARSER' : 0xaee20c0]
(0x03000000:NameValue):MessageSet = 'ESB_PO_OUT' (CHARACTER)
(0x03000000:NameValue):MessageType = '' (CHARACTER)
(0x03000000:NameValue):MessageFormat = '' (CHARACTER)
(0x03000000:NameValue):Encoding = 546 (INTEGER)
(0x03000000:NameValue):CodedCharSetId = 1208 (INTEGER)
(0x03000000:NameValue):Transactional = FALSE (BOOLEAN)
(0x03000000:NameValue):Persistence = FALSE (BOOLEAN)
(0x03000000:NameValue):CreationTime = GMTTIMESTAMP '2014-09-29 08:08:09.911' (GMTTIMESTAMP)
(0x03000000:NameValue):ExpirationTime = -1 (INTEGER)
(0x03000000:NameValue):Priority = 0 (INTEGER)
(0x03000000:NameValue):ReplyIdentifier = X'000000000000000000000000000000000000000000000000' (BLOB)
(0x03000000:NameValue):ReplyProtocol = 'SOAP-AXIS2' (CHARACTER)
(0x03000000:NameValue):Topic = NULL
(0x03000000:NameValue):ContentType = 'application/soap+xml;charset=UTF-8;action="urn:processDocument"' (CHARACTER)
(0x03000000:NameValue):IdentitySourceType = '' (CHARACTER)
(0x03000000:NameValue):IdentitySourceToken = '' (CHARACTER)
(0x03000000:NameValue):IdentitySourcePassword = '' (CHARACTER)
(0x03000000:NameValue):IdentitySourceIssuedBy = '' (CHARACTER)
(0x03000000:NameValue):IdentityMappedType = '' (CHARACTER)
(0x03000000:NameValue):IdentityMappedToken = '' (CHARACTER)
(0x03000000:NameValue):IdentityMappedPassword = '' (CHARACTER)
(0x03000000:NameValue):IdentityMappedIssuedBy = '' (CHARACTER)
)
(0x01000000:Name ):MQMD = ( ['MQHMD' : 0x25585c0]
(0x03000000:NameValue):Version = 2 (INTEGER)
(0x03000000:NameValue):ApplIdentityData = 'MBESB01' (CHARACTER)
(0x03000000:NameValue):CodedCharSetId = 1208 (INTEGER)
(0x03000000:NameValue):Encoding = 546 (INTEGER)
(0x03000000:NameValue):Format = 'MQHRF2 ' (CHARACTER)
(0x03000000:NameValue):Transactional = FALSE (BOOLEAN)
(0x03000000:NameValue):Persistence = 0 (INTEGER)
(0x03000000:NameValue):PutDate = DATE '2014-09-29' (DATE)
(0x03000000:NameValue):PutTime = GMTTIME '08:08:09.911' (GMTTIME)
(0x03000000:NameValue):Expiry = -1 (INTEGER)
(0x03000000:NameValue):Priority = 0 (INTEGER)
(0x03000000:NameValue):CorrelId = X'000000000000000000000000000000000000000000000000' (BLOB)
)
(0x01000000:Name ):MQRFH2 = ( ['MQHRF2' : 0x25829a0]
(0x03000000:NameValue):Version = 2 (INTEGER)
(0x03000000:NameValue):CodedCharSetId = 1208 (INTEGER)
(0x03000000:NameValue):Encoding = 546 (INTEGER)
(0x03000000:NameValue):Format = 'MQSTR ' (CHARACTER)
(0x03000000:NameValue):NameValueCCSID = 1208 (INTEGER)
(0x01000000:Name ):usr = (
(0x01000000:Name):OriginalHTTPInputHeader = (
(0x01000000:Name):X-Original-HTTP-Command = (
(0x02000000:Value): = 'POST http://localHost:7801/OUT/ESB_PO HTTP/1.1' (CHARACTER)
)
|
If we add the below line of code, after the one mentioned above, then the code works, i.e. able to put message to MQ queue with multi-byte characters.
| Code: |
-- Code update to handle Multi-byte characters
SET OutputRoot.Properties.CodedCharSetId = 1208;
|
I have no clue what change it is doing. This value was already set. Please help me to understand. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Sep 29, 2014 4:11 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
It is subtle but changing the value in the Properties folder does trip some kind of internal flag and forces the broker to reevaluate...
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
