| Author |
Message
|
| ghoshly |
 Posted: Mon Dec 02, 2013 7:54 am Post subject: Message Broker capability to handle multiple CCSID together |
 |
|
Partisan
Joined: 10 Jan 2008
Posts: 333
|
Message Broker - V8.0.0.2, AIX
Hello,
We have created some generic flows in WMB which receives SOAP messages from different end systems, lookup profile details from Oracle 11g database, performs necessary transformation using XSLT, generic flow to dispatch messages to different web service destinations. Monitoring events are generated from various points of flows and captured for audit and error handling purpose.
Our system works fine with UTF-8 character set and we can pass messages without any conversion even with multi-byte characters.
Question: How can we gurrantee that, through generic Message Broker flows, we would be able to handle all different codepages.
Can we pass character set data to XSLT while processing (through localenvironment or some other means)
I am expecting lot of questions to answer in order to describe scenario if required. |
|
| Back to top |
|
 |
| ghoshly |
| Posted: Mon Dec 02, 2013 7:58 am Post subject: |
|
|
Partisan
Joined: 10 Jan 2008
Posts: 333
|
| If required, please move this post to Message Broker support community. By mistake I posted to this group. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Dec 02, 2013 8:00 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Moved as requested
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Dec 02, 2013 8:03 am Post subject: Re: Message Broker capability to handle multiple CCSID toget |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ghoshly wrote: |
| Question: How can we gurrantee that, through generic Message Broker flows, we would be able to handle all different codepages. |
WMB works in UTF-16 internally. It uses whatever CCSID data is available from the inbound message (e.g. the CCSID value of the MQMD, the code page of the HTTP header, etc) to parse the inbound data. Hence so long as the inbound data correctly describes the CCSID you're good.
| ghoshly wrote: |
| Can we pass character set data to XSLT while processing (through localenvironment or some other means) |
I've never used XSLT inside WMB so will defer to others on this.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Dec 02, 2013 8:24 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
AFAIK, the XSLT support in Broker is limited to V1 only. Pretty much transformation olny and no enrichment.
Also from my limited experience it is not exactly highly efficient. I'm going to be doing a benchmark in the early part of 2014 on a system where XSLT's are used everywhere in preference to Mapping Nodes or ESQL. It will be interesting to see the results.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| ghoshly |
| Posted: Mon Dec 02, 2013 9:51 am Post subject: No preference to XSLT |
|
|
Partisan
Joined: 10 Jan 2008
Posts: 333
|
Even I have no preference or choice for XSLT, but there are other reasons XSLT is used in the name of ease of deployment etc.
I am trying to put message using SOAP UI tool using UTF-16BE, but while putting the message to MQ queue, getting exception as
| Code: |
RecoverableException
File:CHARACTER:F:\build\slot1\S800_P\src\DataFlowEngine\ImbMqOutputNode.cpp
Line:INTEGER:876
Function:CHARACTER:ImbMqOutputNode::evaluate
Type:CHARACTER:ComIbmMQOutputNode
Name:CHARACTER:EI0050_TMCGetIRMessage_Receiver#FCMComposite_1_7.ICE_Receiver_SubFlow#FCMComposite_1_5.ICE_Invoke_ProfileLoader_SubFlow#FCMComposite_1_2
Label:CHARACTER:EI0050_TMCGetIRMessage_Receiver.ICE.PROFILELOADER.IN
Catalog:CHARACTER:BIPmsgs
Severity:INTEGER:3
Number:INTEGER:2230
Text:CHARACTER:Caught exception and rethrowing
RecoverableException
File:CHARACTER:F:\build\slot1\S800_P\src\CommonServices\ImbConverter.cpp
Line:INTEGER:355
Function:CHARACTER:ImbConverterCPP::internalFromUnicode
Type:CHARACTER:
Name:CHARACTER:
Label:CHARACTER:
Catalog:CHARACTER:BIPmsgs
Severity:INTEGER:3
Number:INTEGER:2133
Text:CHARACTER:String too long to convert
Insert
Type:INTEGER:5
Text:CHARACTER:5200460048002000
Insert
Type:INTEGER:2
Text:CHARACTER:4
Insert
Type:INTEGER:2
Text:CHARACTER:1204 |
Last edited by ghoshly on Mon Dec 02, 2013 9:55 am; edited 1 time in total |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Dec 02, 2013 9:52 am Post subject: Re: No preference to XSLT |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ghoshly wrote: |
| Even I have no preference or choice for XSLT, but there are other reasons XSLT is used in the name of ease of deployment etc. |
Let's hope you can pass CCSID info to it then.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| ghoshly |
| Posted: Tue Dec 03, 2013 12:17 am Post subject: Unable to use UTF-16 |
|
|
Partisan
Joined: 10 Jan 2008
Posts: 333
|
CharacterSet CCSID
---------------- ---------------
UTF-16BE - 1200
UTF-16LE - 1202
UTF-16 - 1204
SOAP Input -> Compute -> MQ Output is used in the flow.
I am trying to use the above mentioned character set using SOAP UI tool (v 3.5), and simply trying to put the message in a MQ queue using the CCSID and Encoding from input. However I am getting "String too long to convert" exception.
Could you please help me to identify the point I am doing mistake. or SOAP UI is not correctly sending the data to WMB. As broker internally uses UTF-16, there should not be any conversion.
| Code: |
SET OutputRoot.Properties = InputRoot.Properties;
CREATE LASTCHILD OF OutputRoot DOMAIN 'MQMD';
SET OutputRoot.MQMD.CodedCharSetId = InputRoot.Properties.CodedCharSetId;
SET OutputRoot.MQMD.Encoding = InputRoot.Properties.Encoding;
SET OutputRoot.MQMD.Format = MQFMT_RF_HEADER_2;
CREATE NEXTSIBLING of OutputRoot.MQMD DOMAIN 'MQRFH2' NAME 'MQRFH2';
SET OutputRoot.MQRFH2.usr.DetailInfo = 'UTF-16';
CREATE LASTCHILD OF OutputRoot DOMAIN 'XMLNSC';
SET OutputRoot.XMLNSC = InputBody.Body; |
|
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Dec 03, 2013 12:43 am Post subject: Re: No preference to XSLT |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| ghoshly wrote: |
Even I have no preference or choice for XSLT, but there are other reasons XSLT is used in the name of ease of deployment etc.
|
 
I'm glad that I don't work for the same company as you then and have to support your environments. (just a personal rant you understand)
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Dec 03, 2013 5:34 am Post subject: Re: Unable to use UTF-16 |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| ghoshly wrote: |
CharacterSet CCSID
---------------- ---------------
UTF-16BE - 1200
UTF-16LE - 1202
UTF-16 - 1204
SOAP Input -> Compute -> MQ Output is used in the flow.
I am trying to use the above mentioned character set using SOAP UI tool (v 3.5), and simply trying to put the message in a MQ queue using the CCSID and Encoding from input. However I am getting "String too long to convert" exception.
Could you please help me to identify the point I am doing mistake. or SOAP UI is not correctly sending the data to WMB. As broker internally uses UTF-16, there should not be any conversion.
| Code: |
SET OutputRoot.Properties = InputRoot.Properties;
CREATE LASTCHILD OF OutputRoot DOMAIN 'MQMD';
SET OutputRoot.MQMD.CodedCharSetId = InputRoot.Properties.CodedCharSetId;
SET OutputRoot.MQMD.Encoding = InputRoot.Properties.Encoding;
SET OutputRoot.MQMD.Format = MQFMT_RF_HEADER_2;
CREATE NEXTSIBLING of OutputRoot.MQMD DOMAIN 'MQRFH2' NAME 'MQRFH2';
SET OutputRoot.MQRFH2.usr.DetailInfo = 'UTF-16';
CREATE LASTCHILD OF OutputRoot DOMAIN 'XMLNSC';
SET OutputRoot.XMLNSC = InputBody.Body; |
|
What is the CCSID of OutputRoot.Properties.CCSID ?
What is the format of OutputRoot.MQRFH2.Format ? (MQFMT_STRING?)
What is the CCSID of OutputRoot.MQRFH2.(MQRFH2.Field)CCSID ?
These things do matter...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| ghoshly |
| Posted: Tue Dec 03, 2013 6:53 am Post subject: |
|
|
Partisan
Joined: 10 Jan 2008
Posts: 333
|
Hi Saper,
Surely they matter. I have added the code
| Code: |
| SET OutputRoot.MQRFH2.Format = MQFMT_STRING; |
But even with this or without CCSID field within Output properties or MQMD or MQRFH2 is populated with the input value, i.e. 1200 or 1202 or 1204.
While trying to see deep, I feel the error is happening for the RFH2 folder. Could you please help me to identify the link of RFH2 header with UTF-16 character set? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Dec 03, 2013 11:23 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| ghoshly wrote: |
Hi Saper,
Surely they matter. I have added the code
| Code: |
| SET OutputRoot.MQRFH2.Format = MQFMT_STRING; |
But even with this or without CCSID field within Output properties or MQMD or MQRFH2 is populated with the input value, i.e. 1200 or 1202 or 1204.
While trying to see deep, I feel the error is happening for the RFH2 folder. Could you please help me to identify the link of RFH2 header with UTF-16 character set? |
You are of course aware of the restrictions on the CCSID used for the usr folder (name-value pairs). What did you set there? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Dec 03, 2013 11:32 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
If I have been given this task I would probably have looked at using a tool like rfhutil to generate a message with the correct MQMD and RFH2 Header.
Then I would have replicated it in ESQL
But that is just my way of doing things, sometimes arse about face.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| ghoshly |
| Posted: Wed Dec 04, 2013 3:18 am Post subject: usr folder in UTF-16 |
|
|
Partisan
Joined: 10 Jan 2008
Posts: 333
|
I tried with RFHUtil tool to put a message with codepage 1200 along with RFH2 header and field in usr folder in a windows Queue Manager and able to put message.
I was not aware about restriction in using usr folder. I am trying to follow the link below which says about surrogate pair. I believe usr folder and surrogate pair is different. Please correct me if I am wrong.
http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/topic/com.ibm.mq.csqzak.doc/fr22470_.htm
In this scenario, the sending application sending message using UTF-16 (CCSID - 1200), broker internally also uses UTF-16,
CCSID 1200 (UCS-2) and 1208(UTF- is supported by MQ as well. So why the conversion is required in the first place? Does it always tries to convert to Queue manager's default code page in order to put message in queue? |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Dec 04, 2013 4:00 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
MQ writes the message using one of the following
1) The CCSID specified by the application
OR if that is not done
2) The CCSID of the queue Manager.
Generally I don't rely on the QMGR CCSID being correct. Far too many sites use the default CCSID for their platform. This leads to no end of problems later on down the track especially if you are connecting qmgrs up. Just think of the problems you might encounter if one node in an MQ Cluster uses CSID 923 and another uses 1208. Been there, seen it and got the 'T-shirt'.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
|
|
