Author |
Message
|
akhnukh |
Posted: Sat Nov 06, 2010 10:35 pm Post subject: Message flow has problem with Arabic character |
|
|
Newbie
Joined: 16 Oct 2008 Posts: 9
|
Hi,
Platform :
OS : AIX
Message Broker : 6.1.0.8
Message Flow Design:
SOAP Input --> Extract --> Java Compute-1 --> Mapping --> SubFlow Consuming WebService --> Mapping --> Java Compute-2 --> SOAP Reply.
Message Flow Usage:
Message flow is exposed as a webservice which takes String data(English/Arabic) covert it to element in "Java Compute-1" node and using Mapping node it invokes subflow(which consume webservice). Then based on reply "Java Computer-2" process the response and reply using SOAP reply node.
Problem :
We are getting "Internal Server Error 500" from external webservice when we tried to consume in our flow with following.
* Error happens only with Arabic Data (If String contain Arabic character MEEM we will get error). Other than that most of Arabic characters are going fine.
* English data is going perfectly fine with no issue.
We have noticed that problematic part is Java Computer Node-1 which is doing conversion from String Data to Element using “CreateElementAsLastChildFromBitStream(stringData.getBytes(“UTF-8”),”XMLNSC”,”messageName”,”messageSetName”,””/null,0,0,0)” method.
We have tried different values for String Encoding and CCSID, Arabic data is only going fine(no question marks, no other error except MEEM) when we use above function call.
Please suggest possible problem cause and respective solution.
Thanks |
|
Back to top |
|
 |
fjb_saper |
Posted: Sat Nov 06, 2010 11:12 pm Post subject: |
|
|
 Grand High Poobah
Joined: 18 Nov 2003 Posts: 20756 Location: LI,NY
|
You are only having problems because of the incorrect use of the code you are doing:
Code: |
outMsgRootEl.createElementAsLastChildFromBitstream(responseBodyXmlData,
parserName, messageType, messageSet, messageFormat, encoding, ccsid,
options); |
Where I understand that you might be passing an empty string for messageType, messageSet, and messageFormat (unless you need validation), why are you passing 0 for encoding and ccsid ? Do you understand the effect / consequences of those values?
Values for encoding and ccsid should come from InputRoot.Properties
This way there is no need to translate your input with stringData.getBytes("UTF-8"), which may or may not be correct depending on each call.
Have fun  _________________ MQ & Broker admin |
|
Back to top |
|
 |
akhnukh |
Posted: Sat Nov 06, 2010 11:56 pm Post subject: |
|
|
Newbie
Joined: 16 Oct 2008 Posts: 9
|
Hi Fjb_saper,
Thanks for your reply.
No I am not passing empty string for message name and type. I am providing values as defined in message set. It was just method call template.
As mentioned earlier i have tried different values for CCSID and encoding and all has different outcome.
Working combination was using getBytes("UTF-8") and 0 in CCSID help to pass most of arabic characters except meem.
Please find below the tests we have done.
UTF-8 in getBytes -- 1208 in CCSID Unconvertable character
UTF-16 in getBytes -- 1200 in CCSID Unconvertable character
UTF-16 in getBytes -- 0 in CCSID An invalid XML character (Unicode: 0xfe) was found ....
empty in getBytes -- 0 in CCSID Arabic characters are going as question marks .
UTF-8 in getBytes -- 0 in CCSID Arabice Characters are passing correctly EXCEPT one character MEEM
If String contains MEEM we are getting Internal server error. |
|
Back to top |
|
 |
Vitor |
Posted: Sun Nov 07, 2010 4:53 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
akhnukh wrote: |
As mentioned earlier i have tried different values for CCSID and encoding and all has different outcome.
Working combination was using getBytes("UTF-8") and 0 in CCSID help to pass most of arabic characters except meem.
Please find below the tests we have done.
|
As my most worthy associate pointed out, you shouldn't be "testing" to find the correct values.
You should be using those associated with the message via it's Properties on the InputRoot. Unless you have a really, really, really good reason for setting them manually. Which I'm guessing doesn't apply here as it doesn't work with them set manually. _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
mqjeff |
Posted: Sun Nov 07, 2010 6:32 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 17447
|
What do you mean "doing conversion from string data to element"?
You only need to use createElementAsLastChildFromBitstream if you have a string that represents a complex message that you have received from something like a database lookup or from a manual file read operation.
If you merely want to set the contents of an element to a string value, you merely using a setter on the relevant MbElement object. |
|
Back to top |
|
 |
fjb_saper |
Posted: Sun Nov 07, 2010 6:57 am Post subject: |
|
|
 Grand High Poobah
Joined: 18 Nov 2003 Posts: 20756 Location: LI,NY
|
akhnukh wrote: |
Hi Fjb_saper,
Thanks for your reply.
No I am not passing empty string for message name and type. I am providing values as defined in message set. It was just method call template.
As mentioned earlier i have tried different values for CCSID and encoding and all has different outcome.
Working combination was using getBytes("UTF-8") and 0 in CCSID help to pass most of arabic characters except meem.
Please find below the tests we have done.
UTF-8 in getBytes -- 1208 in CCSID Unconvertable character |
probably due to MEEM character. Are you sure the content is UTF-8? Did you try just using the byte array (no getBytes function and using InputRoot.Properties.CodedCharSetId?
akhnukh wrote: |
UTF-16 in getBytes -- 1200 in CCSID Unconvertable character
UTF-16 in getBytes -- 0 in CCSID An invalid XML character (Unicode: 0xfe) was found ....
empty in getBytes -- 0 in CCSID Arabic characters are going as question marks . |
As expected. CCSID should contain the value from InputRoot.Properties.CodedCharSetId...
akhnukh wrote: |
UTF-8 in getBytes -- 0 in CCSID Arabice Characters are passing correctly EXCEPT one character MEEM
If String contains MEEM we are getting Internal server error. |
_________________ MQ & Broker admin |
|
Back to top |
|
 |
smdavies99 |
Posted: Sun Nov 07, 2010 8:18 am Post subject: |
|
|
 Jedi Council
Joined: 10 Feb 2003 Posts: 6076 Location: Somewhere over the Rainbow this side of Never-never land.
|
If you go here http://www.fileformat.info/info/unicode/char/fee2/index.htm. Is this the character you are talking about?
If you scroll down you will see its UTF-8 Representation.
What is the HEX values of your input stream for your problem character?
Are you sure that the input stream is a correctly formed UTF-8 Stream?
Can you post a hex dump here? There are people here who see this sort of problem on an almost daily basis. _________________ WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
Back to top |
|
 |
rekarm01 |
Posted: Sun Nov 07, 2010 4:24 pm Post subject: Re: Message flow has problem with Arabic character |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 1415
|
akhnukh wrote: |
Message flow is exposed as a webservice which takes String data (English/Arabic) ... |
String data from where? What is the actual string that's causing the error?
akhnukh wrote: |
... convert it to element in "Java Compute-1" node |
Which element?
akhnukh wrote: |
* Error happens only with Arabic Data (If String contain Arabic character MEEM we will get error). |
Which Arabic character MEEM? Unicode has 10 of them:U+0645 ARABIC LETTER MEEM
U+06D8 ARABIC SMALL HIGH MEEM INITIAL FORM
U+06E2 ARABIC SMALL HIGH MEEM ISOLATED FORM
U+06ED ARABIC SMALL LOW MEEM
U+0765 ARABIC LETTER MEEM WITH DOT ABOVE
U+0766 ARABIC LETTER MEEM WITH DOT BELOW
U+FEE1 ARABIC LETTER MEEM ISOLATED FORM
U+FEE2 ARABIC LETTER MEEM FINAL FORM
U+FEE3 ARABIC LETTER MEEM INITIAL FORM
U+FEE4 ARABIC LETTER MEEM MEDIAL FORM ... plus 159 more Arabic ligatures.
akhnukh wrote: |
We have noticed that problematic part is Java Computer Node-1 which is doing conversion from String Data to Element using “CreateElementAsLastChildFromBitStream(stringData.getBytes(“UTF-8”),”XMLNSC”,”messageName”,”messageSetName”,””/null,0,0,0)” method. |
Don't paraphrase; post the actual relevant code. As Last Child of which element? What is stringData?
createElementAsLastChildFromBitstream() is useful for discarding the input body, and replacing it with a completely different output body. Is that really what the Java Compute node is supposed to be doing?
akhnukh wrote: |
We have tried different values for String Encoding and CCSID |
The ccsid argument should match the getBytes() argument. For example:1200: UTF-16BE
1202: UTF-16LE
1204: UTF-16
1208: UTF-8 But that's not enough; OutputProperties.CodedCharSetID should match them as well.
akhnukh wrote: |
UTF-8 in getBytes -- 1208 in CCSID Unconvertable character
UTF-16 in getBytes -- 1200 in CCSID Unconvertable character
UTF-16 in getBytes -- 0 in CCSID An invalid XML character (Unicode: 0xfe) was found .... |
Which unconvertable character? To/From which codepage? For ccsid=0, what is the qmgr ccsid? For zero-arg getBytes(), what is the default charset? Unicode: 0xfe is a valid XML character. Is that a typo?
Don't paraphrase; post the actual error message. (Except if it looks something like:
Quote: |
BIP2136: Source character <insert_1> in field <insert_2> cannot be converted from Unicode to codepage <insert_3>. |
... and <insert_2> is excessively long, please snip it to include just the relevant substring, or at least line-wrap before posting.)
Run a usertrace with Trace node, to confirm the request message is correct before the external web service. If the message is correct, and the external web service still returns a 500 Internal Server Error, then the problem lies with the external web service, not the message flow. |
|
Back to top |
|
 |
akhnukh |
Posted: Mon Nov 08, 2010 2:56 am Post subject: |
|
|
Newbie
Joined: 16 Oct 2008 Posts: 9
|
Hi ,
Thanks everybody for replying. You guys have initiated lot of points on which i am looking as well.
Meanwhile in order to recreate same problem I have created a messageflow project which is having same functionality and it might help fellow members to understand my message flow. You can download PI from following
https://docs.google.com/leaf?id=0Bx8U0h1aRUFyMWEwNTRlYmYtOGVkZS00OWE3LTkzMzQtZGZiMDRmNWE0NjI2&hl=en&authkey=CPyi2L0C
Message Broker ToolKit : 6.1
Message flow Usage:
* Exposed as a webservice to client
* Receive XML data in the form of String.
* Coverts that string XML Data to Element so that element can be used further in mapping node.
* Invokes external webservice using mapped data.
* Reply to caller.
External webserive is very simple just printing inputs and it was hosted on WAS.
You can find different inputs which i have tried in messageset project folder called as inputData. You will find working Arabic and English data. As well as Arabic non working in different files.
I have test message flow using webservice explorer.
Thanks |
|
Back to top |
|
 |
|