| Author |
Message
|
| hari.krish085 |
 Posted: Wed Jul 14, 2010 12:18 am Post subject: RecoverableException BIP2136E: Source character ''4e0a'' |
 |
|
Novice
Joined: 13 Jul 2010
Posts: 12
|
Hi
My flow has a JMSInput Node --> ComputeNode --> JMSMQTransformation node --> ComputeNode and MQOutput Node
The JMS input message has Chinese character "件数" . While publishing the output xml, it tries to parse the xml and since it is not able to parse this Chinese characters, it throws an error.
I tried using code page 1208, 819, 437, in both properties and MQMD header of the output message. It didn't work.
Since my input was a JMSInput node.
The user trace is pasted below
| Code: |
RecoverableException BIP2136E: Source character ''4e0a'' in field ''003c002f006500730062003a0……………
cannot be converted from unicode to codepage '819'.
The source character is an invalid code point within the given codepage.
Correct the application or message flow that generated the message to ensure that all data within the message
can be represented in the target codepage. |
Any help on this would be much appreciated
Thanks,
Hari |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Wed Jul 14, 2010 12:43 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Both 437 & 819 Character Sets do not have the capability to handle the Chinese Character thus the errors your are seeing is very understandable.
1208 is UTF-8. This also can't OOTB handle Chinese
Ideally you should use UTD-16 or even UTF-32 but with these, you have to start knowing about Big-endian vs little-endian
There is a way to embed other UTF-encodings inside a UTF stream.
No doubt Mr Kimbert will be along soon with a link to how to do this.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Jul 14, 2010 12:45 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
also,
As Broker uses Unicode internally, you should be able to read the JMS message without issue and (from reading the error message), the data is already in Unicode (when it is inside JMS).
My question is, why are you trying to convert it on input?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| hari.krish085 |
| Posted: Wed Jul 14, 2010 1:09 am Post subject: |
|
|
Novice
Joined: 13 Jul 2010
Posts: 12
|
Thanks for your reply.
Here i am inserting the data in database (Colum) - These are the offending characters. Note the two Chinese characters at the end of the FOB_TITLE_PASS_DESC column. - like (Shanghai, 件数)
I’m not exactly sure what the Dump function does. It looks like a decimal representation of the FOB_TITLE_PASS_DESC column, but I don’t see the 4E0A values reported in the Message Broker Error.
here is the coding i done:
SET OutputRoot.MQRFH2 = NULL;
SET OutputRoot.MQMD.CodedCharSetID = 1208;
SET OutputRoot.MQMD.Encoding = 273;
SET OutputRoot.MQMD.Format = MQFMT_STRING
--------------------------------------
User Trace:
| Code: |
><discharge_port>SHTRU</discharge_port><discharge_port_desc>Shanghai Truck</discharge_port_desc>
<lading_port>SHA</lading_port><lading_port_desc>Shanghai</lading_port_desc>
<bill_to_id>INVC</bill_to_id>
<po_type>1000</po_type><po_type_desc>Default PO Type</po_type_desc>
<pre_mark_ind>N</pre_mark_ind>
<currency_code>CNY</currency_code>
<pickup_date><year>2010</year><month>08</month><day>02</day>
<hour>00</hour><minute>00</minute><second>00</second></pickup_date>
</PODesc>'''.
|
-----------------------
Please let me know if you need more info.
Thanks,
Hari. |
|
| Back to top |
|
|
| kimbert |
| Posted: Wed Jul 14, 2010 2:47 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
While publishing the output xml, it tries to parse the xml and since it is not able to parse this Chinese characters, it throws an error.
I tried using code page 1208, 819, 437, in both properties and MQMD header of the output message. It didn't work.
|
There is only one question that you should be asking. No other question matters until you have answered this one.
What is the *real* encoding of the XML?
The sender should know what encoding they used to construct the XML. Do not try to guess the encoding - that's dangerous, because one of your guesses might be wrong, but close enough to make your tests pass.
| Quote: |
| 1208 is UTF-8. This also can't OOTB handle Chinese |
It depends what you mean. UTF-8 can *represent* any character defined by Unicode. But it can't be used to *parse* data that was encoded in a Chinese code page ( or any other code page, for that matter). Data has to be parsed in the same encoding as was used to write it. |
|
| Back to top |
|
|
| hari.krish085 |
| Posted: Wed Jul 14, 2010 2:58 am Post subject: |
|
|
Novice
Joined: 13 Jul 2010
Posts: 12
|
Thanks for your reply.
What is the *real* encoding of the XML? --> the real encoding of xml is *UTF-8* .
so which encoding i can use for Chinese data parsing. |
|
| Back to top |
|
|
| kimbert |
| Posted: Wed Jul 14, 2010 3:31 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| What is the *real* encoding of the XML? --> the real encoding of xml is *UTF-8* |
If the incoming XML claims to be in UTF-8 then you should be able to parse it using UTF-8 ( CCSID 1208 ).
If you get parsing failures, the sending application has sent you a badly-formed XML document. There is nothing that your message flow can do about it - you need to tell the sending application to send good data. |
|
| Back to top |
|
|
| hari.krish085 |
| Posted: Wed Jul 14, 2010 3:34 am Post subject: |
|
|
Novice
Joined: 13 Jul 2010
Posts: 12
|
| thank you so much, i will check with them. |
|
| Back to top |
|
|
| hari.krish085 |
| Posted: Wed Jul 14, 2010 3:35 am Post subject: |
|
|
Novice
Joined: 13 Jul 2010
Posts: 12
|
| Is there is any other alternative we have on message flow side? |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jul 14, 2010 3:57 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| hari.krish085 wrote: |
| Is there is any other alternative we have on message flow side? |
So you're looking for a solution that enables you to parse badly formed XML documents??? 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jul 14, 2010 4:11 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| hari.krish085 wrote: |
| Is there is any other alternative we have on message flow side? |
 Have the sending application send you a valid XML document! and do not use the convert option on the input node... or use an output node with a CCSID that does not support all characters in the message... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| hari.krish085 |
| Posted: Wed Jul 14, 2010 4:14 am Post subject: |
|
|
Novice
Joined: 13 Jul 2010
Posts: 12
|
| Yes, I have checked with sending application side, they are sending valid XML document. there is no bad message from sending side. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jul 14, 2010 4:23 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| hari.krish085 wrote: |
| Yes, I have checked with sending application side, they are sending valid XML document. there is no bad message from sending side. |
If they're sending an XML document coded in UTF-8 which contains Chinese characters that can't be represented in that character set, how is the document valid?
Also, if it's a valid document why are you getting parsing errors and posting here??? 
_________________
Honesty is the best policy.
Insanity is the best defence.
Last edited by Vitor on Wed Jul 14, 2010 4:24 am; edited 1 time in total |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jul 14, 2010 4:24 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| hari.krish085 wrote: |
| Yes, I have checked with sending application side, they are sending valid XML document. there is no bad message from sending side. |
Don't take their word for it. How have you checked that their word is true?
Now you need to investigate the message just before it hits the broker. Is it still well formed XML? Did anything on the way change the CCSID of the message?
Does the broker in any of the handling previous to this point change the CCSID of the message?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| hari.krish085 |
| Posted: Wed Jul 14, 2010 4:36 am Post subject: |
|
|
Novice
Joined: 13 Jul 2010
Posts: 12
|
Sending application is RIB and RIB is publishing the JMS messages to topics, and Message Broker flow is starting with JMSInput Node ->Compute Node -> JMSMQTranfer node and Compute node and MQoutput node.
No idea on message side we can change the CCSID. |
|
| Back to top |
|
|
|
|
