| Author |
Message
|
| ghoshly |
 Posted: Thu May 07, 2009 11:19 am Post subject: CCSID problem to handle '£' character |
 |
|
Partisan
Joined: 10 Jan 2008
Posts: 333
|
Message broker is throwing exception when parsing a message with data containing '£' symbol in the data. XML parsing error ('Invalid character (Unicode: 0x15) ') is returned when we tried with the CCSID (InputRoot.MQMD.CodedCharSetId) 819.
Message Broker receives message from an external application through Queue manager communication where the application puts message to its own Queue Manager using UTF-8 format.
Both the queue manager is in AIX (MQ v6) and Coded character set ID property is set as 819.
We have tried changing the CCSID value of both the queue manager to 1208 and re bounced the applications to effect the change, however no joy with this.. 
Is this would be the problem with Encoding as well ?
Please help. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu May 07, 2009 11:37 am Post subject: Re: CCSID problem to handle '£' character |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ghoshly wrote: |
| Message broker is throwing exception when parsing a message with data containing '£' symbol in the data. XML parsing error ('Invalid character (Unicode: 0x15) ') is returned when we tried with the CCSID (InputRoot.MQMD.CodedCharSetId) 819. |
Does code page 819 have that symbol in it?
| ghoshly wrote: |
| We have tried changing the CCSID value of both the queue manager to 1208 and re bounced the applications to effect the change, however no joy with this.. |
Does code page 1208 have that symbol in it?
Changing the CCSID of the queue manager will not affect the CCSID in the MQMD.
Broker will use UTF-16 internally no matter what you do. Select a code page which can support that symbol, and output in that.
| ghoshly wrote: |
| Is this would be the problem with Encoding as well ? |
Encoding only affects packed numeric data.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| ghoshly |
| Posted: Thu May 07, 2009 11:47 am Post subject: |
|
|
Partisan
Joined: 10 Jan 2008
Posts: 333
|
Vitor,
Could you please help me to find on how I could get the correct code page for '£' symbol , any clue where to hit. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu May 07, 2009 12:07 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ghoshly wrote: |
| any clue where to hit. |
Mr Google is very helpful in these matters. 
You might be better served to speak initially to whoever administers your box. While a little Internet research will yield a multitude of possible code pages, WMQ can only use those pages installed on the server it's running on (and defaults to the server's default if you see what I mean). Hence you'll have to pick a code page from those available, or get the admin to install a better one.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| kimbert |
| Posted: Thu May 07, 2009 12:15 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| Message Broker receives message from an external application through Queue manager communication where the application puts message to its own Queue Manager using UTF-8 format. |
So we can assume that the MQMD of the incoming message contains MQMD.CodedCharSetId=1208.
Check: Is the data in the input message really in code page 1208, or has someone just picked UTF-8 because it sounded good? You could check that by parsing the message as a BLOB and using an ESQL CAST AS CHARACTER statement with CCSID 1208. If the resulting string looks valid, and there are no errors reported, the input message is probably OK.
| Quote: |
| Both the queue manager is in AIX (MQ v6) and Coded character set ID property is set as 819. |
The queue manager's CCSID should not be relevant. It is only used if the MQMD is set to some special value
Check: Is MQMD.CodedCharSetId set to 1208?
| Quote: |
| We have tried changing the CCSID value of both the queue manager to 1208 and re bounced the applications to effect the change, however no joy with this.. |
Wrong solution, as explained above.
| Quote: |
| Could you please help me to find on how I could get the correct code page for '£' symbol |
Even worse. You should *never* approach code page problems like this. The incoming data *is* in some code page. You need to find out what that code page is, and make sure that your flow is using that code page.
Check: Is the message flow using 1208 as its code page when parsing? Or have you been fiddling with the Convert properties on the input node?
Take a user trace to find out exactly which code page is being used. |
|
| Back to top |
|
|
| ghoshly |
| Posted: Fri May 08, 2009 2:03 am Post subject: |
|
|
Partisan
Joined: 10 Jan 2008
Posts: 333
|
Hi Kimbert,
Thanks for the details reply.
I have checked the message what is coming, MQMD.CodedCharSetId=819 there. I have tried the way you guided, i.e taking the message as BLOB and then CAST AS CHARACTER with InputRoot.MQMD.CodedCharSetId which in turns to 819 and it works there.
I am NOT fiddling with the Convert properties on the input node?
I have taken a user trace to find out exactly which code page is being used in the service processing and I find there also it is 819.
The same message I tried to put through RFHUtil changing the code page to 1208 and tried to parse it with 1208, it failed then saying "Unconvertable character".
Please help as I am getting confused which code page should be used. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri May 08, 2009 2:59 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| ghoshly wrote: |
The same message I tried to put through RFHUtil changing the code page to 1208 and tried to parse it with 1208, it failed then saying "Unconvertable character".
Please help as I am getting confused which code page should be used. |
Changing the code page to 1208 in RFHUtil will do nothing good if the content is not also in UTF-8. Check the content in hex view. If your pound sign shows up as A4 this is the value for it in CCSID 819...
So I believe from your experience that the assumption that your content was in CCSID 1208 has been wrong. As stated by InputRoot.Properties and InputRoot.MQMD your CCSID for the incoming message is 819 and it works fine when you use that.
If you need the outgoing message to be in UTF-8 just use
| Code: |
| set OutputRoot.Properties.CodedCharSetId = 1208; |
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Sun May 10, 2009 11:36 am Post subject: Re: CCSID problem to handle '£' character |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| ghoshly wrote: |
| Message Broker receives message from an external application through Queue manager communication where the application puts message to its own Queue Manager using UTF-8 format. |
The sending application needs to label the message as UTF-8, by setting the message header's ccsid=1208, when it puts the message on the queue, so that receiving application(s) will know how to read it.
| ghoshly wrote: |
| Message broker is throwing exception when parsing a message with data containing '£' symbol in the data. XML parsing error ('Invalid character (Unicode: 0x15) ') is returned ... |
Is there any evidence to suggest that the '£' character is causing the XML parsing error? If the '£' character is removed from the message, does the XML parsing error go away?
It actually looks like some other character is causing the error. The '£' character (0xA3, for both Unicode and ISO 8859-1) is a valid XML character, but the <NAK> control character (0x15, for both Unicode and ISO 8859-1) is not a valid XML character, and it would cause an XML parsing error.
| ghoshly wrote: |
| ...when we tried with the CCSID (InputRoot.MQMD.CodedCharSetId) 819. |
Who is 'we'? The sender? The receiver? How was this tried? The message flow ignores the MQInput.CodedCharSetId property, unless the MQInput.Convert flag is checked.
| ghoshly wrote: |
| We have tried changing the CCSID value of both the queue manager to 1208 |
When a message is passed from qmgr to qmgr, MQ uses the qmgr ccsid to convert the character fields in the MQMD header, but not necessarily the message that follows it. |
|
| Back to top |
|
|
| ghoshly |
| Posted: Fri May 15, 2009 11:03 am Post subject: |
|
|
Partisan
Joined: 10 Jan 2008
Posts: 333
|
By 'we' I tried to mention within WMB , i.e the receiver in this case.
rekarm01,
You might be correct for the control character which says 'Invalid character (Unicode: 0x15) ' but definitely the '£' symbol is causing problem as well.
What our investigation says is, UTF needs 2 bytes to represent these type of special character.But in the message probably we are getting single byte for this.So, even if tried to parse the message taking as BLOB and then CAST as CHARACTER and that looks good, need to double check on whether the input message is alright or not.
Thanks for helping.. |
|
| Back to top |
|
|
| kimbert |
| Posted: Mon May 18, 2009 9:55 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| UTF needs 2 bytes to represent these type of special character.But in the message probably we are getting single byte for this |
If that is true, then your sending application is sending junk. |
|
| Back to top |
|
|
|
|
