| Author |
Message
|
| abcnil |
 Posted: Wed Apr 04, 2012 11:36 pm Post subject: URGENT : Some Sp Characters issue in WMQ/WMB |
 |
|
Apprentice
Joined: 29 Mar 2012
Posts: 36
|
I am facing two issue with WMQ and WMB:
Our messages are XML messages with UTF-8
1. Some Sp Characters are getting lost/converted on MQ
Like some non english characters like
Any Bullet Points
n°25
Protégé
Participações
Employees'
When our one system sends messages over MQ and we get it in another system , these characters are getting change into other characters.
Our Broker has CCSID 819.
We tried sending such UTF-8 messages over MQ, but getting transformed characters.
Like for Protégé we are getting Prot?g?
I believe this happens because of mismatching in encoding.
Can anyone please let me know what could be the exact reason and how to tackle it.
2. Our message is XML message. with UTF-8 and we have to transform it in other XML. When we process it we get following error.
<Text>An invalid XML character (Unicode: 0x1c) was found in the element content of the document.</Text>
we tried with CDATA also but getting same sort of error:
<Text>An invalid XML character (Unicode: 0x1c) was found in CDATA Section of document.</Text>
I know that these are Invalid chars for XML message and should not be in XML message. But our requirement is that we have to have sent them to downstream.
How this can be handle in WMB?
I am new to both MQ and WMB.
Thanks in advance. |
|
| Back to top |
|
 |
| mapa |
| Posted: Thu Apr 05, 2012 12:36 am Post subject: |
|
|
Master
Joined: 09 Aug 2001
Posts: 257
Location: Malmö, Sweden
|
1. What matters is the CCSID for the payload of your MQ messages, not the XML declaration encoding. It should be 1208 for UTF-8 and should be set by the sending application.
2. It is an illegal character in XML 1.0 so it is not possible to parse it as such. If I remember correctly it is allowed in 1.1 if you escape it.
|
|
| Back to top |
|
|
| Esa |
| Posted: Thu Apr 05, 2012 12:41 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
I think there are at least two ways to solve the problem:
- change your broker to run CCSDI 1208
- parse the message to BLOB with CCSID 1208 in the flow andadjust Properties.CodedCharSetId and MQMD.CodedCharSetId respectively
Somebody may be able to suggest a better solution. My brain is already on Easter vacation. |
|
| Back to top |
|
|
| Esa |
| Posted: Thu Apr 05, 2012 12:45 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| Actually, I think you should first try to capture the message put out by the flow, with RFHUtil for example. Check if the message and MDMD.CCSID value are correct. If they are, the receiving application should get the message with GMO.convert turned off. |
|
| Back to top |
|
|
| abcnil |
| Posted: Thu Apr 05, 2012 1:06 am Post subject: |
|
|
Apprentice
Joined: 29 Mar 2012
Posts: 36
|
| Quote: |
| 1. What matters is the CCSID for the payload of your MQ messages, not the XML declaration encoding. It should be 1208 for UTF-8 and should be set by the sending application. |
We tried using it. We have set the CCSID 1208 in the sending application. But getting the same issue.
| Quote: |
| change your broker to run CCSDI 1208 |
We cant change it coz broker is used by many other applications
[/quote] |
|
| Back to top |
|
|
| mapa |
| Posted: Thu Apr 05, 2012 1:15 am Post subject: |
|
|
Master
Joined: 09 Aug 2001
Posts: 257
Location: Malmö, Sweden
|
| Esa wrote: |
- change your broker to run CCSDI 1208
|
Why?
Unless the sender "stamps" the payload with the correct CCSID you will always run into these issues sooner or later. At least for non-ascii character sets.
Relying on the Qmgrs CCSID should always be avoided in my opinion. |
|
| Back to top |
|
|
| mapa |
| Posted: Thu Apr 05, 2012 1:21 am Post subject: |
|
|
Master
Joined: 09 Aug 2001
Posts: 257
Location: Malmö, Sweden
|
| abcnil wrote: |
We tried using it. We have set the CCSID 1208 in the sending application. But getting the same issue.
|
Are you sure it is set correctly? You are doing something wrong obviously.
If you do a user trace, what is the CCSID in the properties tree, both for your inbound and outbound? A common mistake is when you have a RFH2 header and set the MQMD CCSID to 1208 (that only points to the fixed length portion of the RFH2 header) and have an incorrect value for the payload in the RFH2 CCSID.
If the above is correct, inspect the message on the target queuemanager, before the receiving application process it. It could be that the receving application is doing something wrong and garbles it there.
Make sure you don't do channel conversions between queuemanagers (is anyone using that these days)?
Then there is always the classic that when looking at the received message you don't have a font that can display UTF-8 properly...  |
|
| Back to top |
|
|
| Esa |
| Posted: Thu Apr 05, 2012 1:23 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
The broker processes messages internally in UCS-2. When the message is serialized by MQOutput node, it is converted to the CCSID of the underlying queue manager. If my Easter brain remembers correctly. |
|
| Back to top |
|
|
| mapa |
| Posted: Thu Apr 05, 2012 1:28 am Post subject: |
|
|
Master
Joined: 09 Aug 2001
Posts: 257
Location: Malmö, Sweden
|
|
| Back to top |
|
|
| Esa |
| Posted: Thu Apr 05, 2012 1:29 am Post subject: Re: URGENT : Some Sp Characters issue in WMQ/WMB |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| abcnil wrote: |
We tried sending such UTF-8 messages over MQ, but getting transformed characters.
|
You are a bit unclear about which application is getting transformed characters. Mapa is working under the assumption that the broker gets the transformed characters, and I understood that it is the target application.
If it is the broker, make sure that the MQInput node does not have convert option selected. |
|
| Back to top |
|
|
| mapa |
| Posted: Thu Apr 05, 2012 1:35 am Post subject: |
|
|
Master
Joined: 09 Aug 2001
Posts: 257
Location: Malmö, Sweden
|
| Esa wrote: |
The broker processes messages internally in UCS-2. When the message is serialized by MQOutput node, it is converted to the CCSID of the underlying queue manager. If my Easter brain remembers correctly. |
I think your brain is on Easter then
It would mean that it is not possible to do character conversion in the broker...that the broker could only produce messages with same CCSID as the underlying queue manager. |
|
| Back to top |
|
|
| abcnil |
| Posted: Thu Apr 05, 2012 1:50 am Post subject: |
|
|
Apprentice
Joined: 29 Mar 2012
Posts: 36
|
| Quote: |
| You are a bit unclear about which application is getting transformed characters. Mapa is working under the assumption that the broker gets the transformed characters, and I understood that it is the target application. |
See this is happening here
Application ABC sends messages to XYZ over mq
ABC sends XML message with data in some tag Protégé to MQ
but XYZ receives Prot?g?
Some chars are getting changed to other because of encoding while sending from ABC and the CCSID on QM.
on QM has CCSID as 819, but for xml parsing we are using CCSID as 1200(UTF-16) while sending from XYZ.
is this causing the issue?
even we tried with 819 from XYZ but getting same issue. |
|
| Back to top |
|
|
| mapa |
| Posted: Thu Apr 05, 2012 2:00 am Post subject: |
|
|
Master
Joined: 09 Aug 2001
Posts: 257
Location: Malmö, Sweden
|
Now I get confused, are you using the broker at all or is this pure MQ to MQ?
The problem is that your application ABC must set the correct CCSID of the payload, whatever it is. What is it? Don't mind the queue manager CCSID, that doesn't matter if you set it correctly on the message itself.
Then you look at the message on the queue before it is processed by XYZ and check that it is correct. That is how I would do it. |
|
| Back to top |
|
|
| Esa |
| Posted: Thu Apr 05, 2012 4:22 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| mapa wrote: |
I think your brain is on Easter then
|
So true. And there are only character sets in my easter eggs  |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Apr 05, 2012 4:42 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mapa wrote: |
The problem is that your application ABC must set the correct CCSID of the payload, whatever it is. What is it? Don't mind the queue manager CCSID, that doesn't matter if you set it correctly on the message itself.
Then you look at the message on the queue before it is processed by XYZ and check that it is correct. That is how I would do it. |
This is true if you're using WMB or not.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
