| Author |
Message
|
| mvs |
 Posted: Tue Jul 21, 2009 11:44 am Post subject: Special character convertion |
 |
|
Voyager
Joined: 06 Jul 2007
Posts: 85
|
Hi,
We have scenario
There are 2 source applications (separate qmgr for each application)sending french and Canadian characters to the broker and broker maps the required fileds and sends to the target queue manager.
source qmgr's CCSID is 819 and broker and target application qmgr CCSID is 1208.
One source application data works fine without any problem.
2nd source application also sends the french/canadian data but its fails at the target application.
one application data before sending to the broker :COMMUNAUTÉ URBAINE DE MONTRÉAL
2nd application data before sending to the broker : 2006 chemin de la canardiéré
The first one process fine and 2nd fails at the target application. The data at target applicaiton looks like below
2006 chemin de la canardir.
each application uses the separate flow for mapping.
Flow1 Target flow
Flow2
Flow1 output goes to the target flow and Flow2 output also goes to the same target flow. The output of TargetFlow goes to the target application queue.
I am not sure why the 2nd application messages failing.
Help is greatly appreciated. |
|
| Back to top |
|
 |
| jbanoop |
| Posted: Tue Jul 21, 2009 12:13 pm Post subject: |
|
|
Chevalier
Joined: 17 Sep 2005
Posts: 401
Location: SC
|
| maybe the target CCSID does not have an equivalent representation for those "special" characters ? |
|
| Back to top |
|
|
| mvs |
| Posted: Tue Jul 21, 2009 12:21 pm Post subject: |
|
|
Voyager
Joined: 06 Jul 2007
Posts: 85
|
| you mean to say target Queue manager CCSID does not have equivalent representation for those "special" characters? |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Jul 21, 2009 12:35 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
This very topic has been discussed recently in this forum.
PErhaps a search might help you resolve your problem.
Also, it does help to mention the things you have tried in your post so that we can take those into consideration before attempting to help you with your problem.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| mvs |
| Posted: Tue Jul 21, 2009 1:01 pm Post subject: |
|
|
Voyager
Joined: 06 Jul 2007
Posts: 85
|
I searched the previous dicussion and found the unique code requires CCSID to 1208 to accept special characters.
But in my case, the broker and Target application queue managers both's CCSID is 1208.
I have tried couple of things, like changing the CCSID of the target application qmgr to 437. becuase it's running on windows. And selected the Cluster Reciver channel Convert property to yes but the result is same.
I didn't find any difference after doing this. |
|
| Back to top |
|
|
| kimbert |
| Posted: Tue Jul 21, 2009 2:00 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Jul 21, 2009 7:34 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| mvs wrote: |
I searched the previous dicussion and found the unique code requires CCSID to 1208 to accept special characters.
But in my case, the broker and Target application queue managers both's CCSID is 1208.
I have tried couple of things, like changing the CCSID of the target application qmgr to 437. becuase it's running on windows. And selected the Cluster Reciver channel Convert property to yes but the result is same.
I didn't find any difference after doing this. |
a) make sure you do not have the convert flag set on the input node
b) set OutputRoot.Properties.CodedCharSetID=1208;
This will probably help you over the hurdle. You still need to do as Kimbert suggested. Worst case scenario the source system may need to send the message in CCSID 1208...
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Mon Jul 27, 2009 12:12 am Post subject: Re: Special character conversion |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| mvs wrote: |
| source qmgr's CCSID is 819 and broker and target application qmgr CCSID is 1208. |
The message header CCSID is what's relevant. The qmgr CCSID is only relevant when it's used (as a default) to populate a message header CCSID.
| mvs wrote: |
| 2nd source application also sends the french/canadian data but its fails at the target application. |
More often than not, the fault lies with the sender, not the receiver. Especially if the 1st source application works, but the 2nd source application does not. The 2nd source application should be the most obvious place to start looking for problems.
The sender physically encodes the source string as a sequence of bytes, before it ever puts the string on a queue. The sender provides the message header CCSID, that's supposed to describe the physical encoding. The sender needs to make sure that it does.
| mvs wrote: |
The first one process fine and 2nd fails at the target application. The data at target applicaiton looks like below
| Code: |
| 2006 chemin de la canardir |
|
It's often more useful to look at a hex dump of the string, in addition to the string itself. Due to variations in web browsers, the string may not look the same after posting, as it did before posting.
It appears that the 'é' character was converted to X'1a1a'. X'1a' is often used as a substitution character when converting, for source characters that don't exist in the target character set. Since the 'é' character does exist in the character sets for ccsid=437, ccsid=819, and ccsid=1208, the more likely explanation is that the sender specified the wrong CCSID, causing the receiver to mis-read the data.
| mvs wrote: |
| I have tried couple of things, like changing the CCSID of the target application qmgr to 437. because it's running on windows. |
Don't try random things. Two mistakes rarely cancel each other out, and it's probably worse if they did.
Try comparing the actual byte sequence with the expected byte sequence for a given ccsid. The IBM CDRA web site is a useful resource for mapping characters to hex. rfhutil is a good tool for browsing messages on a queue, and displaying message header contents, and bytes as hex; IBM provides a sample command-line program amqsbcg0, that does this too. usertrace is useful for examining message contents within a message flow.
Try looking beyond the target application.
| Quote: |
| And selected the Cluster Reciver channel Convert property to yes but the result is same. |
Don't use channel conversion, unless the receiving side doesn't support a required conversion. |
|
| Back to top |
|
|
| mvs |
| Posted: Mon Jul 27, 2009 11:25 am Post subject: |
|
|
Voyager
Joined: 06 Jul 2007
Posts: 85
|
All,
Thanks for the help. my problem got solved.
when they are sending message, the format of the message is none. that's the reason broker QMGR not converting CCSID from 819 to 1208 .
We requested the application team to send message format as MQSTR.
The second point, while sending xml message, they were sending UTF-8 as encoding. They changed from UTF-8 to ibm819. It got worked successfully
I just reffered to information center, ibm819 is alias of ISO-8859-1 which is European and American language converters. |
|
| Back to top |
|
|
| er_pankajgupta84 |
| Posted: Mon Jul 27, 2009 7:00 pm Post subject: |
|
|
Master
Joined: 14 Nov 2008
Posts: 203
Location: charlotte,NC, USA
|
Although your problem got solved..but i have one question for you.. don't you think if we are able to parse a message with ISO-8859-1 encoding then it should also get parsed with UTF-8 as what i know is that ISo-8859-1 is a subset of UTF-8.
Any comments. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Jul 27, 2009 8:42 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| er_pankajgupta84 wrote: |
Although your problem got solved..but i have one question for you.. don't you think if we are able to parse a message with ISO-8859-1 encoding then it should also get parsed with UTF-8 as what i know is that ISo-8859-1 is a subset of UTF-8.
Any comments. |
Not necessarily. You are looking at parsing as if there was only XML parsing.
You do also have positional parsing (MRM). This can be a problem when the parsing is set to bytes instead of characters for the specified text field.
You may need to verify the attribute for each text field declared in the message set.
For instance if you have a 2 byte representation for a character in UTF-8 that has a 1 byte representation in the source and the MRM parser is set to byte and not character, using the MRM parser on the same message in UTF-8 could result in a shift by 1 byte of your data in the message set....
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Tue Jul 28, 2009 12:16 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
| er_pankajgupta84 wrote: |
| don't you think if we are able to parse a message with ISO-8859-1 encoding then it should also get parsed with UTF-8 |
No it wont. look at the how the extended ASCII characters (eg the pound symbol) are physically stored between the different character sets. |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Jul 28, 2009 12:34 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
|
| Back to top |
|
|
|
|
