| Author |
Message
|
| satya2481 |
 Posted: Fri Jul 03, 2009 4:45 am Post subject: Special Characters (like € ” )issue - Pub/sub Flows |
 |
|
Disciple
Joined: 26 Apr 2007
Posts: 170
Location: Bengaluru
|
Hi All,
We are facing an issue with the special characters like - € and ” in the message flows where Publish/Subscribe concept is used. Please let me know if any one have faced this issue and how to resolve it.
There are 2 flows-
1. Publish Message Flow
2. Subscribe Message flow
Publish Message FlowThe message is sent from Source application with Code page 1208 and the same message is published on to a topic. The message sent from Source application has some special characters as indicated above.
Mesage gets published successfully.
Subscribe Message Flow
When this flow reads the published message from the queue its not able to parse the message. The flow throws parse exception indicating - Unconvertable character.
Our Broker is running in Windows platform.
When we check the MQMD and RFH header details in the Subscriber flow its having values like
MQMD.CCSID = 1208
MQRFH2.CCSID = 437 (Windows code page)
I suspect the difference between these values causing this issue.
We have tried below options.
> Instead of publishing the message flow sent the message to MQOutput node. The Subscriber flow works fine and the message gets processed correctly.
> Tried to use the MQInput node convert options in the Subscriber flow the flow again throws the exception
Please let me know how to fix this issue.
Thanks in advance.
Satya
_________________
IBM Certified Solution Developer WebSphere Message Broker V6.0
IBM Certified System Administrator WebSphere MQ V6.0 |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Fri Jul 03, 2009 10:56 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Do not use convert on the MQInput node. The convert option would try to convert from ccsid 1208 to ccsid 437 which is the default on your windows qmgr. With the MQInput node the broker the conversion will then take place anyways but from ccsid 1208 (message) to ccsid (1200) broker internal. Setting the convert flag adds a conversion to ccsid 437 in between. This will make you loose the special chars... 
When parsing with a positional set make sure to set the parsing to character and not byte if you are using a multi-byte charset like 1208....
And make sure to set the ccsid to 1208 in the RFH on publish and on the OutputRoot.Properties....
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| sarathmattam |
| Posted: Sat Jul 04, 2009 4:47 am Post subject: |
|
|
Voyager
Joined: 05 Sep 2008
Posts: 94
|
Hi,
I think in your subscriber flow, use BLOB in the MQ Input node followed by a RCD node to convert to the required format will solve you problem .. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Sat Jul 04, 2009 5:06 pm Post subject: Re: Special Characters (like € ” ) issue - Pub/sub Flows |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| satya2481 wrote: |
| We are facing an issue with the special characters like - € and ” in the message flows where Publish/Subscribe concept is used. Please let me know if any one have faced this issue and how to resolve it. |
If only there were a way to search this message forum for similar topics. 
| satya2481 wrote: |
Publish Message Flow
The message is sent from Source application with Code page 1208 and the same message is published on to a topic. The message sent from Source application has some special characters as indicated above.
Message gets published successfully.
Subscribe Message Flow
When this flow reads the published message from the queue its not able to parse the message. The flow throws parse exception indicating - Unconvertable character.
MQMD.CCSID = 1208
MQRFH2.CCSID = 437 (Windows code page) |
Posting the ccsids inside the Subscriber message flow is inadequate. What's needed is to narrow down where the ccsid changed from 1208 to 437. Working backwards, what was it:- before the Subscriber message flow?
- inside the Publish message flow?
- before the Publish message flow?
- inside the source application?
As a general rule, ccsid=437 is almost always wrong. It's too bad that it's the default choice for Windows QMgr ccsid.
More precisely, ccsid=437 refers to a DOS code page, not a Windows code page. Windows applications do not use it, natively; it is frequently a poor choice to convert to/from, as its character set doesn't include many commonly used characters, such as the '€' or '”' characters.
| satya2481 wrote: |
| I suspect the difference between these values causing this issue. |
The MQMD ccsid describes the character encoding of strings within the MQRFH2 header. The MQRFH2 ccsid describes the character encoding of strings within the message body. They describe different strings. They can be different.
| satya2481 wrote: |
| Please let me know how to fix this issue. |
Here's a start:
| fjb_saper wrote: |
Do not use convert on the MQInput node ...
And make sure to set the ccsid to 1208 in the RFH on publish and on the OutputRoot.Properties. |
If that doesn't resolve the issue, then please provide more detailed information.
[Edit: added emoticon]
Last edited by rekarm01 on Sun Jul 05, 2009 3:46 pm; edited 1 time in total |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Sun Jul 05, 2009 12:58 pm Post subject: Re: Special Characters (like € ” ) issue - Pub/sub Flows |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| fjb_saper wrote: |
| With the MQInput node the broker the conversion will then take place anyways but from ccsid 1208 (message) to ccsid (1200) broker internal. |
To be a bit pedantic, ccsids describe byte sequences, not character sequences. A ccsid identifies (among other things):- character set / code page pairs (to map characters <--> code points)
- an encoding scheme (to map code points <--> bytes)
The broker internal representation is a sequence of UCS-2 characters, with no specified encoding scheme, so technically, it doesn't have a ccsid:- bitstream --> tree: convert from byte(ccsid=InputRoot.Properties.CodedCharSetId) to character(UCS-2)
- tree --> bitstream: convert from character(UCS-2) to byte(ccsid=OutputRoot.Properties.CodedCharSetId)
|
|
| Back to top |
|
|
| Vitor |
| Posted: Sun Jul 05, 2009 1:07 pm Post subject: Re: Special Characters (like € ” ) issue - Pub/sub Flows |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| rekarm01 wrote: |
| If only there were a way to search this message forum for similar topics. |
It's on the top of our wish list of developments. The best we've come up with so far is the search facility top right of this page that allows people to search for similar topics......
(Those readers especially sensitive to sarcasm should use eye drops after reading this post......  )
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Sun Jul 05, 2009 3:42 pm Post subject: Re: Special Characters (like € ” ) issue - Pub/sub Flows |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| Vitor wrote: |
| rekarm01 wrote: |
| If only there were a way to search this message forum for similar topics. |
It's on the top of our wish list of developments. The best we've come up with so far is the search facility top right of this page that allows people to search for similar topics...... |
Oops. I forgot my emoticon. Let me re-edit ... |
|
| Back to top |
|
|
| satya2481 |
| Posted: Sun Jul 05, 2009 9:44 pm Post subject: |
|
|
Disciple
Joined: 26 Apr 2007
Posts: 170
Location: Bengaluru
|
I have tried below options, but still no luck.
> Remove convert option from the MQInput Node.
> Publish the message as it comes from the Source application by setting the RFH2.CCSID = 1208 and also in OutputRoot.Properties
> We are using XML Physical format
| Code: |
| Subscriber flow altered as MQInputNode (With BLOB domain) --> RCD Node |
Tried this option as well and debugged the message flow and getting "ImbRecoverableException caught from worker->parseNext" exception.
Checked the place where the CCSID gets changed and it is just after the Publish flow and Before Subscriber flow.
| Quote: |
| The MQMD ccsid describes the character encoding of strings within the MQRFH2 header. The MQRFH2 ccsid describes the character encoding of strings within the message body. They describe different strings. They can be different. |
Thanks for this information 
Am I missing anything here, please let me know if anyone of you would like to know more details in order to provide your suggessions.
Thanks to all of you for replying to my post... |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Jul 06, 2009 1:11 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| satya2481 wrote: |
I have tried below options, but still no luck.
> Remove convert option from the MQInput Node.
> Publish the message as it comes from the Source application by setting the RFH2.CCSID = 1208 and also in OutputRoot.Properties
> We are using XML Physical format |
Has anybody bothered to check the format of the incoming message?
Do we have MQFMT_STRING and CCSID=1208 on the incoming message?
If not what are the values? What is the value inside the message for the xml encoding qualifier? What is the content of the message, does it conform to CCSID 1208? Verify in hex mode with rfhutil...
| satya2481 wrote: |
| Code: |
| Subscriber flow altered as MQInputNode (With BLOB domain) --> RCD Node |
Tried this option as well and debugged the message flow and getting "ImbRecoverableException caught from worker->parseNext" exception.
Checked the place where the CCSID gets changed and it is just after the Publish flow and Before Subscriber flow.
| Quote: |
| The MQMD ccsid describes the character encoding of strings within the MQRFH2 header. The MQRFH2 ccsid describes the character encoding of strings within the message body. They describe different strings. They can be different. |
Thanks for this information
Am I missing anything here, please let me know if anyone of you would like to know more details in order to provide your suggessions.
Thanks to all of you for replying to my post... |
You are giving us a parse next exception but you do not specify the domain of the parser used... I'd expect it to be XMLNS or XMLNSC. If different please elaborate.
Before the subscription consumer/flow consumes the message, check the message on the queue (rfhutilc) What is the ccsid of the message on the queue? (MQMD-CCSID and MQRFH2-CCSID). What is the format of the message on the queue? Who is the consumer (java, java JMS, XMS, broker on which platform)? Again check the message in hex format to make sure the CCSID corresponds to the content....
This should help you determine if
- The message was broken before you ever got it
- it's definitely the broker (we will need version and fix pack for broker and MQ)
- it's the subscriber
- it's anything in between over which you may or may not have any control
So far it looks suspiciously like some text conversion. If you have any control over the input message you could ask it to be sent with format MQFMT_NONE and check if it makes a difference.
Also make sure none of the channels involved have automatic conversion set.
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Tue Jul 07, 2009 1:35 am Post subject: Re: Special Characters (like € ” ) issue - Pub/sub Flows |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| satya2481 wrote: |
| Code: |
| Subscriber flow altered as MQInputNode (With BLOB domain) --> RCD Node |
|
BLOB -> RCD is not really necessary.
| satya2481 wrote: |
| Tried this option as well and debugged the message flow and getting "ImbRecoverableException caught from worker->parseNext" exception. |
Don't use the debugger. Usertrace is much more useful, particularly with a Trace node before the Publication node in the publish flow, and after the input node in the subscriber flow, to display the message header contents.
Don't post the entire usertrace; try to find the relevant parts, and post that.
| satya2481 wrote: |
| The flow throws parse exception indicating - Unconvertable character. |
Don't paraphrase the exception. Provide the specific error number, and all relevant information. For example:
| Quote: |
| BIP2135E: Source character '????' in field ... cannot be converted to Unicode from codepage ??? |
If the exact exception itself is excessively long, truncate the irrelevant bits, or at least add line breaks before posting.
rfhutil + usertrace should narrow down where the message (or message header) corruption occurs.
| satya2481 wrote: |
| Checked the place where the CCSID gets changed and it is just after the Publish flow and Before Subscriber flow. |
If that's truly the case, try to disable any channel conversion between the two. |
|
| Back to top |
|
|
|
|
