| Author |
Message
|
| fazz |
 Posted: Wed Nov 07, 2007 3:45 am Post subject: SAP WBI Adaptor & Double byte chars |
 |
|
Centurion
Joined: 20 Feb 2004
Posts: 144
Location: England
|
Hello,
I am having a problem with code page conversion on messages coming from SAP using the WBI Adaptor.
It seems SAP uses Unicode and is sending double byte characters.
It also appears that the Adaptor is using ASCII 7 bit which is a single byte code page.
Therefore messages that contain double byte characters are being converted incorrectly.
There is a property in the Adaptors configuration that sets which encoding type to use, the default is ASCII7.
I have tried changing this to ASCII8 but it still fails. There doesn't seem to be an option for selecting Unicode.
Has anyone come across this problem before?
Cheers
Simon |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Wed Nov 07, 2007 3:10 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
There is an option in the SAP configuration to tell it that the target system is unicode (see jco documentation) you need to use it.
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| fazz |
| Posted: Thu Nov 08, 2007 4:37 am Post subject: |
|
|
Centurion
Joined: 20 Feb 2004
Posts: 144
Location: England
|
The SAP developers have confirmed that the target system is set to Unicode, and that SAP is sending Unicode.
As the adaptor does not seem to be Unicode, should this setting be changed to be non-Unicode? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Nov 08, 2007 3:08 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
There is an option in the SAP adapter configuration to tell it that the target SAP system is unicode (see jco documentation) you need to use it.
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| fazz |
| Posted: Fri Nov 09, 2007 2:19 am Post subject: |
|
|
Centurion
Joined: 20 Feb 2004
Posts: 144
Location: England
|
And this option is set.
The unicode test for WBICON in SM59 is run and you get the following message "Target is a unicode system (char.size 2)"
I have tried adding UTF-8 to the connectors config file. But it still outputs one character (an s with a comma above it) as HEX 1A. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Nov 09, 2007 3:22 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| fazz wrote: |
And this option is set.
The unicode test for WBICON in SM59 is run and you get the following message "Target is a unicode system (char.size 2)"
I have tried adding UTF-8 to the connectors config file. But it still outputs one character (an s with a comma above it) as HEX 1A. |
The option just governs the communication between the adapter and SAP.
It has little bearing on the adapter's output to say a file... and will not change the "CCSID". I retrieve the CCSID from the adapter and put that on the XML generated...
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| fazz |
| Posted: Fri Nov 09, 2007 4:27 am Post subject: |
|
|
Centurion
Joined: 20 Feb 2004
Posts: 144
Location: England
|
The adaptor is writing the XML to an MQ queue.
The RFH2 header specifies the CCSID as 1208, but the character is still being converted incorrectly. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Nov 09, 2007 1:44 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| fazz wrote: |
The adaptor is writing the XML to an MQ queue.
The RFH2 header specifies the CCSID as 1208, but the character is still being converted incorrectly. |
Maybe not. This depends on how you retrieve the MQ message. If you are dealing with special characters but retrieve the message in a CCSID that does not support them.... Tough luck.
What you can do is set 1208 as CCSID on your qcf or qmgr object and thus get the converted text in UTF-8....
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| fazz |
| Posted: Mon Nov 12, 2007 3:20 am Post subject: |
|
|
Centurion
Joined: 20 Feb 2004
Posts: 144
Location: England
|
I am viewing the message using Visual Edit, with the 'Convert on get' option unchecked. So I should see the data in 1208, which I would expect to be able to display this character.
I have narrowed the 'bad' character down to code page 852. See http://en.wikipedia.org/wiki/Code_page_852
After some more testing, all characters that have a three or more digit unicode number fail, all those with a two digit unicode number work ok. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Nov 12, 2007 11:53 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| fazz wrote: |
| I am viewing the message using Visual Edit, with the 'Convert on get' option unchecked. So I should see the data in 1208, which I would expect to be able to display this character. |
Huge assumption! I am not sure it is true. Myself I wrote a little browser program in java JMS with a display pane. This is the only way I could truely see the UTF-8 "special" characters as they were meant to display....
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| fazz |
| Posted: Tue Nov 13, 2007 1:03 am Post subject: |
|
|
Centurion
Joined: 20 Feb 2004
Posts: 144
Location: England
|
I think viewing the data is a side issue anyway.
The fact is that the these characters with a three or more digit unicode number get converted to hex 1A, so when the broker tries to parse the XML it fails. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Nov 13, 2007 10:41 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| fazz wrote: |
I think viewing the data is a side issue anyway.
The fact is that the these characters with a three or more digit unicode number get converted to hex 1A, so when the broker tries to parse the XML it fails. |
So why not send the message in Unicode (CCSID 1200) to the broker??
Make sure to manually specify the flow output CCSID to make it unicode too.
Had a problem where everything was fine but the MQOutput node was failing.
Tracked it down to unsupported characters in the output message CCSID.
Force the CCSID to the right value(utf-8 or unicode) and the flow ran fine.
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| fazz |
| Posted: Wed Nov 14, 2007 1:17 am Post subject: |
|
|
Centurion
Joined: 20 Feb 2004
Posts: 144
Location: England
|
Thats the problem I have, as far as I can tell the adaptor is sending the message in Unicode to the broker. But the message still contains a HEX 1A before it even hits the broker.
 thanks for all your help by the way. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Nov 14, 2007 1:40 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| fazz wrote: |
Thats the problem I have, as far as I can tell the adaptor is sending the message in Unicode to the broker. But the message still contains a HEX 1A before it even hits the broker.
thanks for all your help by the way. |
Have you opened a PMR about that? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| fazz |
| Posted: Wed Nov 14, 2007 1:48 am Post subject: |
|
|
Centurion
Joined: 20 Feb 2004
Posts: 144
Location: England
|
Yes PMR just opened.
I will post up any outcome. |
|
| Back to top |
|
|
|
|
