| Author |
Message
|
| CharyChary |
 Posted: Sun Jan 24, 2016 7:46 am Post subject: Special characters in IDOC |
 |
|
Newbie
Joined: 24 Jan 2016
Posts: 5
|
Hi,
I'm receiving an IDOC (over SAPInput node) that has some special characters. For eg., a material number field has a non-breaking space at the end (decimal value = 160) and description/address field has some special ascii characters such as lower case e with acute (decimal value 130). What is the best way to parse this message from SAP? Once parsed, the target message we are generating throws an error in DFDL that it couldn't be converted using encoding 1252 although Windows 1252 includes those characters. Any help in resolving this would be greatly appreciated.
-- Chary |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Sun Jan 24, 2016 8:06 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Are you sure that what you get is in CCSID 1252? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| CharyChary |
| Posted: Sun Jan 24, 2016 8:24 am Post subject: |
|
|
Newbie
Joined: 24 Jan 2016
Posts: 5
|
| I do not know source encoding. But I do know that the record in SAP was created by someone in Mexico. Is there a way to check source encoding by looking at the IDOC message received on a queue? |
|
| Back to top |
|
|
| CharyChary |
| Posted: Sun Jan 24, 2016 8:35 am Post subject: |
|
|
Newbie
Joined: 24 Jan 2016
Posts: 5
|
| OK, I just checked CCSID of the message on the queue. It's encoding is 437. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Jan 24, 2016 10:48 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| CharyChary wrote: |
| OK, I just checked CCSID of the message on the queue. It's encoding is 437. |
I doubt that the CCSID matches the content...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| maurito |
| Posted: Sun Jan 24, 2016 11:27 pm Post subject: |
|
|
Partisan
Joined: 17 Apr 2014
Posts: 358
|
try CCSID 819
ISO 8859-1: Latin Alphabet Number 1 Latin-1 countries and regions |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Jan 25, 2016 12:42 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| maurito wrote: |
try CCSID 819
ISO 8859-1: Latin Alphabet Number 1 Latin-1 countries and regions |
8859-1 DOES not contain the Euro symbol which could be a problem (or not)
Really better to go with 1208 if possible.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| maurito |
| Posted: Mon Jan 25, 2016 1:06 am Post subject: |
|
|
Partisan
Joined: 17 Apr 2014
Posts: 358
|
| smdavies99 wrote: |
| maurito wrote: |
try CCSID 819
ISO 8859-1: Latin Alphabet Number 1 Latin-1 countries and regions |
8859-1 DOES not contain the Euro symbol which could be a problem (or not)
Really better to go with 1208 if possible. |
Well, the OP needs to be careful here. If he needs the euro symbol, which probably does not as in Mexico do not use euros, has to work out what representation it has.
in CCSID 1208 it is x'20AC' but in ASCII it is x'A4', so probably will need CCSID 923 which is a superset of 819 and contains the euro. |
|
| Back to top |
|
|
| timber |
| Posted: Mon Jan 25, 2016 1:50 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
You *may* get lucky by trying out various CCSIDs until you find one that makes all of your tests pass. But you *may* find that it breaks later on, when SAP output a character that your tests did not include. You should always specify the CCSID that was used by the application that wrote the document ( SAP in this case ). Guessing is a fragile strategy.
If possible,
- find a way to make SAP write the IDOC in UTF-8 ( CCSID 1208 ). I'm not a SAP expert, so I cannot advise on how to do that.
- specify 1208 in your message flow / DFDL schema. |
|
| Back to top |
|
|
| maurito |
| Posted: Mon Jan 25, 2016 1:55 am Post subject: |
|
|
Partisan
Joined: 17 Apr 2014
Posts: 358
|
| timber wrote: |
You *may* get lucky by trying out various CCSIDs until you find one that makes all of your tests pass. But you *may* find that it breaks later on, when SAP output a character that your tests did not include. You should always specify the CCSID that was used by the application that wrote the document ( SAP in this case ). Guessing is a fragile strategy.
If possible,
- find a way to make SAP write the IDOC in UTF-8 ( CCSID 1208 ). I'm not a SAP expert, so I cannot advise on how to do that.
- specify 1208 in your message flow / DFDL schema. |
I agree with that. |
|
| Back to top |
|
|
| CharyChary |
| Posted: Mon Jan 25, 2016 6:20 am Post subject: |
|
|
Newbie
Joined: 24 Jan 2016
Posts: 5
|
| Thanks everyone that replied to this. Source CCSID 437 is wrong. 437 was the default QM CCSID that was used to put message on the queue. I have to find what encoding SAP adapter in broker is using. Regardless of what the source encoding is, I think the best way to address this is to convert source to UTF-8 (on the broker side) and send it to target either in UTF-8 or another encoding that the target application can handle. |
|
| Back to top |
|
|
| timber |
| Posted: Mon Jan 25, 2016 8:58 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
| Quote: |
| I have to find what encoding SAP adapter in broker is using. |
Yes, definitely.
| Quote: |
| Regardless of what the source encoding is, I think the best way to address this is to convert source to UTF-8 (on the broker side) and send it to target either in UTF-8 or another encoding that the target application can handle. |
Possibly true. It depends on where the problem is arising.
If you are 100% sure that the message flow is *reading* the SAP IDOC using the correct CCSID then that strategy is OK. If you have even the tiniest sliver of a doubt in your mind, then take steps to remove that doubt. Because, if the message tree contains the wrong characters then no amount of Unicode goodness on the output side will fix that! |
|
| Back to top |
|
|
|
|
