| Author |
Message
|
| wbi_telecom |
 Posted: Tue Oct 06, 2009 11:44 am Post subject: Unconvertable Character error |
 |
|
Disciple
Joined: 15 Feb 2006
Posts: 188
Location: Harrisburg, PA
|
Enviroment wmb 6.1.0.3 on Zlinux
We are using file nodes to get a flat file from mainframe and convert it into XML. This morning we got an error which was due to an Uncovertable Character in the file. We found that user typed "[" in a Text field because of which the code failed at the following line
| Code: |
| SET typeVar = CAST(inVar AS CHAR CCSID 1208); |
Where inVar is InputRoot.BLOB.BLOB;
I read a lot of posts about CCSID related issues but all of them were with MQ messages. In this case we are reading the messages from a file which gets created on mainframe and is FTPed on broker directory.
Is there a way we can preserve the "[" character in the XML when we change the message from EBCDIC to ASCII?
Cheers, |
|
| Back to top |
|
 |
| dk27 |
| Posted: Tue Oct 06, 2009 12:45 pm Post subject: |
|
|
Acolyte
Joined: 28 Apr 2008
Posts: 51
|
| You can try using asBitStream with CCSID option. Also catch any exception thrown, if invalid character is thrown then scan thru the message and either remove that character or replace it with some ASCII equivalent. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Oct 06, 2009 1:16 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I'm confused about why the file ever goes into ASCII in the first place.
Wouldn't it be simpler to just go from EBCDIC to Unicode in the first place? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Oct 06, 2009 7:57 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Are we sure this is actually a CCSID matter and not just an XML matter?
IIRC the "[" character has special meaning in XML like at the start of a Cdata section and such. You may not be allowed to use it in XML text without escaping it. Assigning the text to an element might have done that... however I have no idea how your message was built.
Anyways I'm sure you can look that up on google or somewhere and fix it.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Wed Oct 07, 2009 12:33 am Post subject: Re: Unconvertable Character error |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| wbi_telecom wrote: |
| ... This morning we got an error which was due to an Unconvertable Character in the file. We found that user typed "[" in a Text field ... |
What was the actual error message?
Run a usertrace. |
|
| Back to top |
|
|
| wbi_telecom |
| Posted: Wed Oct 07, 2009 4:07 am Post subject: |
|
|
Disciple
Joined: 15 Feb 2006
Posts: 188
Location: Harrisburg, PA
|
Thanks for all the replies and here is some more information.
As I said before the file contains flat messages based on copybook formats. We read the file for fixed length records and then parse each record using message set based on copybook. Before parsing we convert the record from BLOB to CHAR using CCSID 1208.
After conversion we use CREATE....PARSE to parse the message using message set and then build the output XML using esql.
This code has been running in Production for last 4 months without any issues. Yesterday a CICS user fat fingered a "[" instead of P in one of the fields which caused the CAST function to fail.
The actual error was "XMLUnconvertable 8d 1208" in the exceptionlist. I debugged the code and it was failing at the CAST using CCSID 1208 so I thought may be using a different CCSID will allow us to preserve "[". I tried using 819 and I did not get the exception but then the "[" was converted to another character. Although "[" was put in the message in error we would like to send the message exactly as entered by the user because of the auditing requirements.
I admit that I am not an expert in conversions involving CCSID and I am doing my research. I thought I would ask the question here so if I am doing some basic mistake, it will get pointed out early in the game.
Cheers, |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Oct 07, 2009 7:00 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| wbi_telecom wrote: |
| This code has been running in Production for last 4 months without any issues. Yesterday a CICS user fat fingered a "[" instead of P in one of the fields which caused the CAST function to fail. |
I trust you've mentioned to whoever supports the application the user was entering the data through that their validation sucks?
| wbi_telecom wrote: |
| The actual error was "XMLUnconvertable 8d 1208" in the exceptionlist. I debugged the code and it was failing at the CAST using CCSID 1208 so I thought may be using a different CCSID will allow us to preserve "[". I tried using 819 and I did not get the exception but then the "[" was converted to another character. |
You can't rely on characters between code pages. If you have the ability to change the CCSID of the message like this, follow the suggestion given above and use Unicode.
| wbi_telecom wrote: |
| Although "[" was put in the message in error we would like to send the message exactly as entered by the user because of the auditing requirements. |
If the message is in error (presumably wherever the message is going won't like a '[' where it expects a 'P') then why audit it?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Oct 07, 2009 7:31 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
And FJ's point about the XML requirements on the "[" remain.
If you really require that the users be able to enter this character and pass it through, you must either transform it into an entity encoding or you must wrap the field in a CData section. |
|
| Back to top |
|
|
| kimbert |
| Posted: Wed Oct 07, 2009 1:58 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| rekarm01 said:
I can't work out why you have not done that yet. It is the obvious next step. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Wed Oct 07, 2009 6:48 pm Post subject: Re: Unconvertable Character error |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| wbi_telecom wrote: |
| The actual error was "XMLUnconvertable 8d 1208" in the exceptionlist. |
No, that's a paraphrased error message; it omits important information such as the BIP error number, and any relevant inserts. The actual error message might look more like:
| Quote: |
| BIP2135E: Source character '????' in field ... cannot be converted to Unicode from codepage ??? |
A usertrace will provide the actual error message. Some inserts may consist of long BLOBs that don't line-wrap well. In that case, please add manual line-breaks as needed, (or, truncate wisely), before posting.
| wbi_telecom wrote: |
| Before parsing we convert the record from BLOB to CHAR using CCSID 1208. |
The message sender gets to choose the CCSID here; the message flow is not free to hard-code an arbitrary value. The only valid CCSID is the one that came with the message:
| Code: |
| SET typeVar = CAST(inVar AS CHAR CCSID InputRoot.Properties.CodedCharSetId); |
If that doesn't work, then the input message is corrupt, and there's very little that the message flow can do about it.
| wbi_telecom wrote: |
Yesterday a CICS user fat fingered a "[" instead of P in one of the fields which caused the CAST function to fail.
The actual error was "XMLUnconvertable 8d 1208" in the exceptionlist. |
If "8d" is hexadecimal for some character, it's probably not the "[" character; the CICS user's fingers may be fatter than it first seemed. |
|
| Back to top |
|
|
| wbi_telecom |
| Posted: Thu Oct 08, 2009 4:10 am Post subject: |
|
|
Disciple
Joined: 15 Feb 2006
Posts: 188
Location: Harrisburg, PA
|
Here's the excerpt from the user trace where the error occurs.
| Code: |
An error occurred when casting a value to a different data type. This may be because no conversions exist between the two data types or because the particular value was unsuitable.
Subsequent messages will indicate the context of the error.
2009-10-08 08:02:41.831480 52 RecoverableException BIP2135E: Source character ''8d'' in field ''44323030392d31302d30325432313a33373a30332e38363239
3233373939353239323337393935323932333739393530353139343933393
4313938382d31322d3135434f57454c4c20202020202020202020202020202
0202020202020202020202020202050656c6c202020323031302020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020303120202020
20323637352e3030323030392d30392d3038543031202020202020202020204d8d504c20202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020
20202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020'' cannot be converted to Unicode from codepage '1208'.
The source character is an invalid code point within the given codepage.
Correct the application that generated the message.
|
[edited for readability] |
|
| Back to top |
|
|
| kimbert |
| Posted: Thu Oct 08, 2009 4:50 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
Your CAST statement is wrong. This statement
| Code: |
| CAST char_field AS CHARACTER CCSID 1208 |
will convert char_field from UTF-8 to the broker's internal code page ( UTF-16 ).
You need to find out the actual code page of your character string, and specify that code page in the CCSID clause of the CAST.
If you want your output message to be in UTF-8, set OutputRoot.Properties.CodedCharSetId in the normal way before the message arrives at the output node.
| Quote: |
| Before parsing we convert the record from BLOB to CHAR using CCSID 1208 |
Why? The CWF or TDS parser can easily parse it from the raw bytes, as long as you specify the correct CCSID. |
|
| Back to top |
|
|
| wbi_telecom |
| Posted: Thu Oct 08, 2009 6:52 am Post subject: |
|
|
Disciple
Joined: 15 Feb 2006
Posts: 188
Location: Harrisburg, PA
|
Hi Kimbert,
We are reading 4 seprate formats from the file and the CAST was to determine which message set to use for CREATE and PARSE. I have removed the CAST for now and am directly using the CREATE..PARSE as follows
CREATE LASTCHILD OF OutputRoot DOMAIN('MRM') PARSE(hexVar ENCODING InputRoot.Properties.Encoding CCSID InputRoot.Properties.CodedCharSetId SET 'NB2KT4O002001' TYPE 'msg_CODSCOMMONRECORDSTUSCHL' FORMAT '');
Where hexVar is InputRoot.BLOB.BLOB
This does not throw the exception but the "[" in the input message transforms into "�"
The CCSID used for CAST was taken from InputRoot.Properties.CodedCharSetId field and 1208 is the value in that field.
Cheers, |
|
| Back to top |
|
|
| kimbert |
| Posted: Thu Oct 08, 2009 7:19 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
The CCSID used for CAST was taken from InputRoot.Properties.CodedCharSetId field and 1208 is the value in that field.
|
As others have said, if the input message is telling lies about its encoding there is nothing that you can do to correct that.
You need to find the people who maintain the upstream application and apply some 'pressure' to get them to fix it. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Thu Oct 08, 2009 4:06 pm Post subject: Horizontal scrolling is evil |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| rekarm01 wrote: |
| Some inserts may consist of long BLOBs that don't line-wrap well. In that case, please add manual line-breaks as needed, (or, truncate wisely), before posting. |
Again, some inserts may consist of long BLOBs that don't line-wrap well. In that case, please add manual line-breaks as needed, (or, truncate wisely), before posting.
The 'Preview' button is useful here, to preview posts before submitting.
The 'Edit' button is also useful, to fix posts after submitting. |
|
| Back to top |
|
|
|
|
