| Author |
Message
|
| limal_raja |
 Posted: Mon May 07, 2012 7:55 am Post subject: To remove all EBCDIC chaaracters from the message |
 |
|
Novice
Joined: 11 Nov 2009
Posts: 19
|
We have a service where we are getting back some special characters in the response mainframe is sending to our brokers. We are getting some parsing exception and we started putting a ESQL replace statements:
| Code: |
SET "ABC"[I].Efghiklmnop.Memo = REPLACE("ABC"[I].Efghiklmnop.Memo,'¨' , ' ');
SET "ABC"[I].Efghiklmnop.Memo = REPLACE("ABC"[I].Efghiklmnop.Memo,'¦' , ' ');
SET "ABC"[I].Efghiklmnop.Memo = REPLACE("ABC"[I].Efghiklmnop.Memo,'°' , ' ');
SET "ABC"[I].Efghiklmnop.Memo = REPLACE("ABC"[I].Efghiklmnop.Memo,'æ' , ' ');
SET "ABC"[I].Efghiklmnop.Memo = REPLACE("ABC"[I].Efghiklmnop.Memo,'Í' , ' '); |
But now the list is growing on and on and everytime there is a defect assigned. Is there a way where we can get rid of all the EPCDIC special characters we get in the response mainframe is sending us.
The mainframe team does not want to handle this, and ha asked us to come up with a solution to clear/replace the special characters. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Mon May 07, 2012 8:01 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
first you have to make sure that you know that all the special characters are indeed "bad" characters, and that you should not instead be processing them or converting them into "good" characters.
Then you can do a replace on a copy of the input, that removes everything that is known to be a GOOD character. Then you can use the string that's left over as the set of bad characters for another replace. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon May 07, 2012 8:05 am Post subject: Re: To remove all EBCDIC chaaracters from the message |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| limal_raja wrote: |
| We are getting some parsing exception |
What parsing exception? Post details.
What platform is the WMB running on?
| limal_raja wrote: |
| The mainframe team does not want to handle this, and ha asked us to come up with a solution to clear/replace the special characters. |
WMB speaks EBCDIC and handles special characters. Unless you've got some kind of bizarre intermediate step that's trying to do the conversion into ASCII & making a mesh of it rather than letting WMB do it. Or mis-setting the CCSID so WMB uses the wrong code page for conversion.
Certainly I don't see "|" or "@" as a "special" character that WMB could choke on. It's not alphanumeric to be sure, but special characters to me indicates unprintable or control charatcers.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| limal_raja |
| Posted: Mon May 07, 2012 8:12 am Post subject: |
|
|
Novice
Joined: 11 Nov 2009
Posts: 19
|
The errors we are seeing is something like below:
| Code: |
See the following messages for details of the error.
2012-04-04 11:11:16.511131 6452 ParserException BIP5010E: XML Writing Errors have occurred.
Errors have occurred during writing of XML.
Review further error messages for an indication to the cause of the errors.
2012-04-04 11:11:16.511360 6452 RecoverableException BIP2136E: Source character ''8000'' in field ''3......3e00'' cannot be converted from unicode to codepage '437'.
The source character is an invalid code point within the given codepage. |
We are running MB version 6.0 on windows in lower regions and AIX in production |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon May 07, 2012 8:15 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Do not set the convert option on the MQInput node. Let the broker handle conversions and not MQ.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| limal_raja |
| Posted: Mon May 07, 2012 8:15 am Post subject: |
|
|
Novice
Joined: 11 Nov 2009
Posts: 19
|
No we are not doing any modifications to CCSID or encoding.
| Code: |
SET OutputRoot.MQMD."Encoding" = MQENC_NATIVE;
SET OutputRoot.MQMD.CodedCharSetId = MQCCSI_Q_MGR; |
And no, none of the convert options are selected in the MQInput node. they are all blank. [/img] |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon May 07, 2012 8:30 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| limal_raja wrote: |
No we are not doing any modifications to CCSID or encoding.
| Code: |
SET OutputRoot.MQMD."Encoding" = MQENC_NATIVE;
SET OutputRoot.MQMD.CodedCharSetId = MQCCSI_Q_MGR; |
And no, none of the convert options are selected in the MQInput node. they are all blank. [/img] |
Well this is your problem right away. The qmgr's CCSID may not support the content of your message. Set the CCSID to UTF-8 (ccsid 1208).
See what happens then...
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| limal_raja |
| Posted: Mon May 07, 2012 8:33 am Post subject: |
|
|
Novice
Joined: 11 Nov 2009
Posts: 19
|
We have tried that also. It didnt help:
| Code: |
SET OutputRoot.Properties.MessageFormat = 'XML';
SET OutputRoot.MQMD."Encoding" = MQENC_NATIVE;
--SET OutputRoot.MQMD.CodedCharSetId = MQCCSI_Q_MGR;
SET OutputRoot.Properties.CodedCharSetId = 1208; |
|
|
| Back to top |
|
|
| Vitor |
| Posted: Mon May 07, 2012 9:04 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| limal_raja wrote: |
We have tried that also. It didnt help:
| Code: |
SET OutputRoot.Properties.MessageFormat = 'XML';
SET OutputRoot.MQMD."Encoding" = MQENC_NATIVE;
--SET OutputRoot.MQMD.CodedCharSetId = MQCCSI_Q_MGR;
SET OutputRoot.Properties.CodedCharSetId = 1208; |
|
None of this will help. Setting the OutputRoot properties is not going to affect how WMB is handling the inbound message. Granted the problem doesn't manifest until you try and write the data out but that's on demand parsing for you.
How does the flow receive this data? You talk about a service and my most prize winning associate has assumed an WMQ message but what is it? Does it contain an embedded CCSID (in an MQMD or similar)? Does it match the embedded data?
More specifically what hex bytes are being recieved by WMB? How are they handled?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| limal_raja |
| Posted: Mon May 07, 2012 10:11 am Post subject: |
|
|
Novice
Joined: 11 Nov 2009
Posts: 19
|
The flow recieves the data from mainframe in a blob(CWF format) and we convert it into a XML.
Actually the layout is something like Aggregated Request flow --> Request flow --> Mainframe --> Response flow --> Aggregated response flow.
The error is happening on the Aggregated response flow. So whatever is coming from the mainframe we are able to put it into a XML and send it out but the problem is when the other flow reads it. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon May 07, 2012 10:20 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| limal_raja wrote: |
| The flow recieves the data from mainframe in a blob(CWF format) and we convert it into a XML. |
How? WMQ message? File? Paper Airplane?
Why BLOB? Why not MRM? More specifically if you're handling it as a BLOB how are you parsing it to produce an XML document?
| limal_raja wrote: |
| The error is happening on the Aggregated response flow. So whatever is coming from the mainframe we are able to put it into a XML and send it out but the problem is when the other flow reads it. |
This doesn't make sense. The error you posted is an XML writing error, i.e. when an XML document is being written. So how can you say that you are writing the document fine, but reading it is an issue? Is this why you keep changing OutputRoot to try and fix it?
I wonder if you're "writing" an XML document that has an unparsed blob of mainframe data in it, and you think it's a problem reading the XML because the first flow to actually try and make sense of the mainframe data is going boom?
Alternatively you can keep doing what you're doing & apply the band aid my most worthy associate suggested earlier to replace the REPLACE statements. Much easier than actually fixing the problem & you can have confidence it will work. Until something else breaks loose.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| kimbert |
| Posted: Mon May 07, 2012 11:34 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
I think we need to revisit all of the assumptions here.
| Quote: |
| We are getting some parsing exception and we started putting a ESQL replace statements: |
Sounds like fixing the symptoms instead of curing the disease. I guess that's what you are starting to think as well.
| Code: |
2012-04-04 11:11:16.511131 6452 ParserException BIP5010E: XML Writing Errors have occurred.
Errors have occurred during writing of XML.
Review further error messages for an indication to the cause of the errors.
2012-04-04 11:11:16.511360 6452 RecoverableException BIP2136E: Source character ''8000'' in field ''3......3e00'' cannot be converted from unicode to codepage '437'.
The source character is an invalid code point within the given codepage. |
So you are attempting to write an XML message in code page 437. Code page 437 is a single-byte encoding - it can represent exactly 256 different Unicode characters. Any character not in that set will be not be mappable to code page 437. That is what the error message is saying.
Sounds as if you have a Chinese character somewhere in the message tree, and the XML writer is ( quite correctly ) complaining about it.
| Quote: |
| We have tried that ( changing the output encoding to 1208) also. It didnt help: |
Wrong. Setting OutputRoot.Properties.CodedCharSetId=1208 will prevent that BIP5010E from happening. If you do not understand why, read this: http://www.joelonsoftware.com/articles/Unicode.html
Either your method was wrong, or your reporting of the results was not accurate ( maybe you got a different error message when you tried with 1208? ).
| Quote: |
| The flow recieves the data from mainframe in a blob(CWF format) and we convert it into a XML. |
So you are parsing the input message using the MRM parser with a CWF physical format. Two questions:
1. Did you expect to find the character 0x8000 in the message tree? I didn't know that EBCDIC was able to represent Chinese characters.
2. Are you sure that the encoding is set correctly when you parse the incoming COBOL data? Maybe the 0x8000 character is a wrongly decoded EBCDIC character. Maybe some or all of the other 'EBCDIC special' characters arise from the same problem? |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon May 07, 2012 11:53 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| kimbert wrote: |
| I didn't know that EBCDIC was able to represent Chinese characters. |
Good catch! It can't but EBCDIC DBCS can. Assuming one is the same as the other is a receipe for the sort of disaster we see here.
| kimbert wrote: |
| 2. Are you sure that the encoding is set correctly when you parse the incoming COBOL data? Maybe the 0x8000 character is a wrongly decoded EBCDIC character. Maybe some or all of the other 'EBCDIC special' characters arise from the same problem? |
Again, setting an EBCDIC code page on a DBCS message would do this.
When I grow up, I want to be half as smart as you. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon May 07, 2012 11:55 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| limal_raja wrote: |
The flow recieves the data from mainframe in a blob(CWF format) and we convert it into a XML.
Actually the layout is something like Aggregated Request flow --> Request flow --> Mainframe --> Response flow --> Aggregated response flow.
The error is happening on the Aggregated response flow. So whatever is coming from the mainframe we are able to put it into a XML and send it out but the problem is when the other flow reads it. |
I suspect the problem is when you send out the response from the MF.
Make that output UTF-8 This should fix all the problems due to "there is no corresponding character in 437 so I'll just go and put a substitution char" (the value of which I believe is prohibitted in XML
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Wed May 09, 2012 2:47 am Post subject: Re: To remove all EBCDIC chaaracters from the message |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| kimbert wrote: |
| 1. Did you expect to find the character 0x8000 in the message tree? I didn't know that EBCDIC was able to represent Chinese characters. |
That's more than likely little-endian (0x8000 = U+0080 = <XXX>). See also the last two bytes of the XML message (0x3e00 = U+003e = '>').
| limal_raja wrote: |
Actually the layout is something like Aggregated Request flow --> Request flow --> Mainframe --> Response flow --> Aggregated response flow.
The error is happening on the Aggregated response flow. So whatever is coming from the mainframe we are able to put it into a XML and send it out but the problem is when the other flow reads it. |
No, the error might be detected in the Aggregated response flow, but is most likely happening much before then. It's difficult to determine what's going on exactly, without a lot more detail about what each application is sending (message headers and message bytes), but it seems like one of the applications is mixing up EBCDIC, ASCII, or non-character bytes into a single string, and expecting the next application to make sense of it. |
|
| Back to top |
|
|
|
|
