| Author |
Message
|
| Pegazus |
 Posted: Tue Feb 17, 2009 8:15 am Post subject: Preserving NewLine Chars on conversion from EBCDIC to ASCI |
 |
|
Novice
Joined: 10 Aug 2006
Posts: 16
|
We are receiving data from the Mainframes in EBCDIC format and on conversion from EBCDIC to ASCII the new line characters seem to disappear.
Here is the Flow CICS -> MQ -> WMB (Convert EBCDIC to ASCII + BLOB to MRM) -> MRM to XML -> HTTP
We change the Enconding on the MQ Input node to convert from EBCDIC to CCSID 500. Then the mapping of the data seems to be fine from BLOB to MRM. But one of the field contains many New Line characters, which seems to be lost on conversion.
Can anyone help me here. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Tue Feb 17, 2009 8:23 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
I would theorise the EBCDIC has <CR> characters, while the ASCII platform uses <CRLF> for end of line.
Have you actually looked at the hex in the string, to confirm the characters are actually missing, or if it just looks like they are when you view in ASCII?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Pegazus |
| Posted: Tue Feb 17, 2009 8:26 am Post subject: |
|
|
Novice
Joined: 10 Aug 2006
Posts: 16
|
Thank you for your response. You are correct. Is there a way to convert CR to CRLF's
Is there a standard way of doing it ? |
|
| Back to top |
|
|
| Tibor |
| Posted: Tue Feb 17, 2009 8:34 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Feb 17, 2009 8:37 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Pegazus wrote: |
| Is there a standard way of doing it ? |
Yes - don't send them! 
Seriously, it's a problem. If you know that the input is pure character and contains no packed numbers then you can do a simple replace, but I notice you refer to BLOB in your original post. Obviously if you have packed decimal that could legitimately be x'0d' and you don't want that replaced.
Mind you, if you're moving packed numbers from z/OS to most ASCII platforms you've got other issues anyway!
One way (but often the hardest to implement) is for the z/OS to either tweak the message to contain <CRLF> or better still use a different and unlikely combination as an end-of-line marker. In this way you reduce the odds of hitting a combination you don't really want to replace.
The question you need to ask is do you actually need a <CRLF>, or do you just need a way of finding the end of a line in an application? The <CR> is a perfectly valid marker for that, even if it's not what most ASCII file viewers are expecting.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Tue Feb 17, 2009 7:52 pm Post subject: Re: Preserving NewLine Chars on conversion from EBCDIC to AS |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| Pegazus wrote: |
| We are receiving data from the Mainframes in EBCDIC format and on conversion from EBCDIC to ASCII the new line characters seem to disappear. |
There is a one-to-one mapping of control characters between EBCDIC character sets and ISO-8 ("ASCII") character sets:
| Code: |
ISO-8 EBCDIC
----- ------
(LF/line feed): X'0a' <--> X'25' (LF/line feed)
(CR/carriage return): X'0d' <--> X'0d' (CR/carriage return)
(NEL/next line): X'85' <--> X'15' (NL/new line) |
Exactly which combination of these characters constitutes an "end-of-line" delimiter might vary from application to application, but the characters don't just disappear.
| Pegazus wrote: |
| Here is the Flow CICS -> MQ -> WMB (Convert EBCDIC to ASCII + BLOB to MRM) -> MRM to XML -> HTTP |
The XML parsers may discard white space though, and normalize "end-of-line" delimiters. Don't spend a lot of time "fixing" end-of-lines, if the XML parser is going to undo that work anyway. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Feb 18, 2009 12:59 am Post subject: Re: Preserving NewLine Chars on conversion from EBCDIC to AS |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| rekarm01 wrote: |
| There is a one-to-one mapping of control characters between EBCDIC character sets and ISO-8 ("ASCII") character sets: |
But control characters are not used in the same way.
| rekarm01 wrote: |
| Exactly which combination of these characters constitutes an "end-of-line" delimiter might vary from application to application, but the characters don't just disappear. |
I think the poster had already determined that further up the thread.
| rekarm01 wrote: |
| Don't spend a lot of time "fixing" end-of-lines, if the XML parser is going to undo that work anyway. |
This is a similar issue to the other thread I linked to. The point is that these characters are being treated as payload not whitespace and the intention seems to be that the XML document should include them.
So it becomes an issue of data transport and packaging. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Sat Feb 21, 2009 5:49 pm Post subject: Re: Preserving NewLine Chars on conversion from EBCDIC to AS |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
Parts of the original problem description are unclear:
| Pegazus wrote: |
| We change the Enconding on the MQ Input node ... |
The MQInput "Convert encoding" property governs the conversion of numeric data, not character data.
The MQInput "Convert coded character set ID" property governs the conversion of character data, not numeric data.
| Pegazus wrote: |
| ... to convert from EBCDIC to CCSID 500. |
CCSID=500 is EBCDIC, not ASCII.
| Vitor wrote: |
| rekarm01 wrote: |
| Don't spend a lot of time "fixing" end-of-lines, if the XML parser is going to undo that work anyway. |
This is a similar issue to the other thread I linked to. The point is that these characters are being treated as payload not whitespace and the intention seems to be that the XML document should include them. |
One important difference between this issue and the other thread is the XML parser, which treats these characters as whitespace, not payload; it has it's own translation rules for end-of-line handling, making it pointless to map CR->CRLF beforehand. |
|
| Back to top |
|
|
|
|
