| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| Conversion problem with ! inside an XML message |
« View previous topic :: View next topic » |
| Author |
Message
|
| rekarm01 |
 Posted: Sun Aug 01, 2010 2:04 pm Post subject: Re: How to determine platform CCSID on z/OS |
 |
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| gbaddeley wrote: |
| rekarm01 wrote: |
| gbaddeley wrote: |
| There are 10 well known characters that are subject to accidental perbertatation in EBCDIC code pages 37, 500 and 1047. |
I count 9 characters: ... |
A table I put together in 2002 for a related problem contained 10 characters of interest at the time. |
Ok. Perhaps that table compares more than just code pages 37, 500, and 1047.
Appendix F compares eleven different EBCDIC Latin-1 code pages, (and still, that's not even all of them). But it's a good list to help identify problem characters.
Other parts of the Data Conversion document could use some work though. For example:
| Quote: |
Common Procedures when doing the MQPUT
Regardless of whether the message is character, numeric, or a mixture of both, in the application that is doing the initial MQPUT of the message, you should:- Set MQMD.CodedCharSetId to MQCCSI_Q_MGR ...
Note: In a client application, MQCCSI_Q_MGR refers to the CCSID of the application locale, not the CCSID of the remote queue manager. In all cases, when setting MQMD.CodedCharSetId to MQCCSI_Q_MGR, it is the responsibility of the application to ensure that the implied CCSID correctly represents the string data in the message.
|
Before an MQPUT, the application should set MQMD.CodedCharSetId to whichever CCSID correctly represents the string data in the message.
The application should not just assume that MQCCSI_Q_MGR is the correct CCSID. |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Sun Aug 01, 2010 2:23 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| Before an MQPUT, the application should set MQMD.CodedCharSetId to whichever CCSID correctly represents the string data in the message. |
The "usual" behavior is to let the qmgr set the ccsid to its own.
The developer would need to set the ccsid manually if the application is sensitive to the character set - that is, the programmer manages (manipulates) the bits/bytes from internal tables (ccsids).
What if the app was written on one platform/ccsid, but deployed on another OR many different platforms/ccsids?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Sun Aug 01, 2010 3:20 pm Post subject: Re: How to determine platform CCSID on z/OS |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| rekarm01 wrote: |
Before an MQPUT, the application should set MQMD.CodedCharSetId to whichever CCSID correctly represents the string data in the message.
The application should not just assume that MQCCSI_Q_MGR is the correct CCSID. |
I have rarely seen this done and I'm not too concerned about it. Most apps programmers don't even know or care about the CCSID that the program is using, and everything works OK.
If there is a local CCSID conflict between MQ, the program, and other resources it is using such as databases, adapters, terminal screens, web interfaces, and it is causing problems with character translation, THEN pay attention to setting CCSID in the right places and do rigourous testing.
This issue turns up in EBCDIC more often than ASCII.
_________________
Glenn |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sun Aug 01, 2010 4:11 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Some characters the C character set are not available in all environments.
You can enter these characters into a C source program by using a sequence of three characters that are called a trigraph.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Mon Aug 02, 2010 3:59 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| bruce2359 wrote: |
Some characters the C character set are not available in all environments.
You can enter these characters into a C source program by using a sequence of three characters that are called a trigraph. |
Yes. I've written a lot of cross-platform C code that uses MQ on various single-byte ASCII and EBCDIC platforms. The main issue is that EBCDIC has a different collation sequence to ASCII, so coding tricks of doing arithmetic on alphabetic characters can't be done. eg. case conversion, hex literal conversion, range checking for printable characters. Its safer to stick to the system functions for doing this sort of thing, eg. strupper, isprint, sscanf.
_________________
Glenn |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Aug 02, 2010 6:31 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| gbaddeley wrote: |
| Its safer to stick to the system functions for doing this sort of thing |
This is always true.
I remember a section of C code designed to find out what the current time was.
By - opening a file
- ... ... no, really... writing data to it....
- closing the file
- checking the modification date.
- deleting the file
It wasn't even written by an RPG programmer.... |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Tue Aug 03, 2010 1:14 am Post subject: Re: How to determine platform CCSID on z/OS |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| bruce2359 wrote: |
| Quote: |
| Before an MQPUT, the application should set MQMD.CodedCharSetId to whichever CCSID correctly represents the string data in the message. |
The "usual" behavior is to let the qmgr set the ccsid to its own. |
... and that's fine, just as long as the qmgr ccsid matches the string data. The Data Conversion document could be more explicit about that.
One of the most frequent causes of conversion issues posted here is the MQPUT application providing the wrong ccsid, and more often than not, it's the qmgr ccsid that doesn't match the data.
| bruce2359 wrote: |
| What if the app was written on one platform/ccsid, but deployed on another OR many different platforms/ccsids? |
I don't know; you tell me. Does the app encode string data for the platform it was written on, or for the platform it is deployed to?
| bruce2359 wrote: |
Some characters the C character set are not available in all environments.
You can enter these characters into a C source program by using a sequence of three characters that are called a trigraph. |
I'm not entirely sure how this relates to the topic at hand, but that's ok.
The C basic source character set is a subset of the 7-bit US ASCII character set, (excluding $, @, `, and most of the control characters). The C standard defines trigraphs for nine of these characters:
Which of these characters are not available in which environments?
The C standard originally introduced trigraphs due to variations in 7-bit ISO/IEC 646 code pages (ASCII, not EBCDIC). The characters were available, but different code pages mapped them to different values. C defined trigraphs for those characters that fall outside of the ISO/IEC 646 Invariant code set.
8-bit ISO/IEC 8859 has largely replaced 7-bit ISO/IEC 646, so the original need for trigraphs no longer exists. But EBCDIC programmers found them useful too, so it looks like they won't be going away soon. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Aug 03, 2010 5:41 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| ... and that's fine, just as long as the qmgr ccsid matches the string data. The Data Conversion document could be more explicit about that. |
The subject of data conversion is discussed in the APR and APG.
These are three fields in the MQMD that describe the application data component of a message: ENCODING, FORMAT and CCSID. There are initial values for each; the application may change these values.
| Quote: |
| I don't know; you tell me. Does the app encode string data for the platform it was written on, or for the platform it is deployed to? |
Usually, neither.
| Quote: |
Some characters the C character set are not available in all environments.
I'm not entirely sure how this relates to the topic at hand, but that's ok. |
C offers an example where there is an exception to the best-practice of not encoding for a specific platform.
I've seen apps that created a msg on Intel, but destined for z/OS. The creative developer build an internal table and manually translated Intel-ish data to z. The app worked fine until the destination platform changed to Intel. The developer asked me why WMQ couldn't deal with this (creative coding technique he'd used).
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Wed Aug 04, 2010 2:15 am Post subject: Re: How to determine platform CCSID on z/OS |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| bruce2359 wrote: |
| The subject of data conversion is discussed in the APR and APG. |
Yes, it is. Yet, some of the statements in the Data Conversion doc are misleading, regardless of where else it may be discussed.
| bruce2359 wrote: |
| These are three fields in the MQMD that describe the application data component of a message: ENCODING, FORMAT and CCSID. There are initial values for each; the application may change these values. |
Yes, but you wouldn't know that, by reading this:
| Quote: |
... in the application that is doing the initial MQPUT of the message, you should:- Set MQMD.CodedCharSetId to MQCCSI_Q_MGR. ...
- Set MQMD.Encoding to MQENC_NATIVE. ...
|
Sometimes, you shouldn't do that.
| bruce2359 wrote: |
| Quote: |
| Quote: |
| What if the app was written on one platform/ccsid, but deployed on another OR many different platforms/ccsids? |
Does the app encode string data for the platform it was written on, or for the platform it is deployed to? |
Usually, neither. |
It has to do something; otherwise the what-if scenario is rather pointless.
| bruce2359 wrote: |
| Some characters the C character set are not available in all environments. |
All of the characters of the C basic character set must be available in any environment that can compile or execute C programs. The C standard requires this.
| bruce2359 wrote: |
| C offers an example where there is an exception to the best-practice of not encoding for a specific platform. |
That's not quite what this is an example of. C trigraphs can make it easier for the compiler to correctly read the source code. They have no effect whatsoever on the runtime behavior of the compiled executable.
Edit: added urls
Last edited by rekarm01 on Wed Aug 04, 2010 5:01 pm; edited 1 time in total |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Aug 04, 2010 5:35 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
Yes, but you wouldn't know that, by reading this:
Quote:
... in the application that is doing the initial MQPUT of the message, you should:
Set MQMD.CodedCharSetId to MQCCSI_Q_MGR. ...
Set MQMD.Encoding to MQENC_NATIVE. ... |
When you quote something, please cite your source (with URL or doc name number).
I suppose you are reading the Data Conversion supportpac doc, yes? It is not the only source of information. You must read other stuff.
The two Set statements merely enforce that 1) you should set values to all MQMD fields (even though these might be the initial values of the MQMD); and 2) set the ccsid field to the qmgrs ccsid; and 3) that numerics in this message were encoded in the platform-specific method of encoding.
| Quote: |
| Sometimes, you shouldn't do that. |
While the qmgr can/will set some of the MQMD fields, it is the responsibility of the app developer to ensure that all of the MQMD fields are set appropriately for the application message.
If you have developed an app on Win/Intel, AND have force-fed AIX or z/OS characters into the message, you will need to set the MQMD fields to whatever works, so that the consumer app will successfully drive data conversion with MQGET MQGMO_CONVERT. Or, I suppose you can develop the consumer app to manually convert the force-fed data back to an appropriate ccsid.
CCSIDs are discussed at length in the APG (and elsewhere).
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Wed Aug 04, 2010 7:11 pm Post subject: Re: How to determine platform CCSID on z/OS |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| bruce2359 wrote: |
| Quote: |
Yes, but you wouldn't know that, by reading this:
| Quote: |
... in the application that is doing the initial MQPUT of the message, you should:
Set MQMD.CodedCharSetId to MQCCSI_Q_MGR. ...
Set MQMD.Encoding to MQENC_NATIVE. ... |
|
When you quote something, please cite your source (with URL or doc name number). |
OK. And when you quote someone, please indicate whom.
Here's the URL again:
as quoted earlier in this thread.
| bruce2359 wrote: |
| I suppose you are reading the Data Conversion supportpac doc, yes? |
No, I'm discussing the Data Conversion supportpac doc.
| bruce2359 wrote: |
| It is not the only source of information. You must read other stuff. |
Yes. Thanks again. I've read other stuff. I'm not discussing other stuff right now.
| bruce2359 wrote: |
| it is the responsibility of the app developer to ensure that all of the MQMD fields are set appropriately for the application message. |
Yes, it is.
It's unfortunate the Data Conversion supportpac doc says something different. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Aug 04, 2010 9:22 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| I've read other stuff. I'm not discussing other stuff right now. |
I'd say that you are bitching about the Data Conversion doc. I'm sorry you depended solely on its contents.
| Quote: |
| It's unfortunate the Data Conversion supportpac doc says something different. |
Data conversion is an advanced topic. The Data Conversion doc, like most references, is a reference. It is one more source of information on the subject.
I'd suspect that the authors intended that it not be the sole source of data conversion information.
WMQ behaves according to the official IBM manuals (or InfoCenter) specifications. If it doesn't, open a PMR.
I'm not sure I/we can offer you anything further if your only source continues to be just the Data Conversion doc.
I've been working with IBM- (and other vendor-) hardware and software for 30+ years. The WMQ manuals are the best I've seen from any vendor.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Fri Aug 06, 2010 3:39 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
What??
A bad MQPUT ccsid is a very common problem, frequently caused by developers doing exactly what the "Data Conversion" doc tells them to do. Sometimes qmgrs get misconfigured, and sometimes developers don't bother to check the resulting ccsid against their data. Why wouldn't it be helpful to point that out?
| bruce2359 wrote: |
| I'd say that you are bitching about the Data Conversion doc. |
Not bitching, ... warning. The APR and APG don't have this same issue, so there's no reason to warn about them, but thanks for the non-warnings anyway.
| bruce2359 wrote: |
| I'd suspect that the authors intended that it not be the sole source of data conversion information. |
So, don't ever mention that the statements in one document might be misleading, because there happen to be other documents out there??
| bruce2359 wrote: |
| I'm not sure I/we can offer you anything further ... |
Please try to keep better track of who is actually asking for help, and who isn't. |
|
| Back to top |
|
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
