| Author |
Message
|
| beth_carlin |
 Posted: Thu Aug 28, 2014 5:30 am Post subject: Chinese DBCS char convert from 1208 to 437 CCSID in MQ msg |
 |
|
Acolyte
Joined: 08 Jun 2004
Posts: 64
|
Hi,
I'd to ask one question about the conversion of a Chinese dbcs character data from ccsid=1208 msg to 437.
Please see the description:
1) I put a msg thru servconn onto a remote queue in Window QMGR1 -> (sender ) -> localq on Window QMGR2. Sender has convert(no). xmlMsg.characterSet = 1208 is set on incoming msg header. Both QMGR1 and QMGR2 have ccsid=437. My understanding is, the Chinese dbcs char should be corrupted on the localq of QMGR2 because it goes from 1208 -> 437 and 437 does not have the code page to represent DBCS. However, the CN DBCS char is still ok on the localq of QMGR2.
2) However if I remove the xmlMsg.characterSet = 1208 before the put, then the ccsid on the MQMD of the msg on localq on QMGR2 shows 819 . I am confused where 819 comes from because my put
programs run on window laptop which shows 437 via chcp command and I only have int openOptions = MQConstants.MQOO_OUTPUT | MQConstants.MQOO_FAIL_IF_QUIESCING ;
also i see the CN DBCS msg is corrupted on localq of QMGR2.
Would you please explain to me why CN DBCS char is not corrupted in #1 and is corrupted in #2.
Thanks so much for your time to explain to me
Beth |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Thu Aug 28, 2014 7:17 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
You are writing a message in 2) using the Queue Manager CCSID. (_AS_QMGR_DEF).
If you look at the CCSID of the queue manager then it shoud say 819
By setting it explicitly then you override the QMGR default.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Aug 28, 2014 7:21 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Also, why do you think scenario #1 should convert or corrupt the data?
What facts lead you to that conclusion? |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Aug 28, 2014 8:27 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| smdavies99 wrote: |
You are writing a message in 2) using the Queue Manager CCSID. (_AS_QMGR_DEF).
If you look at the CCSID of the queue manager then it shoud say 819
By setting it explicitly then you override the QMGR default. |
If you set the MQMD_CCSID of the message, only the MQMD_CCSID is changed - the data in the message payload remains exactly as it was created.
If the consuming application asks for conversion, the qmgr will attempt to convert the payload from the code-page specified in the MQMD_CCSID to the CCSID of the qmgr. As with most things, it's more complicted than that.
Think of the MQMD as the envelope of a letter, and the message data payload as the paper contents of the letter. I write a letter to a friend in US English, but on the envelope I write "I wrote the letter in French." Doing so, I've misled my friend. He will not be able to translate the words I've written based on what I've told him.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| beth_carlin |
| Posted: Thu Aug 28, 2014 12:23 pm Post subject: |
|
|
Acolyte
Joined: 08 Jun 2004
Posts: 64
|
Hi Bruce,
Thanks you so much taking the time to explain to me:
"if the consuming application asks for conversion, the qmgr will attempt to convert the payload from the code-page specified in the MQMD_CCSID to the CCSID of the qmgr. As with most things, it's more complicted than that. "
So now we have this scenario, the MQMD_CCSID is 1208 and the
QMGR2_CCSID is 437 and the payload data is 1208 because it was written as 1208 with valid CN DBCS.
I have a Java client consuming application connects to Window QMGR2
,browses the message and save it to a file in UTF-8 format. The laptop
where my consuming app runs is also on window with 437 codepage.
the openOptions and gmo are:
int openOptions = MQConstants.MQOO_INPUT_AS_Q_DEF | MQConstants.MQOO_FAIL_IF_QUIESCING | MQConstants.MQOO_BROWSE;
and
MQGetMessageOptions gmo = new MQGetMessageOptions();
gmo.options = MQConstants.MQGMO_BROWSE_FIRST | MQConstants.MQGMO_WAIT | MQConstants.MQGMO_FAIL_IF_QUIESCING;
I assume the consuming app should convert? the payload data in (1208) to the ccsid from QMGR2 which is 437 which should corrupt the data because there is no code point for the CN dbcs char in 437. However the data is not corrupt. It is still good.
Please correct my understanding of this flow.
Thanks again for your time
Beth |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Aug 28, 2014 1:28 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Conversion isn't done until you ask it to be done.
At no point in your flow do you ask for conversion to be done. So the data is received by Java as 1208 characters.
You don't specify MQGMO_CONVERT, so no conversion is done.
Except it's java, which is more complicated than that, but even then Java knows it's 1208 and handles 1208 and so there's no problem. |
|
| Back to top |
|
|
| beth_carlin |
| Posted: Thu Aug 28, 2014 2:42 pm Post subject: |
|
|
Acolyte
Joined: 08 Jun 2004
Posts: 64
|
Thanks you so much Jeff. So to prevent DBCS char from being converted
inadvertently between sender/receiver on diff qmgrs on diff servers, Would you please comment on the following guidelines
MQMD.CCSID should be set to 1208
payload data should always be written as UTF8
sender should have convert(no)
all qmgrs (ccsid=1208)
and all consumimg apps should not use GMO_CONVERT.
would it be ok like that?
Thanks you
Beth |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Aug 28, 2014 3:05 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| beth_carlin wrote: |
Thanks you so much Jeff. So to prevent DBCS char from being converted
inadvertently between sender/receiver on diff qmgrs on diff servers, Would you please comment on the following guidelines
MQMD.CCSID should be set to 1208
payload data should always be written as UTF8
sender should have convert(no)
all qmgrs (ccsid=1208)
and all consumimg apps should not use GMO_CONVERT.
would it be ok like that?
Thanks you
Beth |
Keep in mind that MQ doesn't care about or for your data payload unless you ask.
Research the three MQMD fields that may be important regarding your data - MQMD_FORMAT, MQMD_ENCODING, MQMD_CCSID.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Aug 28, 2014 3:49 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| beth_carlin wrote: |
Thanks you so much Jeff. So to prevent DBCS char from being converted
inadvertently.... |
MQ does not "inadvertently" or randomly convert. If an application is coded to request MQ to convert it, or if the Sender channel is set to convert, MQ will attempt to convert the data for you.
If the sending application wants to insure no conversion is done, it can set the MQMD Format field to MQFMT_NONE. Although this is not typical of apps that send character data. Apps that send character data will normally produced valid character data, identify it correctly with the proper MQMD CCSID value, set the MQMD Format field to MQFMT_STRING and then be done. Its duty was fulfilled in producing valid data and describing it correctly. Whether any downstream applications want conversion or not is not the sending app's worry at this point.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Thu Aug 28, 2014 10:01 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| Quote: |
MQMD Format field to MQFMT_NONE.
|
whilst this is a good tip (and I have used it myself in the past) I must ask that if it is used then it is used with a degree of caution.
In my experience there are some clients out there that when used OOTB they get into a hissy fit internally if the MQMD.Format field is blank. The messages are consumed but the systems don't get updated etc with the data.
Yes, these clients are badly coded but getting them changed is next to impossible.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Aug 29, 2014 4:51 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| smdavies99 wrote: |
| ...if the MQMD.Format field is blank. The messages are consumed but the systems don't get updated etc with the data. |
Please explain in MQ terminology exactly what you mean by 'the systems don't get updated, etc with the data.'
Are you saying that setting the MQMD format field to blanks causes MQ to lose messages? CURDEPTH is not maintained? The MQMD is consumed, but the data payload is lost? Please be precise.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Aug 29, 2014 4:53 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| PeterPotkay wrote: |
| MQ does not "inadvertently" or randomly convert. |
This is the big point here! It doesn't happen unless you ask it to happen!
The MQMD should *accurately represent the data in the message*.
So if you write string data into a message, then the MQMD.Format should indicate that it's String data.
If you write that string data in codepage 1208, then the MQMD.CCSID should indicate that.
If you write binary/numeric data into the message, and that numeric data is in Encoding 217 or whatever, then the MQMD.Encoding should represent that.
As long as the MQMD *accurately* represents the contents of the message, then the only problems you'll have if the data gets converted will be one of two things: a) the receiving application can't convert from the sending ccsid because there isn't a codpage locally, b) the data can't be converted because it's something like dbcs->sbcs.
The easiest way to avoid those two problems is to ensure that everyone uses a Unicode character set - 1200, 1208, whatever it takes... - then there's no problems.
But again, string data should still be marked as string data. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Aug 29, 2014 9:06 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| bruce2359 wrote: |
| smdavies99 wrote: |
| ...if the MQMD.Format field is blank. The messages are consumed but the systems don't get updated etc with the data. |
Please explain in MQ terminology exactly what you mean by 'the systems don't get updated, etc with the data.'
|
Ok, here goes.
This is from a real investigation that I undertook last month.
The client was reading the messages under sync point (so no rollback) and then trying to parse the input. The code (written in Java) had the ability to traverse additional headers (eg RFH2) but had a fatal bug where it just ignored FORMAT_NONE. All the other possible headers were accounted for but a blank format anywhere and the code just threw the message away without reporting an error. Yes, the coding of the client left a lot to be desired and is currently being re-tested with all the problems that were identified and hopefully corrected.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
|
|
