| Author |
Message
|
| anar |
 Posted: Fri Jul 16, 2010 3:45 am Post subject: CCSID & Encoding |
 |
|
Apprentice
Joined: 28 Jun 2010
Posts: 31
|
Hi.
I got probably problems with CCSID and Encoding.
The workflow is:
Step_1:
- ASP.NET WebService (hosted in Windows XP) posts a message in queue_A of queueManager with CCSID 1208 on MQ installed in HP-UX 11.31 v2, LC_CTYPE=univ.utf8.
Step_2:
- application_X hosted in the same HP-UX server Gets the message from queue_A through JAVA library. (local development)
Step_3:
- message processed
Step_4:
- application_X Posts the reply(the same message string) to queue_B
Step_5:
- ASP.NET WebService Gets the message from queue_B.
The following happens:
- In Step_4 as message Posted, its length changes. (probably due CCSID conversions)
- In Step_5 as message retrieved by ASP.NET WebService, the message aint readable.
Would be pleased if someone would point to some comprehensive documentation on CCSID and Encoding settings.
Or, what would be the right values in all parties for CCSID and Encoding?
Thanks in advance.
Best Regards, |
|
| Back to top |
|
 |
| Vitor |
| Posted: Fri Jul 16, 2010 4:19 am Post subject: Re: CCSID & Encoding |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| anar wrote: |
The following happens:
- In Step_4 as message Posted, its length changes. (probably due CCSID conversions) |
Probably? How can you diagnose a conversion problem when you don't know where conversion is happening? WMQ doesn't convert messages unless the administrator (presumably you) or the application requests that it does so.
| anar wrote: |
| - In Step_5 as message retrieved by ASP.NET WebService, the message aint readable. |
How not readable? In Chinese? Did the web service request conversion or is it just reading the message? If so, why isn't it attempting conversion into a readable format? This follows on my comments above about diagnosing conversion problems without knowing where conversion is being attempted.
| anar wrote: |
| Would be pleased if someone would point to some comprehensive documentation on CCSID and Encoding settings. |
Try Mr Google or the majority of posts in here from kimbert
| anar wrote: |
| Or, what would be the right values in all parties for CCSID and Encoding? |
CCSID and encoding are 2 different things that cover different data. Any values are acceptable providing that conversion is done where needed (hint!) and all parties understand that.
Alternatively put everything in Unicode and end all your problems.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Jul 16, 2010 6:03 am Post subject: Re: CCSID & Encoding |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Vitor wrote: |
| anar wrote: |
The following happens:
- In Step_4 as message Posted, its length changes. (probably due CCSID conversions) |
Probably? How can you diagnose a conversion problem when you don't know where conversion is happening? WMQ doesn't convert messages unless the administrator (presumably you) or the application requests that it does so. |
Certainly because if the poster did not specify the CCSID on the message or on the CF (JMS) ... the message will be converted to the default CCSID of the sender's (or attached qmgr's, you'd have to check) platform. In this case I would imagine it to be possibly 819. Note that in JMS you can also assign a CCSID to the destination...
| Vitor wrote: |
| anar wrote: |
| - In Step_5 as message retrieved by ASP.NET WebService, the message aint readable. |
How not readable? In Chinese? Did the web service request conversion or is it just reading the message? If so, why isn't it attempting conversion into a readable format? This follows on my comments above about diagnosing conversion problems without knowing where conversion is being attempted. |
Can anar please confirm that with not readable he means some chars have been transposed to the not convertible output and as such rendered the XML as an invalid document?
| Vitor wrote: |
| anar wrote: |
| Would be pleased if someone would point to some comprehensive documentation on CCSID and Encoding settings. |
Try Mr Google or the majority of posts in here from kimbert
| anar wrote: |
| Or, what would be the right values in all parties for CCSID and Encoding? |
CCSID and encoding are 2 different things that cover different data. Any values are acceptable providing that conversion is done where needed (hint!) and all parties understand that.
Alternatively put everything in Unicode and end all your problems. |
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Jul 16, 2010 6:09 am Post subject: Re: CCSID & Encoding |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| fjb_saper wrote: |
| Certainly because if the poster did not specify the CCSID on the message or on the CF (JMS) ... the message will be converted to the default CCSID of the sender's (or attached qmgr's, you'd have to check) platform. In this case I would imagine it to be possibly 819. Note that in JMS you can also assign a CCSID to the destination... |
To me this is still application (though by something of a stretch) and doesn't invalidate my point about knowing where conversion is happening or suposed to be happening.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Jul 16, 2010 6:20 am Post subject: Re: CCSID & Encoding |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Vitor wrote: |
| fjb_saper wrote: |
| Certainly because if the poster did not specify the CCSID on the message or on the CF (JMS) ... the message will be converted to the default CCSID of the sender's (or attached qmgr's, you'd have to check) platform. In this case I would imagine it to be possibly 819. Note that in JMS you can also assign a CCSID to the destination... |
To me this is still application (though by something of a stretch) and doesn't invalidate my point about knowing where conversion is happening or suposed to be happening. |
Agreed. But I believe we should also be aware that a lot of people just use the default setup for the msg, CF, Destination without thinking of the effect on CCSID. While this will work in about 90 % of the cases, there is that pesky 10% that you will run into, every once in a while. I thought the analysis done by the OP was pinpointing his problem to the server program putting the reply to the queue/topic.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| anar |
| Posted: Sun Jul 18, 2010 10:01 pm Post subject: |
|
|
Apprentice
Joined: 28 Jun 2010
Posts: 31
|
Thanks for clues and questions to clarify the issue.
| Quote: |
Vitor
Probably? How can you diagnose a conversion problem when you don't know where conversion is happening? WMQ doesn't convert messages unless the administrator (presumably you) or the application requests that it does so. |
Message data as Step_2 passed:
00000 00 0B 46 54 2E 53 4F 4D--45 2E 30 39 33 |. FT.SOME.093 |
Message data as Step_4 passed:
00000 00 0E C0 80 0B 46 54 2E--53 4F 4D 45 2E 30 39 33 |................|
| Quote: |
Vitor
How not readable? In Chinese? |
| Quote: |
fjb_saper
Can anar please confirm that with not readable he means some chars have been transposed to the not convertible output and as such rendered the XML as an invalid document? |
Message data after Step_5 is not the data entered in Step_1, they appear in Chenise exactly 
| Quote: |
fjb_saper
Certainly because if the poster did not specify the CCSID on the message or on the CF (JMS) ... the message will be converted to the default CCSID of the sender's (or attached qmgr's, you'd have to check) platform. In this case I would imagine it to be possibly 819. Note that in JMS you can also assign a CCSID to the destination... |
The situation is following:
We got non ASCII characters, so I did set `export LC_CTYPE=univ.utf8' before `crtmq`. The CCSID of the QMGR is 1208, thus.
In all steps CCSID explicitly set to 1208. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jul 19, 2010 4:01 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| anar wrote: |
| We got non ASCII characters, so I did set `export LC_CTYPE=univ.utf8' before `crtmq`. The CCSID of the QMGR is 1208, thus. |
I'd take that "thus" with a pinch of salt & would like you to post the result of a DISPLAY QMGR CCSID. I'd also have though an ALTER QMGR command would be much eaiser than fiddling with envrionment variables.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| anar |
| Posted: Mon Jul 19, 2010 4:11 am Post subject: DIS QMGR CCSID |
|
|
Apprentice
Joined: 28 Jun 2010
Posts: 31
|
16 : dis qmgr ccsid
AMQ8408: Display Queue Manager details.
QMNAME(TST12.QMGR.01) CCSID(1208) |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jul 19, 2010 4:39 am Post subject: Re: DIS QMGR CCSID |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| anar wrote: |
16 : dis qmgr ccsid
AMQ8408: Display Queue Manager details.
QMNAME(TST12.QMGR.01) CCSID(1208) |
Fair point.
So how does the application in Step 4 process & put the message? Specifically what JMS / .NET methods is it using?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| anar |
| Posted: Mon Jul 19, 2010 5:15 am Post subject: |
|
|
Apprentice
Joined: 28 Jun 2010
Posts: 31
|
Step_1:
| Code: |
IBM.WMQ.MQQueueManager v_qmng = new MQQueueManager(this.v_qmng_str);
IBM.WMQ.MQQueue v_que = v_qmng.AccessQueue(this.v_rqQue_str, IBM.WMQ.MQC.MQOO_OUTPUT + IBM.WMQ.MQC.MQOO_FAIL_IF_QUIESCING);
IBM.WMQ.MQMessage v_qmsg = new MQMessage();
v_qmsg.CharacterSet = 1208;
v_qmsg.Encoding = IBM.WMQ.MQC.MQENC_NATIVE;
v_qmsg.WriteUTF(v_reqStr);
v_qmsg.Format = IBM.WMQ.MQC.MQFMT_STRING;
v_qmsg.ReplyToQueueName = v_rpQue_str;
IBM.WMQ.MQPutMessageOptions v_qPutOpts = new MQPutMessageOptions();
v_qPutOpts.Options = IBM.WMQ.MQC.MQPMO_NEW_MSG_ID + IBM.WMQ.MQC.MQPMO_NEW_CORREL_ID + IBM.WMQ.MQC.MQPMO_FAIL_IF_QUIESCING;
v_que.Put(v_qmsg, v_qPutOpts);
byte[] v_corrID = new byte[v_qmsg.CorrelationId.Length];
v_qmsg.CorrelationId.CopyTo(v_corrID, 0);
v_str_resp += "[crrID: " + System.Text.Encoding.UTF8.GetString(v_qmsg.CorrelationId) + "]";
v_que.Close(); |
In "MQ Explorer":
CCSID: 1208
Encoding: 546
Length: 13
Data: 00000 00 0B 46 54 2E 53 4F 4D--45 2E 31 30 37 |. FT.SOME.107 |
Step_2:
| Code: |
com.ibm.mq.MQEnvironment.CCSID=1208;
com.ibm.mq.MQQueueManager v_qmgr=new com.ibm.mq.MQQueueManager(v_qmgr_str);
int v_rqQue_opts=com.ibm.mq.constants.CMQC.MQOO_INPUT_AS_Q_DEF+com.ibm.mq.constants.CMQC.MQOO_FAIL_IF_QUIESC
ING;
com.ibm.mq.MQQueue v_que=v_qmgr.accessQueue(v_rqQue_name_str, v_rqQue_opts);
com.ibm.mq.MQMessage v_rqMsg=new com.ibm.mq.MQMessage();
v_rqMsg.format=com.ibm.mq.constants.CMQC.MQFMT_STRING;
v_rqMsg.characterSet=1208;
com.ibm.mq.MQGetMessageOptions v_msgGetOpts=new com.ibm.mq.MQGetMessageOptions();
v_msgGetOpts.options=com.ibm.mq.constants.CMQC.MQGMO_CONVERT;
v_que.get(v_rqMsg, v_msgGetOpts); |
Step_4:
| Code: |
com.ibm.mq.MQEnvironment.CCSID=1208;
com.ibm.mq.MQQueueManager v_qmgr=new com.ibm.mq.MQQueueManager(v_qmgr_str);
int v_rpQue_opts=com.ibm.mq.constants.CMQC.MQOO_OUTPUT+com.ibm.mq.constants.CMQC.MQOO_FAIL_IF_QUIESCING;
com.ibm.mq.MQQueue v_que=v_qmgr.accessQueue(v_rpQue_name_str, v_rpQue_opts);
com.ibm.mq.MQMessage v_rpMsg=new com.ibm.mq.MQMessage();
v_rpMsg.format=com.ibm.mq.constants.CMQC.MQFMT_STRING;
v_rpMsg.correlationId=v_rpCorrelId;
v_rpMsg.characterSet=1208;
v_rpMsg.encoding=com.ibm.mq.constants.CMQC.MQENC_NATIVE;
v_rpMsg.writeUTF(v_rpMsgStr);
com.ibm.mq.MQPutMessageOptions v_msgPutOpts=new com.ibm.mq.MQPutMessageOptions();
v_msgPutOpts.options=com.ibm.mq.constants.CMQC.MQPMO_NEW_MSG_ID+com.ibm.mq.constants.CMQC.MQGMO_CONVERT+com.ibm.mq.constants.CMQC.MQPMO_FAIL_IF_QUIESCING;
v_que.put(v_rpMsg, v_msgPutOpts); |
In "MQ Explorer":
CCSID: 1208
Encoding: 273
Length: 16
Data: 00000 00 0E C0 80 0B 46 54 2E--53 4F 4D 45 2E 31 30 37 |.... FT.SOME.107| |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Jul 19, 2010 5:37 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Don't use writeUTF.
It doesn't do what you think it does. |
|
| Back to top |
|
|
| anar |
| Posted: Mon Jul 19, 2010 8:25 pm Post subject: |
|
|
Apprentice
Joined: 28 Jun 2010
Posts: 31
|
:D :D :D
it certainly doesn. Grateful for, Mr.Mqjeff.
The result with the writeString, instead, is:
After Step_1 in "MQ Explorer":
CCSID: 1208
Encoding: 546
Length: 13
Data: 00000 00 0B 46 54 2E 53 4F 4D--45 2E 31 30 37 |. FT.SOME.107 |
After Step_4 in "MQ Explorer":
CCSID: 1208
Encoding: 273
Length: 13
Data: 00000 00 0B 46 54 2E 53 4F 4D--45 2E 31 30 37 |. FT.SOME.107 |
Thanks to all contributors. Grateful for the patience shown, efforts, and time. |
|
| Back to top |
|
|
|
|
