| Author |
Message
|
| Nigelg |
 Posted: Tue Feb 20, 2007 12:58 am Post subject: |
 |
|
Grand Master
Joined: 02 Aug 2004
Posts: 1046
|
It is not a problem that there is no DBCS element in ccsid.tbl. This is used for mixed byte character sets where the SBCS and DBCS elements of the code page are represented by different CCSIDs, e.g. 932 (Japanese) has DBCS 301 and SBCS 897. For Unicode UTF-8, there is a single CCSID for the whole character set.
On Solaris the conversion table 1208 -> 819 is /opt/mqm/lib/iconv/34B00333.tbl.
What exactly is the result of the conversion of 0x00B0?
_________________
MQSeries.net helps those who help themselves.. |
|
| Back to top |
|
 |
| Julia |
| Posted: Tue Feb 20, 2007 1:50 am Post subject: |
|
|
Newbie
Joined: 15 Feb 2007
Posts: 3
Location: France
|
| Nigelg wrote: |
On Solaris the conversion table 1208 -> 819 is /opt/mqm/lib/iconv/34B00333.tbl.
|
Well this one does not exist, so I guess that i the problem then? I tried to look for it on the IBM homepage yesterday. Is it normal that we don't have it? (As I said yesterday we have MQ V5.3 CSD 4 on Solaris 8.) Do you know where I can download this conversion table?
| Nigelg wrote: |
What exactly is the result of the conversion of 0x00B0?
|
The conversion of 0x00B0 does not take place at all. I have an error 2150. But for example with the double byte character é (0x00E9) the following happens:
Not converted:
3530 3035 3633 3032 3038 3330 3830 3630 '5005630208308060'
3030 3031 454D 4241 5345 2043 4F55 44E9 '0001EMBASE COUD.'
3133 3520 2020 2020 2020 2020 2020 2020 '135 '
2020 2020 2020 2020 2020 2020 2020 2020 ' '
Converted:
3530 3035 3633 3032 3038 3330 3830 3630 '5005630208308060'
3030 3031 454D 4241 5345 2043 4F55 441A '0001EMBASE COUD.'
3520 2020 2020 2020 2020 2020 2020 2020 '5 '
2020 2020 2020 2020 2020 2020 2020 ' '
Thank you for your help! |
|
| Back to top |
|
|
| Nigelg |
| Posted: Tue Feb 20, 2007 1:52 am Post subject: |
|
|
Grand Master
Joined: 02 Aug 2004
Posts: 1046
|
Further to my previous answer...
0x00B0 is not a valid UTF-8 character; the UTF-8 sequence for the degree character represented in 819 by 0xB0 is xC2B0.
This is the reason for the 2150 reason code.
You can test this on a machine that has a command line interface to iconv, e.g. AIX:
| Quote: |
$ uname -a
AIX forth 2 5 0042267A4C00
$ hex16 00b0.h
File: 00b0.h offset 0
00000000: 00B0 ................
$ iconv -f UTF-8 -t ISO8859-1 00b0.h
iconv: 0791-006 invalid chararacter found
|
_________________
MQSeries.net helps those who help themselves.. |
|
| Back to top |
|
|
| Julia |
| Posted: Tue Feb 20, 2007 2:27 am Post subject: |
|
|
Newbie
Joined: 15 Feb 2007
Posts: 3
Location: France
|
| Nigelg wrote: |
Further to my previous answer...
0x00B0 is not a valid UTF-8 character; the UTF-8 sequence for the degree character represented in 819 by 0xB0 is xC2B0.
This is the reason for the 2150 reason code.
|
Yes that's what I thought, but then I did not know if MQ actually receive C2B0 and if it is the browse message sample that only show me B0, or if MQ receive B0 without the C2.... I was not even sure if MQ was supposed to receive C2B0 or only B0, since on my computer when I look at the file with Ultra Edit in hexadecimal mode I only see B0 and not C2B0. (And for the é character i see E9 on my computer and in the MQ dump, but it should be C3A9...?)
My sending application is in Java. It is this application that picks up the file, creates the MQ message and chooses the CCSID, but I guess the choice of 1208 it not correct then?... I admit that I do not have a big knowledge in code pages and conversion between them.
/Julia |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Feb 20, 2007 4:28 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Julia, If your application is java, do not set any ccsid.
The qmgr should handle it with the default settings.
Now if you "know" that the contents of the byte stream (text or otherwise) is UTF-8 you may set a ccsid of 1208 on the message. Unicode = 1200.
But before setting any CCSID you should make sure the content of the byte stream matches the CCSID you are setting...
Now there is nothing wrong in setting the CCSID of your qmgr object to the CCSID of the platform you are running on. Just make sure the conversion table exists in the receiving qmgr.
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Pierre |
| Posted: Wed Feb 21, 2007 2:36 am Post subject: |
|
|
Newbie
Joined: 20 Feb 2007
Posts: 4
Location: Lyon, France
|
Hello,
I work with Julia and followed her issue almost from the beginning.
| fjb_saper wrote: |
Julia, If your application is java, do not set any ccsid.
The qmgr should handle it with the default settings. |
I agree with this. We should not force any CCSID as it is up to the java client and queue manager to handle this.
| fjb_saper wrote: |
| Now there is nothing wrong in setting the CCSID of your qmgr object to the CCSID of the platform you are running on. Just make sure the conversion table exists in the receiving qmgr. |
The table Nigelg talked about is actually on the server (the conversion table 1208 -> 819 is /opt/mqm/lib/iconv/34B00333.tbl).
I dumped the message received from MQ as bytes in a file, keeping it not converted and the hex string shows only B0. That means that the MQ message contains only B0 and when we perform a MQGET with CONVERT option, the queue manager has to translate B0 from CCSID 1208 to CCSID 819. And that doesn't work.
I had a look on the conversion table 34B00333.tbl and it seems that the only part of hex values that can be converted is between 21 and 7E. This is very confusing. I don't know if the message should contain C2B0 or if we should use another CCSID that supports conversion from B0.
/Pierre |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Feb 21, 2007 4:05 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Pierre,
If you have a problem then likely the message is not UTF-8.
So if you did set a ccsid of UTF-8 on the message you are running into all sorts of problems.
What happens if you don't set any ccsid and let the qmgr and client defaults handle it on the put?
Remember if you are using .NET that the default would be Unicode for .NET and that you might have to set it as the platform default is either 437 or 850?
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Pierre |
| Posted: Wed Feb 21, 2007 4:44 am Post subject: |
|
|
Newbie
Joined: 20 Feb 2007
Posts: 4
Location: Lyon, France
|
| fjb_saper wrote: |
So if you did set a ccsid of UTF-8 on the message you are running into all sorts of problems.
What happens if you don't set any ccsid and let the qmgr and client defaults handle it on the put?
|
The client application is a JMS one running on a Windows XP machine. We did so that no RFH2 header is set in the data of the message and we don't force any CCSID; we leave the default value.
| fjb_saper wrote: |
| If you have a problem then likely the message is not UTF-8 |
From what I understood, Java handles strings as UTF-8 so I am not surprise to receive a UTF-8 CCSID. What I don't understand is why the string doesn't look like a UTF-8 string.
Another thing is that 34B00333.tbl table does not show any way to get a ° (B0) charracter in CCSID 819. This is the reason why we are wondering if we should not use another CCSID than 819 on the queue manager.
/Pierre |
|
| Back to top |
|
|
| Nigelg |
| Posted: Wed Feb 21, 2007 6:00 am Post subject: |
|
|
Grand Master
Joined: 02 Aug 2004
Posts: 1046
|
I would not get too hung up on the hex contents of the conversion table. The conversion is nothing like as simple as looking up the position in the table using the character value, and then reading off the converted character, as in single byte conversions.
I have tried the conversion by putting a msg containing a character with the legitimate value of the degree sign in 1208, 0xC2B0, and then getting the msg with convert to 819, and the conversion succeeded, returning 0xB0.
The problem is that the msg contains an character, 0xB0, which is not valid in the stated CCSID, 1208, and so will return an error when the msg is converted.
The solution is to ensure that the msg contains valid characters only in the specified code CCSID.
_________________
MQSeries.net helps those who help themselves.. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Feb 21, 2007 5:03 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
@Pierre,
Just to ease my mind, you are using a javax.jms.TextMessage to put the information to the queue... Right?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Pierre |
| Posted: Thu Feb 22, 2007 1:04 am Post subject: |
|
|
Newbie
Joined: 20 Feb 2007
Posts: 4
Location: Lyon, France
|
| Nigelg wrote: |
| I have tried the conversion by putting a msg containing a character with the legitimate value of the degree sign in 1208, 0xC2B0, and then getting the msg with convert to 819, and the conversion succeeded, returning 0xB0 |
I just made that test and it is working fine for me as well. The data is retrieved from a file and I am wondering if we make the right calls to read the file. In the file "°" char is 0xB0. The issue we have may come from the format of the file.
| fjb_saper wrote: |
| Just to ease my mind, you are using a javax.jms.TextMessage to put the information to the queue... Right? |
I just had a look to the way we read the message and put it in the queue. The message is put as BytesMessage. The reason for this is that we build a RFH v1 structure at the begining of the message and we can only copy Integers values if message is of class BytesMessage.
The problem is then probably when we read the file: we read it as String and transform it to Bytes. There, the transformation should make 0xBO become 0xC2B0
/Pierre |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Feb 22, 2007 3:09 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Unfortunately, if the message is a Bytes Message, then it is likely that MQMD.Format = MQFMT_NONE, rather than MQFMT_STRING.
And if MQMD.Format=MQFMT_NONE, then text conversion isn't going to happen at the MQGet, which means you're relying on JMS to convert the data.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Pierre |
| Posted: Thu Feb 22, 2007 4:19 am Post subject: |
|
|
Newbie
Joined: 20 Feb 2007
Posts: 4
Location: Lyon, France
|
| jefflowrey wrote: |
Unfortunately, if the message is a Bytes Message, then it is likely that MQMD.Format = MQFMT_NONE, rather than MQFMT_STRING.
And if MQMD.Format=MQFMT_NONE, then text conversion isn't going to happen at the MQGet, which means you're relying on JMS to convert the data. |
The problem we have is that the conversion issues a 2150 warning message.
Actually, we set message format to MQFMT_STRING. I am sure now that the problem is that we put a message as UTF-8 (CCSID 1208) but the string we put is not converted since it comes from an ASCII text file, read as Bytes.
/Pierre |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Feb 22, 2007 2:04 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Pierre wrote: |
| jefflowrey wrote: |
Unfortunately, if the message is a Bytes Message, then it is likely that MQMD.Format = MQFMT_NONE, rather than MQFMT_STRING.
And if MQMD.Format=MQFMT_NONE, then text conversion isn't going to happen at the MQGet, which means you're relying on JMS to convert the data. |
The problem we have is that the conversion issues a 2150 warning message.
Actually, we set message format to MQFMT_STRING. I am sure now that the problem is that we put a message as UTF-8 (CCSID 1208) but the string we put is not converted since it comes from an ASCII text file, read as Bytes.
/Pierre |
Sounds about right.
What if you let the client handle the put with the default CCSID of the platform?
The question really is what format is the data in the file in. You can still use the JMS Text message and transform the byte array into a String before putting it to the message.
If you need conversion from the file format remember to use the nio classes....
This should then allow for automatic conversion...
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
