| Author |
Message
|
| halbe01 |
 Posted: Thu May 17, 2007 5:14 am Post subject: UTF8 to UTF16 Convert on Get |
 |
|
Novice
Joined: 02 Feb 2005
Posts: 16
|
In theory (at least as far as I can tell) it is possible to get MQ to convert a UTF8 message into UTF16 on a Get. I have been trying this for a while now, with no success.
I have enqueued a message to a queue, with UTF8 content (in fact I'm using a simple ASCII text file, but this is valid UTF8). I have tried using CCSIDs of the enqueued message as 437, 850 and 1208. (1208 is UTF- .
I perform my dequeue, using either a bespoke tool or using RFHUtil, and I set the MQMD.CCSID to 1200 (which is UTF-16) and get the Convert option in the GMO. Hence I should get an automatic conversion to UTF16 on my Get. However this doesn't work and my output is still ASCII/UTF8. I have also ensured I have no headers other than the MQMD, and the MQMQ.Format field is set to "MQSTR ".
If I try the conversion the other way around it works perfectly - i.e. enqueueing a UTF16 message with a CCSID of 1200, and performing my get with the MQMD.CCSID set to 1208 and it converts the UTF16 perfectly into UTF8.
I do get an error message in my amqerr.log file which is interesting:
| Quote: |
AMQ6175: The system could not dynamically load the library C:\Program
Files\IBM\WebSphere MQ\exits\MQSTR.dll. The system return code was 536895861.
The queue manager will continue without this module. |
It seems somehow that MQ is trying to call a Data conversion exit because it doesn't recognise the type of data in the mesasge. However as I specified the MQFORMAT field to be "MQSTR " I don't understand how this could happen.
If anyone has any experience or pointers on this, I'd appreciate it. Even if someone can confirm for me whether or not this is possible using MQ.
I am using Windows XP boxes for testing, and have tried this on both MQ5.3 and MQ6.
Thanks,
Ben. |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Thu May 17, 2007 5:24 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
That error means that the format field is not correct somehow - at least it almost always does.
Instead of setting MQMD.Format to a string literal, set it to the offical MQ constant (which is a string literal underneath, but don't worry about that) of MQFMT_STRING.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| markt |
| Posted: Thu May 17, 2007 6:14 am Post subject: |
|
|
Knight
Joined: 14 May 2002
Posts: 508
|
| Quote: |
| That error means that the format field is not correct somehow - at least it almost always does. |
When it mentions a "standard" format, like MQSTR, it can almost be guaranteed that it means there is no available converter between the two codepages. |
|
| Back to top |
|
|
| halbe01 |
| Posted: Thu May 17, 2007 6:44 am Post subject: |
|
|
Novice
Joined: 02 Feb 2005
Posts: 16
|
Thanks for the ideas guys.
Interestingly, I tried using a tool to enqueue the test file (and set the MQFORMAT) I don't get the DLL error in the log. However, still no actual conversion seems to happen!
I found some references in the documentation (and web) that the conversion I want to do is possible... so I'm working on the basis I don't need a data conversion exit. But I'm still not 100% sure on this... |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu May 17, 2007 8:31 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| markt wrote: |
| Quote: |
| That error means that the format field is not correct somehow - at least it almost always does. |
When it mentions a "standard" format, like MQSTR, it can almost be guaranteed that it means there is no available converter between the two codepages. |
It seems unlikely that there would be a converter available from 1200 to 1208 but not from 1208 to 1200.
I could understand if it was not possible to deterministically go from UTF-16 to UTF-8. It seems harder to believe that it's not possible to deterministically go from UTF-8 to UTF-16.
Of course, I haven't looked at my install directory to see what conversion tables are actually shipped.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| halbe01 |
| Posted: Fri May 18, 2007 3:32 am Post subject: |
|
|
Novice
Joined: 02 Feb 2005
Posts: 16
|
After some late night working, I've determined the cause of this issue:
The files I thought were UTF8 were not in fact UTF8 enough for MQ to recognise them as such. UTF8 in MQ terms needs to have a three byte BOM marker at the start. My files didn't have this. Even those files which MQ converted to UTF8 from UTF16 didn't have these bytes.
If you put a proper UTF8 or UTF16 message through, with the right CCSID and the Convert flag set, all is well and MQ does it's job.
Thanks for all the advice.
Cheers,
Ben. |
|
| Back to top |
|
|
| halbe01 |
| Posted: Mon May 21, 2007 10:03 am Post subject: |
|
|
Novice
Joined: 02 Feb 2005
Posts: 16
|
Hi all,
I've hit another problem with this - and although I've implemented a work-around, I'd love to find out if MQ can really do what I was hoping to get it to do.
The first requirement is to send a message, the body of which will be text encoded in UTF8, and to convert it to UTF16 on the Get. This conversion now works perfectly. However there is a further requirement here to include an RFH2 header with data stored in the <usr> section of this header. When I try this, specifying the message body CCSID of 1208 within the RFH.CCSID field, and trying the same Get I get an "MQRC_CONVERTED_STRING_TOO_BIG" error. I understand why this occurs as it is specified in the documentation that it is not possible to convert an RFH2 header to a double-byte character set like UTF16. However is seems that it is also not possible to tell MQ to convert only the body of a message, not the headers. Hence you cannot convert a message with an RFH2 header to UTF16 on Get. My question is - is this really not possible or have I missed something?
Thanks for any information,
Cheers,
Ben. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon May 21, 2007 6:29 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
The RFH header allows in its user section only a handfull of CCSID codes as documented. There should be no reason to translate this because most implementations of java / JMS will understand the correct value of the usr folder...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| halbe01 |
| Posted: Tue May 22, 2007 12:52 am Post subject: |
|
|
Novice
Joined: 02 Feb 2005
Posts: 16
|
That makes sense - however the presence of the RFH2 header therefore seems to prevent MQ from being able to translate the mesage body (i.e. you can only attempt to translate *everything* which fails because of the header). This doesn't seem right to me so I'm hoping I've just missed something.
Regards,
Ben. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue May 22, 2007 12:47 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| halbe01 wrote: |
That makes sense - however the presence of the RFH2 header therefore seems to prevent MQ from being able to translate the mesage body (i.e. you can only attempt to translate *everything* which fails because of the header). This doesn't seem right to me so I'm hoping I've just missed something.
Regards,
Ben. |
Looks like you missed to specify the format of the part following the RFH on the RFH.
Remember header linking/ chaining ....
Once you insert an RFH you need to:
Set the RFH.CCSID to the MQMD.CCSID
Set the RFH.FORMAT to the MQMD.FORMAT.
Set the MQMD Format to describe the RFH
set the RFH.encoding to the MQMD.encoding
And handle the removal of an RFH in this manner as well....
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| halbe01 |
| Posted: Fri May 25, 2007 2:08 am Post subject: |
|
|
Novice
Joined: 02 Feb 2005
Posts: 16
|
I have doubled checked, and I do set all these correctly. Basically the various fields in the MQMD state that RFH2 header comes next, and that it is in a single-byte format, and the various fields in the RFH2 state that the next section of the message is an "MQSTR" i.e. the message body and that this is in CCSID of 1208 (UTF8).
So the message itself looks fine to me, however when I perform the Get with convert, I still get this MQRC_CONVERTED_STRING_TOO_BIG message - although this is actually a warning not an error. And the conversion doesn't happen.
I'm still wondering if this is supposed to be possible - can it be that you can't convert a message body to UTF16 just because an attached header can't be converted? Is it not possible to convert just the body? I can't see a way...
Many Thanks,
Ben. |
|
| Back to top |
|
|
| Nigelg |
| Posted: Fri May 25, 2007 3:20 am Post subject: |
|
|
Grand Master
Joined: 02 Aug 2004
Posts: 1046
|
What do you have the NameValueCCSID field in the RFH2 set to? This is the code page of the name/value pairs. If it is 1208 this is probably the cause of the problem. The reason code means that the conversion would expand a fixed length field, which is not allowed.
The CodedCharSetId and Format fields define the data after the RFH2 header. This can expand as much as reuqired, since there is no defined length.
There is no way to convert the msg body, i.e. a later data structure, and leave earlier data structures unconverted.
_________________
MQSeries.net helps those who help themselves.. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri May 25, 2007 10:34 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| halbe01 wrote: |
I have doubled checked, and I do set all these correctly. Basically the various fields in the MQMD state that RFH2 header comes next, and that it is in a single-byte format, and the various fields in the RFH2 state that the next section of the message is an "MQSTR" i.e. the message body and that this is in CCSID of 1208 (UTF8).
So the message itself looks fine to me, however when I perform the Get with convert, I still get this MQRC_CONVERTED_STRING_TOO_BIG message - although this is actually a warning not an error. And the conversion doesn't happen.
I'm still wondering if this is supposed to be possible - can it be that you can't convert a message body to UTF16 just because an attached header can't be converted? Is it not possible to convert just the body? I can't see a way...
Many Thanks,
Ben. |
This is absolutely normal behavior. Remember that your buffer size in UTF-16 needs to be about double the buffer size for a UTF-8 message. (Going from a single byte charset (for non special chars) to a multiple byte char set (2 byte) here)
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| halbe01 |
| Posted: Mon May 28, 2007 9:27 am Post subject: |
|
|
Novice
Joined: 02 Feb 2005
Posts: 16
|
| Quote: |
| There is no way to convert the msg body, i.e. a later data structure, and leave earlier data structures unconverted. |
Well, that is at least an answer!
The requirement is not one I've come across before, and I've been doing MQ design and implementation for a few years now, however I remain somewhat surprised at the restriction. I guess the need to convert a message body to a double-byte character set and also include headers which can only be single-byte is fairly rare.
Many thanks for the help, from everyone. (Unless someone else would like to disagree of course..!)
Regards,
Ben. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon May 28, 2007 3:12 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Nigelg wrote: |
What do you have the NameValueCCSID field in the RFH2 set to? This is the code page of the name/value pairs. If it is 1208 this is probably the cause of the problem. The reason code means that the conversion would expand a fixed length field, which is not allowed.
The CodedCharSetId and Format fields define the data after the RFH2 header. This can expand as much as reuqired, since there is no defined length.
There is no way to convert the msg body, i.e. a later data structure, and leave earlier data structures unconverted. |
You mean to say that if I have the RFH in ccsid 1208 and the body in cssid 1208 and try a get with convert I would first have to get rid of the RFH?.
I thought all I needed was a big enough buffer (i.e. twice the message length including twice the header length)....
I am confused now...
I thought halbe's previous message just meant that he had failed to allocate a big enough buffer and as such the conversion failed because not all the text could have been converted....
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
