| Author |
Message
|
| sunny_30 |
 Posted: Mon Jun 08, 2009 6:54 pm Post subject: Data conversion question |
 |
|
Master
Joined: 03 Oct 2005
Posts: 258
|
We have a sender application (written in C) that processes files to queue with MQMD.Format set to MQFMT_STRING.
The receiver application (also written in C) had MQGMO_CONVERT enabled while doing MQGET (queue to file)
For this reason, we ran into data-conversion problems when the file contained special characters.
In order to fix this, MQGMO_CONVERT on MQGET is disabled in the Receiver app. But the sender is still continuing to send the data in MQFMT_STRING (character) format.
In order to ensure data is never manipulated, do we also need to change the format at sender to "bytes" i.e MQFMT_NONE (or) just the disabling of MQGMO_CONVERT at receiver going to be sufficient? |
|
| Back to top |
|
 |
| gbaddeley |
| Posted: Mon Jun 08, 2009 7:21 pm Post subject: Re: Data conversion question |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| sunny_30 wrote: |
We have a sender application (written in C) that processes files to queue with MQMD.Format set to MQFMT_STRING.
The receiver application (also written in C) had MQGMO_CONVERT enabled while doing MQGET (queue to file)
For this reason, we ran into data-conversion problems when the file contained special characters. |
What problem did you run into? We may be able to help you out.
| Quote: |
In order to fix this, MQGMO_CONVERT on MQGET is disabled in the Receiver app. But the sender is still continuing to send the data in MQFMT_STRING (character) format.
In order to ensure data is never manipulated, do we also need to change the format at sender to "bytes" i.e MQFMT_NONE (or) just the disabling of MQGMO_CONVERT at receiver going to be sufficient? |
Disabling MQGMO_CONVERT is usually sufficient. If the message is being transported across a MQ channel the Message Channel Agent will convert (aka manipulate) the data if the CONVERT(YES) parameter is set on the channel. MQ Exits can also manipulate date.
_________________
Glenn |
|
| Back to top |
|
|
| aditya.aggarwal |
| Posted: Mon Jun 08, 2009 8:56 pm Post subject: |
|
|
Master
Joined: 13 Jan 2009
Posts: 252
|
Hi Sunny_30,
What is the CCSID of Sender and Recevier queue managers? What is the O/S types?
What kind of Data Format you are expecting at Recevier?
You can stop the data conversion across Channels by specifying the CONVERT=NO at Sender channel.
Cheers,
Aditya |
|
| Back to top |
|
|
| sunny_30 |
| Posted: Tue Jun 09, 2009 6:33 am Post subject: |
|
|
Master
Joined: 03 Oct 2005
Posts: 258
|
Thanks you for the responses.
We have file-connectors on both ends of MQ (files move between windows/ linux / aix systems). No data conversion is needed for transfer, it needs to be strictly binary.
The sender-connector was sending the data as 'character'(MQFMT_STRING) but we didnt realize the receiver was converting the data until one day some special-characters failed to convert (cant look into the connector code). So as an immediate fix the receiver-connector was provided a switch on start to disable or enable convert.
Even when the convert was set to 'no' i.e MQGMO_CONVERT get is disabled, the sender was still continuing to send the data as 'character' only not as binary. The conversion is disabled on all MQ-channels. I wasnt sure if this setup behaves exactly as 'binary' transfer, wanted to confirm in the forum. Looking at the response from gbaddeley, it looks if unless the channels or receiver-connector is doing some conversion, the data will not transform even though the transfer is in 'character' format.
what abt if at the receiver-connector the convert is set to ON.
are there any restrictions for ccsid at sender to be 1208 (utf-8 ) ?
note that the data can be anything not just xml.
in case if data is sent as utf-8, should there be anything else done at the receiver apart from setting the convert to 'ON' ?
such as enabling utf-8 conversion to local-os format etc? |
|
| Back to top |
|
|
| aditya.aggarwal |
| Posted: Tue Jun 09, 2009 9:32 pm Post subject: |
|
|
Master
Joined: 13 Jan 2009
Posts: 252
|
Sunny_30 wrote:
| Quote: |
| No data conversion is needed for transfer, it needs to be strictly binary. |
Your need to code your application to handle the binary data in MQGMO_CONVERT. While using data conversion by specifying MQGMO_CONVERT in the get message options on an MQGET, you must specify a UNICODE CCSID in the md in MQGET call.
Sunny_30 wrote:
| Quote: |
| The sender-connector was sending the data as 'character'(MQFMT_STRING) but we didnt realize the receiver was converting the data until one day some special-characters failed to convert (cant look into the connector code) |
If you require specific handling of “special” or "unkonwn" characters, you may need to perform the data conversion within your application using conversion tables and algorithms that satisfy your requirements.
Sunny_30 wrote:
| Quote: |
| what abt if at the receiver-connector the convert is set to ON. |
You can set CONVERT ON at RCVR channel.
Sunny_30 wrote:
| Quote: |
are there any restrictions for ccsid at sender to be 1208 (utf-8 ) ?
note that the data can be anything not just xml.
in case if data is sent as utf-8, should there be anything else done at the receiver apart from setting the convert to 'ON' ?
such as enabling utf-8 conversion to local-os format etc? |
Support for UTF-8 in WebSphere MQ is limited to those Unicode characters that may be encoded in UCS-2.
All WebSphere MQ products will pass messages in any encoding (including any UNICODE encoding) as data, but not all can convert messages from UNICODE to other platform specific encodings.
it is also not possible to use UTF-8 as the queue manager CCSID because UTF-8 is based on ASCII. If you want to pass user data in UNICODE you must always specify a UNICODE CCSID in the message descriptor (md) in an MQPUT call.
Cheers,
Aditya |
|
| Back to top |
|
|
| aditya.aggarwal |
| Posted: Tue Jun 09, 2009 9:35 pm Post subject: |
|
|
Master
Joined: 13 Jan 2009
Posts: 252
|
| Quote: |
| You can set CONVERT ON at RCVR channel. |
You can not set CONVERT ON at RCVR channel.
Cheers,
Aditya |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Wed Jun 10, 2009 4:27 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
As long as you don't set MQGMO_CONVERT on the MQGET, and CONVERT is not activated on the channels, and the Message Data doesn't contain MQ Headers, it is treated by MQ as binary (ie. it does not manipulate the data).
However, "binary" character data that is put in a message by an app on one system (with a particular native CCSID) may not be intepreted as the same characters by an app on another system (that has a different CCSID).
A classic example is a z/OS app (using CCSID 37 internally and may or may not be using MQFMT_STRING) sending an EBCDIC character data message to another z/OS app (using CCSID 500 internally but not using MQGMO_CONVERT) via MQ (on same system or different system). There are several well known printable characters that have different EBCDIC codes in these CCSIDs and this scenario may result in apparently screwed characters. MQ is not to blame in this instance.
_________________
Glenn |
|
| Back to top |
|
|
| sunny_30 |
| Posted: Thu Jun 11, 2009 12:18 pm Post subject: |
|
|
Master
Joined: 03 Oct 2005
Posts: 258
|
| Thank you both. Your responses are very informative & helpful. |
|
| Back to top |
|
|
|
|
