| Author |
Message
|
| mohan.mmm |
 Posted: Fri Aug 26, 2011 9:01 am Post subject: IBM.WMQ.MQQueue.Put() is removing BOM while sending message |
 |
|
Newbie
Joined: 26 Aug 2011
Posts: 7
|
I am using amqmdnet.dll 1.0.0.2 to send the messages to MQ. I am using VB.Net 1.1.
The problem is it is stripping the Byte Order Mark (BOM) while sending the XML message to MQ. The other application which reads messages from MQ is unable to read the XML message as the BOM is missing. I saw couple of articles about QueueManager object's CodedCharSetId property. But that property is read only and it is not allowing me to change it. I have tried to change the MQMessage object's CharacterSet property but it didn't work. As mentioned in the below thread I have tried 1204 and 1202. But it says 1204 and 1202 are not valid CharacterSets. I have used 1200, 1208 but none of those added the BOM.
View topic "Encoding problem with .NET framework Vs VB script"
I need to post the XML message to MQ with the BOM. Any help would be great. |
|
| Back to top |
|
 |
| gbaddeley |
| Posted: Fri Aug 26, 2011 1:44 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
I don't have any experience with BOMs, but a quick look at http://www.opentag.com/xfaq_enc.htm suggests that BOMs are 2, 3 or 4 binary bytes at the start of a Unicode file. If this file is stored in a MQ message I doubt that MQ will treat this as normal character bytes which can be automatically converted from one CCSID to another.
If the file is put in MQ as a MQFMT_NONE message, MQ will not strip bytes from the start of the message. Maybe .NET is stripping the bytes?
_________________
Glenn |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sat Aug 27, 2011 5:38 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| gbaddeley wrote: |
| f this file is stored in a MQ message I doubt that MQ will treat this as normal character bytes which can be automatically converted from one CCSID to another. |
It should, actually, as the BOM is required to indicate how to understand the rest of the bitstream as characters. I mean, it's either use the BOM or make a guess as to whether you've got big-endian or little-endian data.
And MQ is not in the habit of guessing.
| gbaddeley wrote: |
| If the file is put in MQ as a MQFMT_NONE message, MQ will not strip bytes from the start of the message. Maybe .NET is stripping the bytes? |
It's more likely that .NET is not writing out the data as the correct code page and instead is using 1200 or 1208 without a BOM.
mohan.mmm - are you setting the MQMessage object's CharacterSet property *before* or *after* you WriteString into it? You need to set it before you write into it, so the .NET libraries do the necessary conversion from the internal wchar structures into the right codepage bits.
Also, you're not using WriteUTF(), are you? That's the wrong method to use for *anything*, even UTF data... |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Sat Aug 27, 2011 1:35 pm Post subject: Re: IBM.WMQ.MQQueue.Put() is removing BOM while sending mess |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| mqjeff wrote: |
| It's more likely that .NET is not writing out the data as the correct code page and instead is using 1200 or 1208 without a BOM. |
It's much more likely that any problems are introduced when writing the MQ Message, not when putting it on a queue.
| mohan.mmm wrote: |
| The other application which reads messages from MQ is unable to read the XML message as the BOM is missing. |
Is the reason really because the BOM is missing, or is it because the MQ header ccsid doesn't match the message data? What's the actual error?
| mohan.mmm wrote: |
| I have tried 1204 and 1202. But it says 1204 and 1202 are not valid CharacterSets. I have used 1200, 1208 but none of those added the BOM. |
The only Unicode ccsids that .NET applications support are 1200 and 1208, so the application would need to write out the MQMessage using one of those as the CharacterSet instead.
If it's really necessary, the receiving application can add a BOM by performing a convert on get to ccsid=1204, (unless it's also a .NET application). |
|
| Back to top |
|
|
| mohan.mmm |
| Posted: Mon Aug 29, 2011 12:16 pm Post subject: |
|
|
Newbie
Joined: 26 Aug 2011
Posts: 7
|
The problem is in .Net. I have the xml message in the string variable "<?xml version="1.0" encoding="utf-16"?><LName>mohan.mmm</LName>" .Net understands it as UTF-16, but I need to write this string variable to MQ using MQQueue.Put method. Since the string value doesn't have the BOM the MQQueue.Put method is not adding it. I want to add the BOM and then put it in MQ. Is there any easy way other than adding the BOM bytes to the string using a loop?
I have tried MQMessage.Format to MQC.MQFMT_NONE. But it is still not putting the message with BOM.
The receiving application is also .Net application. So it is failing to read the XML message. It reads the first character and couldn't find the BOM and it fails at that point. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Aug 29, 2011 12:20 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mohan.mmm wrote: |
| Since the string value doesn't have the BOM the MQQueue.Put method is not adding it. |
All strings in .Net are Unicode so it should have a BOM. Are you setting the MQMessage CCSID before calling the put, and if so to what value(s)?
And most important of all - are you using MQMessage.writestring (which writes it as a string) or MQMessage.writeUTF (which writes Unicode)?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mohan.mmm |
| Posted: Mon Aug 29, 2011 12:48 pm Post subject: |
|
|
Newbie
Joined: 26 Aug 2011
Posts: 7
|
I am using MQMessage.writestring with CharacterSet as 1208. string posted without BOM.
If I use MQMessage.writeUTF and CharacterSet as 1208 then it is adding 0xA7 0xF3. It is not adding BOM.
I am setting MQMessage.CharacterSet to 1208. I don't see ccsid property for MQMessage. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Aug 29, 2011 1:03 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mohan.mmm wrote: |
| If I use MQMessage.writeUTF and CharacterSet as 1208 then it is adding 0xA7 0xF3. It is not adding BOM. |
How is that not a BOM?
Are you sure 1208 is the right CCSID?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Mon Aug 29, 2011 1:32 pm Post subject: Re: IBM.WMQ.MQQueue.Put() is removing BOM while sending mess |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| mohan.mmm wrote: |
| I need to write this string variable to MQ using MQQueue.Put method. Since the string value doesn't have the BOM the MQQueue.Put method is not adding it. |
The MQMessage.Write* methods write the message.
The MQQueue.Put method puts the message on the queue.
The MQQueue.Put method does not add anything to, subtract anything from, or otherwise modify the MQMessage in any way.
| mohan.mmm wrote: |
| I want to add the BOM and then put it in MQ. Is there any easy way other than adding the BOM bytes to the string using a loop? |
A loop? To add one character to a String?
The sending application should:- Create MQMessage, set Format=MQC.MQFMT_STRING, set characterSet=1200
- Call MQMessage.WriteString to write xml message
- Call MQQueue.Put to put MQMessage on the queue
The receiving application should:- Call MQQueue.Get to get MQMessage off the queue
- Call MQMessage.ReadString to read xml message
If that does not work as expected, then post just the portion of the code that writes, puts, gets, and reads the message, a hex dump of the message on the queue (including headers), an indication of where/when the error occurred, and the exact text of any error messages. Please use [code] tags when posting code, and try not to post more code than necessary. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Mon Aug 29, 2011 3:00 pm Post subject: Re: IBM.WMQ.MQQueue.Put() is removing BOM while sending mess |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| Vitor wrote: |
| How is that not a BOM? |
That is not a BOM. It is a 2-byte length field that WriteUTF produces, and ReadUTF consumes.
That's why ReadUTF/WriteUTF work better with each other, than with any of the other Read/Write methods. (And partly why many users recommend against using them at all.)
| Vitor wrote: |
| Are you sure 1208 is the right CCSID? |
WriteString converts String->bytes, using the MQMessage CCSID (characterSet), when writing.
ReadString converts bytes->String, using the MQMessage CCSID (characterSet), when reading.
WriteString/ReadString/ReadLine are the only methods that use the MQMessage CCSID (characterSet) to convert when writing/reading the message, which makes them much easier to use with Strings than the other Read/Write methods.
Because of the conversion on both ends, it shouldn't really matter what the CCSID is, as long as it can convert all the characters in the String. |
|
| Back to top |
|
|
| mohan.mmm |
| Posted: Mon Aug 29, 2011 4:38 pm Post subject: |
|
|
Newbie
Joined: 26 Aug 2011
Posts: 7
|
Here is the code I am using to put the MQMessage.
| Code: |
Dim sInputMsg As String
Dim sr As New StreamReader("C:\OUT\JustSavedinNotepad.xml", Encoding.UTF8)
sInputMsg = sr.ReadToEnd()
' sInputMsg variable will have the string "<?xml version="1.0" encoding="utf-16"?><LName>mohan.mmm</LName>"
'--- Inserted this code to just check the HEX values ---
Dim byteArray() As Byte = Encoding.UTF8.GetBytes(sInputMsg)
Dim hexString As String = BitConverter.ToString(byteArray, 0, 10)
Console.WriteLine(hexString)
' -- Output shows as "3C-3F-78-6D-6C-20-76-65-72-73"
Dim byteArrayBOM(byteArray.Length + 2) As Byte
byteArrayBOM(0) = Convert.ToByte(&HFF)
byteArrayBOM(1) = Convert.ToByte(&HFE)
For i As Integer = 0 To byteArray.Length - 1 Step 1
byteArrayBOM(i + 2) = byteArray(i)
Next
hexString = BitConverter.ToString(byteArrayBOM, 0, 10)
Console.WriteLine(hexString)
' -- Output shows as "FF-FE-3C-3F-78-6D-6C-20-76-65"
' -- FF-FE are BOM
'--- Inserted above code to just check the HEX values ---
Dim objMessage As MQMessage
Dim objMessageOptions As MQPutMessageOptions
'I have some tocde to establish connection to queue and get the MQQueue object as objMQQueue
objMessage = New MQMessage
objMessageOptions = New MQPutMessageOptions
objMessageOptions.Options = CType(MQC.MQPMO_NO_SYNCPOINT, Integer)
objMessage.Format = MQC.MQFMT_STRING
objMessage.CharacterSet = 1200
objMessage.Expiry = -1
objMessage.WriteString(CStr(sInputMsg))
objMQQueue.ClearErrorCodes()
objMQQueue.Put(objMessage, objMessageOptions)
'Clear the message before attempting another
objMessage.ClearMessage()
'--- MQMessage in RFHUtil tool shows as 3C003F00 78006D00 6C002000 76006500....
'--- Then I have set sInputMsg = Encoding.UTF8.GetString(byteArrayBOM)
' Because the byteArrayBOM variable has the BOM.
'--- MQMessage in RFHUtil tool shows as FDFFFDFF 3C003F00 78006D00 6C002000.....
|
The receiving application is MQSC adaptor of BizTalk 2010. It internally uses .Net code to pull the message and load in to XMLDocument. We don't have control on how to read the message. If the BOM is absent then it will throw an error message "No Disassemble stage components can recognize the data. ". If I put the message directly using the RFHUtl toool then it will have the BOM and BizTalk is able to recognize the data. So I am trying to see if the .Net applocation can add BOM while pushing the message to MQ. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Aug 29, 2011 7:43 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
You are setting ccsid 1200 on the message. 1200 is a ccsid without a BOM.
If you want a BOM with the text use ccsid 1202 or 1204!
You don't have to use hexString etc... .NET should know it's internal representation of a String and when you use the writeString method MQ will take care of the details.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mohan.mmm |
| Posted: Tue Aug 30, 2011 5:07 am Post subject: |
|
|
Newbie
Joined: 26 Aug 2011
Posts: 7
|
| 1202 and 1204 are not valid CaracterSets for .Net. I am getting MQRC_UNEXPECTED_ERROR exception (rason code 2195) with 1202 and 1204. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Aug 30, 2011 5:24 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Okay, reading your code carefully, you are reading the contents of the file into sInputMsg and then using WriteString to put this to the message.
But first you cast it to a CStr. This doesn't appear to be necessary, at least according to the .NET sample nqmput.cs.
I would a) remove the cast, b) remove the explicit CCSID setting to 1200, c) consider reading the file as Bytes rather than a String and using writeBytes.
If you try c), then make sure you DO explicitly set the CCSID and also make sure you continue to set MQFMT_STRING.
And rekarm01 - there are also some length restrictions on WriteUTF/ReadUTF - since they only ever write a two-byte length, they can never exceed string lengths that would need more than two bytes to describe. So that's the *other* reason I personally recommend that they NEVER be used. |
|
| Back to top |
|
|
| mohan.mmm |
| Posted: Tue Aug 30, 2011 8:35 am Post subject: |
|
|
Newbie
Joined: 26 Aug 2011
Posts: 7
|
Removing the CStr and removing the CCSID setting to 1200 is not helping.
The XML string is constructed in the .Net application. Here I have created a sample application to read the xml string from file. I have tried both the ways. Still no BOM. |
|
| Back to top |
|
|
|
|
