| Author |
Message
|
| vijeer2001 |
 Posted: Tue Mar 09, 2010 6:38 pm Post subject: |
 |
|
Novice
Joined: 08 Mar 2010
Posts: 16
|
Apart from MQCCSI_Q_MGR, we also tried explicitly passing different CCSID (like 1208, 819, 1250 etc) but got the same result.
We are stuck and not able to move forward because of this issue. Interfaces are sending files with all european characters, some of which are getting converted. |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Tue Mar 09, 2010 7:15 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9470
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| .Net application used to put the file on MQ. Data read correctly by .Net app, but when put on MQ, some of the special characters gets converted. |
Let me see if I understand this...
The FTP is successful - you have used a dump/display utility to look at the file in the filesystem, and have verified that the x'8a' exists correctly in the file.
The .Net app (successfully) reads the file in the filesystem... and you verified this by inserting some code in the .NET app to dump/display the data as the app reads it?
... and the .NET app MQPUTs the message data to the queue. When you use a dump/display utility (like amqsbcg) what was (in the filesystem) a x'8a' is now a x'60'? So this (the display of the message as it sits in the queue) is the very first place where the x'8a' has become a x'60'? Have I understood the problem correctly?
Also, what MQ version/release/maintenance level is qmgr where this problem exists? (I ask because I've seen some code-page ptfs on older mq versions.)
[edit]
What o/s version/level for this box?
What locale (a windows thing) for this box?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Mar 09, 2010 8:26 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| bruce2359 wrote: |
... and the .NET app MQPUTs the message data to the queue. When you use a dump/display utility (like amqsbcg) what was (in the filesystem) a x'8a' is now a x'60'? So this (the display of the message as it sits in the queue) is the very first place where the x'8a' has become a x'60'? Have I understood the problem correctly?
|
But when you perform the same MQPUT using the q utility the value in the queue is indeed correct?
And you still think this is a WMQ problem not a .NET one?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Mar 09, 2010 8:36 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
( 4)
It does not even need to be a .NET problem. It could just be an application problem.
You must understand that .NET's internal representation is Unicode. Now you want to put the Data to MQ as UTF-8. You can set UTF-8 on the message but that does not make the message content be UTF-8. Some methods for text messages may translate (especially using XMS/JMS). Otherwise YOU as a programmer have to ensure that the content (payload) being written is in the correct CCSID...
Get back to the thinking board and experiment some.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Mar 09, 2010 9:21 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9470
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| you can set UTF-8 on the message but that does not make the message content be UTF-8. |
For clarity: the CCSID in the MQMD (message descriptor) does nothing to, and does not change anything in the MQMD or the application payload.
The MQMD contains information about the application data; but that information is taken from what the qmgr knows, and what the application programmer puts in the MQMD.
CCSID is only used when conversion is requested by the consuming application (MQGMO_CONVERT), OR the Sender channel attribute CONVERT(YES) is specified.
Your app does not request conversion; and your message did not traverse an MQ message channel. Therefore, it is an application coding problem.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Wed Mar 10, 2010 3:49 am Post subject: Re: MQ - Having issue in processing special character |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| vijeer2001 wrote: |
| The data CCSID of the data is 1250, but as we do not want any conversion on queue, we intentionally pass CCSID as MQCCSI_Q_MGR i.e. 1208. |
Maybe conversion is not the right word here. Conversion typically replaces the bytes that represent a given character in one coded character set with bytes that represent the same character in a different coded character set. For example:'Å ' -> X'8A' (ccsid=1250) -> convert -> X'C580' (ccsid=1208) -> 'Å ' The only way to preserve the character data, where the input file is Windows Latin-2, but the output message is UTF-8, is through conversion.
| vijeer2001 wrote: |
| Code: |
mqMsg = new MQMessage();
mqMsg.MessageFlags = MQC.MQMF_LAST_SEGMENT;
mqMsg.CharacterSet = MQC.MQCCSI_Q_MGR;
mqMsg.Format = MQC.MQFMT_STRING;
mqMsg.WriteBytes(alMessage[intIdx].ToString()); |
|
WriteBytes() is definitely the wrong method to use in this case:
| Quote: |
public void WriteBytes(String s)
Writes the string to the message buffer as a sequence of bytes. Each character in the string is written in sequence by discarding its high eight bits. |
WriteBytes() would corrupt 'Å ':'Å ' -> X'8A' (ccsid=1250) -> ToString() -> U+160 -> WriteBytes() -> X'60' (ccsid=1208) -> '`' WriteString(), however, would convert 'Å ':'Å ' -> X'8A' (ccsid=1250) -> ToString() -> U+160 -> WriteString() -> X'C580' (ccsid=1208) -> 'Å ' |
|
| Back to top |
|
|
| vijeer2001 |
| Posted: Wed Mar 10, 2010 6:30 am Post subject: |
|
|
Novice
Joined: 08 Mar 2010
Posts: 16
|
Yes, we did check
1- Data recieved on FTP,
2- .Net Data Dump just before Write to MQ
3- & on the Queue directly
The hex values changes only on (3).
Also, we tried doing WriteString along with the CCSID of the data, i.e. 1250 and it does work and show the right character. Thanks a lot. On the queue I see same 1450 bytes (same as input file).
Now we have a new issue. As Q Manager CCSID is 1208, IBM WDI 3.3 is trying to do conversion is failing with 2190 error (because WDI 3.3 is trying to do conversion and probably failing as length is going beyond).
If we use WriteString with 1208, on the queue itself I see more bytes (1466 against 1450 actually in the input file) and again WDI 3.3 is not processing it correctly. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Mar 10, 2010 6:51 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
The CCSID of the message must match the CCSID of the data in the message.
If WDI is trying to do conversion and failing when the CCSID is correct, then either change WDI to not do conversion or troubleshoot why the conversion is failing. |
|
| Back to top |
|
|
| vijeer2001 |
| Posted: Wed Mar 10, 2010 7:55 am Post subject: |
|
|
Novice
Joined: 08 Mar 2010
Posts: 16
|
We are trying to figure that out on WDI side.
But again, using Q Utility or RFHUTIL, we just plainly pick the file and drop to Queue, and everything works perfectly. Special characters are not converted, plus WDI is able to pick and process the data without any issues (the data by default is 1208 using these utilities and stays same bytes as input on the queue). Wondering what is heppning withing these C code, which I am not able to replicate in .Net code. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Wed Mar 10, 2010 7:37 pm Post subject: Re: MQ - Having issue in processing special character |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| vijeer2001 wrote: |
| Also, we tried doing WriteString along with the CCSID of the data, i.e. 1250 and it does work and show the right character. Thanks a lot. On the queue I see same 1450 bytes (same as input file). |
Problem solved. ... Right?
| vijeer2001 wrote: |
| Now we have a new issue. As Q Manager CCSID is 1208, ... |
Why should this matter? The message data ccsid can be (and often should be) unrelated to the qmgr ccsid.
| vijeer2001 wrote: |
... IBM WDI 3.3 is trying to do conversion is failing with 2190 error (because WDI 3.3 is trying to do conversion and probably failing as length is going beyond).
If we use WriteString with 1208, on the queue itself I see more bytes (1466 against 1450 actually in the input file) and again WDI 3.3 is not processing it correctly. |
Converting a string from a single-byte character set (ccsid=1250), to a multi-byte character set (ccsid=1208) can increase the number of bytes in the message. Applications need to account for that.
| vijeer2001 wrote: |
| (the data by default is 1208 using these utilities and stays same bytes as input on the queue). |
At least one of those statements is wrong.
If the .NET app converts the data from ccsid=1250 to ccsid=1208, then the output bytes should be different from the input bytes.
If the .NET app does not convert the data, then the output ccsid should be the same as the input ccsid (1250), and the output bytes should be the same as the input bytes.
One other possible scenario is that the .NET app is the only thing working correctly now, and the errors in the other applications are canceling each other out:- Q/RFHUTIL reads the input file, writes the bytes, as-is, to an output queue, but incorrectly sets the MQMD ccsid=1208 instead of ccsid=1250
- WDI enables MQGMO_CONVERT to the default qmgr ccsid=1208, but the MQMD ccsid is already 1208, so MQ does not attempt to convert the data; it's still encoded using ccsid=1250, and the MQMD ccsid is still wrong
- WDI happily ignores the incorrect MQMD ccsid, and processes the data as if it were encoded using ccsid=1250 (which it is) ...
To really disable conversion all the way through, get rid of WriteString() and ToString():
| Code: |
// code to declare and populate alMessage, intIdx not posted
// assuming it correctly reads the input file as bytes
// ...
mqMsg.CharacterSet = 1250; // Windows Latin-2
mqMsg.Format = MQC.MQFMT_STRING; // or MQC.MQFMT_NONE?
mqMsg.Write(alMessage[intIdx]); // or mqMsg.Write(alMessage)? |
Then unset MQGMO_CONVERT in WDI. |
|
| Back to top |
|
|
| vijeer2001 |
| Posted: Thu Mar 11, 2010 9:59 am Post subject: |
|
|
Novice
Joined: 08 Mar 2010
Posts: 16
|
Thanks a ton!!!
mqMsg.Write[Bytes] worked fine and have resolved the issue for us.
Though I am not able to understand why it worked though. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Mar 11, 2010 10:03 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| vijeer2001 wrote: |
| Though I am not able to understand why it worked though. |
Because bytes and strings are handled differently in .NET
In a way they're not in C.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
