| Author |
Message
|
| Ramphart |
 Posted: Tue Sep 15, 2009 2:03 am Post subject: Encoding - Non ASCII Chars - iSeries to Windows platform |
 |
|
Disciple
Joined: 21 Jul 2004
Posts: 150
Location: South Africa, JHB
|
Hi,
- Windows Platform
- Message Broker 6.1 (Ver 6.1.0.1)
- Toolkit 6.1.0
- Platform: Windows XP Professional
1. I'm getting an input message (CSV) from the iSeries (Webshpere MQ 5.3), Data Conversion is set on the channel to "Conversion", and it lands on a QManager on a windows machine (Websphere MQ Ver 6.0.2.3)
2. MQMD Properties when I get the message
| Code: |
(0x01000000):MQMD = (
(0x03000000):Encoding = 546
(0x03000000):CodedCharSetId = 437
(0x03000000):Format = 'MQSTR '
(0x03000000):Version = 2
|
3. The input message contains non ASCII characters like:
4. I'm mapping the message to an XML output message which I've defined as a message set, MRM Domain, XML1 Wire Format. I've set the following under "XML Declaration" for the message set:
| Code: |
XML Version = 1.0
XML Encoding = As document text |
5. My output message is as follows: (note the encoding)
| Code: |
<?xml version="1.0" encoding="ibm-437_P100-1995"?>
<CustomerList xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<CustomerDetailList>
<AddressLine2>RUA EMILIO M¿BINDI N§</AddressLine2>
<AddressLine3>38,ALVALADE</AddressLine3>
</CustomerDetailList>
</CustomerList> |
6. The message in 5. above does parse properly into the client application due to the non ascii chars. I think I need the encoding="Windows-1252" or encoding="ISO-8859-1"
7. What changes do I need to make to either the message set? or the eSQL to generate the proper encoding on the XML output message? I'm not sure if this is relevant, but as mentioned before , I'm using the MRM domain [not the XMLNSC domain]. Output required:
| Code: |
<?xml version="1.0" encoding="Windows-1252"?>
<CustomerList xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<CustomerDetailList>
<AddressLine2>RUA EMILIO M¿BINDI N§</AddressLine2>
<AddressLine3>38,ALVALADE</AddressLine3>
</CustomerDetailList>
</CustomerList> |
Thanks in advance for assistance
_________________
Applications Architect |
|
| Back to top |
|
 |
| aditya.aggarwal |
| Posted: Tue Sep 15, 2009 4:41 am Post subject: |
|
|
Master
Joined: 13 Jan 2009
Posts: 252
|
You need to set the MQMD encoding and CCSID value correctly in your code.
Cheers,
Aditya |
|
| Back to top |
|
|
| Ramphart |
| Posted: Tue Sep 15, 2009 5:15 am Post subject: |
|
|
Disciple
Joined: 21 Jul 2004
Posts: 150
Location: South Africa, JHB
|
| aditya.aggarwal wrote: |
You need to set the MQMD encoding and CCSID value correctly in your code.
Cheers,
Aditya |
Hi,
I've done that before I posted this issue here. Had the following:
| Code: |
SET OutputRoot.Properties.CodedCharSetId = 1252;
SET OutputRoot.MQMD.CodedCharSetId = 1252;
|
Did not do the trick. What is the exact Encoding and CCIS that I should specify?
Tx
_________________
Applications Architect |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Sep 15, 2009 5:20 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
There was a lot of disussion here 1-2 months ago about this sort of thing.
It is probably worth reviewing them.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Ramphart |
| Posted: Tue Sep 15, 2009 5:50 am Post subject: |
|
|
Disciple
Joined: 21 Jul 2004
Posts: 150
Location: South Africa, JHB
|
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Sep 15, 2009 11:25 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
The bottom of it is that you need to check whether the message you RECEIVE with message broker has the right encoding and CCSID. If this is not the case, what ever you try to do will not fix the problem. 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| aditya.aggarwal |
| Posted: Tue Sep 15, 2009 11:39 am Post subject: |
|
|
Master
Joined: 13 Jan 2009
Posts: 252
|
Ramphrt wrote:
aditya.aggarwal wrote:
You need to set the MQMD encoding and CCSID value correctly in your code.
Cheers,
Aditya
| Quote: |
Hi,
I've done that before I posted this issue here. Had the following:
Code:
SET OutputRoot.Properties.CodedCharSetId = 1252;
SET OutputRoot.MQMD.CodedCharSetId = 1252;
Did not do the trick. What is the exact Encoding and CCIS that I should specify?
Tx |
You should use:
SET OutputRoot.MQMD.Encoding = MQENC_NATIVE;
SET OutputRoot.MQMD.CodedCharSetId = MQCCSI_Q_MGR;
make this change..problem should get resolved. 
Cheers,
Aditya |
|
| Back to top |
|
|
| kimbert |
| Posted: Tue Sep 15, 2009 12:30 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
aditya said:
| Quote: |
You should use:
SET OutputRoot.MQMD.Encoding = MQENC_NATIVE;
SET OutputRoot.MQMD.CodedCharSetId = MQCCSI_Q_MGR;
make this change..problem should get resolved |
You may well be correct, but can you explain why? Would your suggested change fix the problem if the input MQMD was quoting the wrong CodedCharSetId? |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Wed Sep 16, 2009 1:05 am Post subject: Re: Encoding - Non ASCII Chars - iSeries to Windows platform |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| Ramphart wrote: |
| 1. ... Data Conversion is set on the channel to "Conversion" ... |
Why? Channel conversion should be the last resort, not the first. Neither the sending or receiving application have any control over the destination ccsid/encoding, nor can they handle (or even detect) any channel conversion errors. If the receiving application has to perform a second conversion to get the desired ccsid, that's a good clue that the channel conversion is, at best, unnecessary.
| Ramphart wrote: |
2. MQMD Properties when I get the message
| Code: |
(0x01000000):MQMD = (
(0x03000000):Encoding = 546
(0x03000000):CodedCharSetId = 437
(0x03000000):Format = 'MQSTR '
(0x03000000):Version = 2
|
|
What was the ccsid on iSeries, prior to channel conversion? Some conversions (such as ccsid=1252<-->ccsid=437) can be ... lossy. It's best to avoid converting to/from ccsid=437 in those cases.
| Ramphart wrote: |
3. The input message contains non ASCII characters like:
|
... and this is the expected result, correct? Otherwise, there's an entirely different problem in the sending application, that can't be fixed on the receiving side.
| Ramphart wrote: |
4. ... I've set the following under "XML Declaration" for the message set:
| Code: |
XML Version = 1.0
XML Encoding = As document text |
|
And, of course, unchecked the Suppress XML Declaration check box.
| Ramphart wrote: |
| 6. I think I need the encoding="Windows-1252" or encoding="ISO-8859-1" |
If the sending application already uses ccsid=1252, or ccsid=819, then disable the channel conversion, and the receiving application doesn't need to convert at all.
Otherwise, either this:
| Code: |
| SET OutputRoot.Properties.CodedCharSetId = 1252; -- "windows-1252" |
or this:
| Code: |
| SET OutputRoot.Properties.CodedCharSetId = 819; -- "iso-8859-1" |
should be sufficient; double-check that the Compute Mode includes "Message".
| Ramphart wrote: |
| I've done that before I posted this issue here. ... Did not do the trick. |
Be more specific. Put a Trace node after the Compute node. Post the results.
| aditya.aggarwal wrote: |
You should use:
| Code: |
SET OutputRoot.MQMD.Encoding = MQENC_NATIVE;
SET OutputRoot.MQMD.CodedCharSetId = MQCCSI_Q_MGR; |
|
No. The channel conversion already does this. It would be pointless to do it again. It is particularly wrong in this case, as the qmgr ccsid=437, but the desired output ccsid=819 (or 1252). |
|
| Back to top |
|
|
| aditya.aggarwal |
| Posted: Wed Sep 16, 2009 2:16 am Post subject: |
|
|
Master
Joined: 13 Jan 2009
Posts: 252
|
rekarm01 wrote
| Quote: |
Ramphart wrote:
1. ... Data Conversion is set on the channel to "Conversion" ...
Why? Channel conversion should be the last resort, not the first. Neither the sending or receiving application have any control over the destination ccsid/encoding, nor can they handle (or even detect) any channel conversion errors. If the receiving application has to perform a second conversion to get the desired ccsid, that's a good clue that the channel conversion is, at best, unnecessary.
aditya.aggarwal wrote:
You should use: Code:
SET OutputRoot.MQMD.Encoding = MQENC_NATIVE;
SET OutputRoot.MQMD.CodedCharSetId = MQCCSI_Q_MGR;
No. The channel conversion already does this. It would be pointless to do it again. It is particularly wrong in this case, as the qmgr ccsid=437, but the desired output ccsid=819 (or 1252). |
I agree Once should not set conversion=Yes acrsoo the channels. It is not good practise.
After removing this property from the channel below changes need to made in the code as said earlier.
SET OutputRoot.MQMD.Encoding = MQENC_NATIVE;
SET OutputRoot.MQMD.CodedCharSetId = MQCCSI_Q_MGR;
Cheers,
Regards, |
|
| Back to top |
|
|
| Luke |
| Posted: Wed Sep 16, 2009 2:30 am Post subject: |
|
|
Centurion
Joined: 10 Nov 2008
Posts: 128
Location: UK
|
| aditya.aggarwal wrote: |
After removing this property from the channel below changes need to made in the code as said earlier.
SET OutputRoot.MQMD.Encoding = MQENC_NATIVE;
SET OutputRoot.MQMD.CodedCharSetId = MQCCSI_Q_MGR;
Cheers,
Regards, |
Hi Aditya, I'm not convinced that this will help - isn't that going to mean removing conversion from the channel, then doing the same thing on the put from the broker?
As rekarm01 indicated, there may be no need to convert the data at all ...
Did you see Kimbert's earlier post? Can you answer why you think this should be coded?
Thanks |
|
| Back to top |
|
|
| aditya.aggarwal |
| Posted: Wed Sep 16, 2009 5:53 am Post subject: |
|
|
Master
Joined: 13 Jan 2009
Posts: 252
|
Luke Wrote:
| Quote: |
aditya.aggarwal wrote:
After removing this property from the channel below changes need to made in the code as said earlier.
SET OutputRoot.MQMD.Encoding = MQENC_NATIVE;
SET OutputRoot.MQMD.CodedCharSetId = MQCCSI_Q_MGR;
Cheers,
Regards,
Hi Aditya, I'm not convinced that this will help - isn't that going to mean removing conversion from the channel, then doing the same thing on the put from the broker?
As rekarm01 indicated, there may be no need to convert the data at all ...
Did you see Kimbert's earlier post? Can you answer why you think this should be coded? |
You need to go through the Data Conversion document.
Kimbert Wrote:
| Quote: |
aditya said: Quote:
You should use:
SET OutputRoot.MQMD.Encoding = MQENC_NATIVE;
SET OutputRoot.MQMD.CodedCharSetId = MQCCSI_Q_MGR;
make this change..problem should get resolved
You may well be correct, but can you explain why? Would your suggested change fix the problem if the input MQMD was quoting the wrong CodedCharSetId? |
First it need to be ensured that there is no Data Conversion problem because of two different O/S and CCSID.
After changing converT= NO at iserieis SDR channel, user need tto browse the message and check the CCSID and Encoding value in MQMD of incoming message. it should contain the iseries qmgr CCSID and encoding details.
After this in code one need to specify the CCSID and encoding using options told earlier.
This should work. We faced the similar kind of issue and after ensuring that there is no problem in Data Conversion b/w qmgrs we performed these changes in code. issue was resolved.
Ramphart can only verify and confirm all of us after doing this change.
Cheers,
Aditya |
|
| Back to top |
|
|
| Luke |
| Posted: Wed Sep 16, 2009 9:31 am Post subject: |
|
|
Centurion
Joined: 10 Nov 2008
Posts: 128
Location: UK
|
| aditya.aggarwal wrote: |
| You need to go through the Data Conversion document. |
Which one? ... Will it tell me to always code this:
SET OutputRoot.MQMD.Encoding = MQENC_NATIVE;
SET OutputRoot.MQMD.CodedCharSetId = MQCCSI_Q_MGR;
when I have a problem relating to codepages in Message Broker?
Maybe I've lost the plot completely (happy to be told this, if someone can explain where I've gone wrong), but these are my thoughts on it:
Why would it be a good idea to set the MQMD.CodedCharSetId to the default CCSID of the local Queue Manager? How would you know that's the codepage the receiving app wants the messages in? What if the message contains characters that can't be converted to that codepage? What if you want to deploy your flow on another environment where the queue manager may have a different default CCSID?
It might work, or appear to work, in some situations, but I don't think it's a universal solution.
Anyway, FWIW I think rekarm01 made lots of good points, hopefully they will help Ramphart to get to the bottom of his problem. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Wed Sep 16, 2009 7:12 pm Post subject: Re: Encoding - Non ASCII Chars - iSeries to Windows platform |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| Ramphart wrote: |
| 6. The message in 5. above does parse properly into the client application due to the non ascii chars. I think I need the encoding="Windows-1252" or encoding="ISO-8859-1" |
Should that be "does parse properly", or "doesn't parse properly"? Is the problem just that the XML encoding declaration has the wrong value, or is there another problem with the endpoint parsing non-ASCII data?
| aditya.aggarwal wrote: |
After removing this property from the channel below changes need to made in the code as said earlier.
| Code: |
SET OutputRoot.MQMD.Encoding = MQENC_NATIVE;
SET OutputRoot.MQMD.CodedCharSetId = MQCCSI_Q_MGR; |
|
No, this is still wrong:- It would be better to modify OutputRoot.Properties, and let the Properties parser sort out which other headers need updating.
- For Format=MQFMT_STRING, it doesn't really matter what Encoding is set to; Encoding=MQENC_NATIVE is probably better than anything else, but why bother?
- If MQCCSI_Q_MGR resolves to ccsid=437, but the endpoint requires some other ccsid (such as ccsid=1252 or ccsid=819), then setting the output ccsid=437 would violate that requirement.
| aditya.aggarwal wrote: |
| You need to go through the Data Conversion document. |
Regarding the Data Conversion under WebSphere MQ document, the following recommendation is either wrong, or at least misleading:
| Quote: |
Common Procedures when doing the MQPUT
Regardless of whether the message is character, numeric, or a mixture of both, in the application that is doing the initial MQPUT of the message, you should:- Set MQMD.CodedCharSetId to MQCCSI_Q_MGR. ...
|
The reason why it's misleading is buried in the paragraph that follows:
| Quote: |
| ... it is the responsibility of the application to ensure that the implied CCSID correctly represents the string data in the message. |
The application needs to set the message ccsid based on the character encoding of the string data, not on the qmgr ccsid, (unless the application can ensure that the two are one and the same). |
|
| Back to top |
|
|
| WMBEAI |
| Posted: Wed Sep 16, 2009 8:56 pm Post subject: |
|
|
Acolyte
Joined: 02 Mar 2009
Posts: 66
|
Setting the below code along the channel conversion as NO worked.
SET OutputRoot.MQMD.Encoding = MQENC_NATIVE;
SET OutputRoot.MQMD.CodedCharSetId = MQCCSI_Q_MGR;
i have done a similar thing between AIX and Z/OS |
|
| Back to top |
|
|
|
|
