| Author |
Message
|
| paulgroo |
 Posted: Mon May 28, 2007 2:03 am Post subject: CCSID Conversion |
 |
|
Centurion
Joined: 07 Jul 2005
Posts: 138
Location: Ireland
|
Hello everyone!
Im having a bit of an issue with converting message data in an application and I was wondering if some of you might be able to help please?
First of all, the application is sending XML messages from iSeries (QMgr CCSID:37) to Windows (QMgr CCSID:437).
Certain messages are then posted to url but are failing due to incorrect characters, (example below).
<Description>Spec.lak(perleÃ…Â¥,metalÃÂza) - stÅ™ÃÂbrná Lightning</Description>
Is there a way in which I can convert these messages to a suitable CCSID (1208, I think should do the trick)?
I've messed about with ESQL before but I'm having difficulty getting it to exactly what I want it to do. I can post up my ESQL effort if needed!
Any help would be greatly appreciated! |
|
| Back to top |
|
 |
| marcin.kasinski |
| Posted: Mon May 28, 2007 3:39 am Post subject: Re: CCSID Conversion |
|
|
Sentinel
Joined: 21 Dec 2004
Posts: 850
Location: Poland / Warsaw
|
You can get message with MQGMO_CONVERT get option.
With this option QMGR will converts message into your get application CCSID.
_________________
Marcin |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Mon May 28, 2007 4:33 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
I remember seeing something recently about an issue with EBCDIC XML files... but I don't remember what it said. I was searching IBMs website for a completely unrelated ProbeID.
Regardless, Broker should handle this just fine - assuming that a) all the characters you send are correctly in the originating codepage (37) and b) that there is a proper and well defined translation between that codepage and the one Broker uses internally (1200? 1208?).
Looking at your data, I halfway suspect that it's not pure EBCDIC, or at least pure CCSID 37.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| paulgroo |
| Posted: Mon May 28, 2007 8:08 am Post subject: |
|
|
Centurion
Joined: 07 Jul 2005
Posts: 138
Location: Ireland
|
Thank you very much for your replies!
It seems that you are both correct. Marcin, I have tried using the conversion (using both XML and BLOB) on the 'Input Node' and I'm still getting weirdo characters. I think this is as a result of what Jeff is saying about the whole message not being fully EBCDIC or fully CCSID37.
I've been trying tto get the Broker to read the message in as CCSID 1208 by specifying a few different configuration changes. I've tried specifying the Message Domain as both BLOB and XML, and setting the CCSID to 1208 and Convert Encoding to 546. But it doesnt seem to want to work at all.
Is there a way that I can "force" the Broker to read the whole message in through the Input Node as a CCSID1208 message perhaps? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon May 28, 2007 9:00 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
- First you need to identify correctly the source CCSID (37/500/language specific EBCDIC).
- Make sure that the input message carries the CCSID corresponding to its data
- Set outgoing CCSID (1208=UTF-8 ) and Encoding on the OutputRoot.Properties of the outgoing message
- If the incoming/outgoing message is in BLOB format you will have to translate the BLOB to Char and back to make sure you have the right CCSID in the output. This assumes that no other parsing of the BLOB happens in the flow between MQInput and MQOutput nodes.
- if reading with CCSID 437 with convert you need to make sure that all the chars in the message are displayable in ccsid 437... You might have to change to code page 850 or something more country specific...
Finally remember that UTF-8 is a multi byte char representation language and you should not do any cobol/positional parsing on a UTF-8 bitstream ....
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Mon May 28, 2007 9:19 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Really though, if the message is not all in one codepage, you're hosed.
Broker will "just work" as long as the input message is properly identified.
If you know that the entire contents of a particular XML tag will be in a different code page, then you can look at the options on the MRM-XML parser. You might be able to indicate that the contents of those particular tags should be treated as opaque (i.e. binary) elements, and then use CAST to convert them.
Otherwise, it's entirely likely that you a) are out of luck, and b) the sending application is producing invalid XML data. I don't believe that the XML spec allows you to mix codepages. If you say you're in UTF-8, then it should all be in UTF-8.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| paulgroo |
| Posted: Mon Jun 11, 2007 7:11 am Post subject: Hi again |
|
|
Centurion
Joined: 07 Jul 2005
Posts: 138
Location: Ireland
|
Thanks for the feedback everyone!
Sorry if I'm sickening everyone with this problem but I think I might have made a little bit of headway (although its still not fixed). I just wanted to give a quick update of what I've found and see if anyone has anything to add perhaps?
On the iSeries side, the queue manager is CCSID 37. On the Windows side, the queue manager is CCSID437. The application database on the iSeries is formatted as UTF-8. When I send messages across onto the Windows machine and look at them on the queue before they are processed (by the Broker), the CCSID is 819 (?!!?). (I dont have the sender channel doing any 'Convert').
So I've tried getting the Input Node on the Broker to do some work by setting the "Convert Encoding" to 1208 and the "Convert Coded Character Set ID" to 852.
I'm still getting these wonky characters though... (they should be czech format though).
<Description>Spec.lak(perleÃ…Â¥,metalÃÂza) - stÅ™ÃÂbrná Lightning</Description> |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Mon Jun 11, 2007 11:06 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
1) Confirm that all of the data is in the SAME code page.
2) Confirm that the data is in the same codepage as specified in the MQMD
3) Confirm that there are valid translations between the characters you need in each codepage
4) Confirm that the message indicates that the data is String data.
5) Remember that broker translates everything in the logical tree into Unicode (either 1200 or 1208, I forget)
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| kimbert |
| Posted: Mon Jun 11, 2007 12:13 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Broker tree is in codepage 1200, endianness matches platform endianness. |
|
| Back to top |
|
|
| paulgroo |
| Posted: Thu Jun 14, 2007 5:10 am Post subject: |
|
|
Centurion
Joined: 07 Jul 2005
Posts: 138
Location: Ireland
|
Okay, I have one last question on this (well...one or two)...
This is purely from an MQ perspective.
When the iSeries queue manager (CCSID 37) posts a messages to the Windows Queue manager (CCSID 1208), the message that lands on the Windows queue has a CCSID of 819.
1.
Is this because it's picking up some sort of CCSID setting from the OS?
2.
In order for it to post the message in 1208, do I have to do a 'message convert' on the iSeries sender channel?
(By using WMQTool, when I send the message from iSeries and specify 1208, it arrives perfectly on the Windows side).
I just can't figure out where it's getting 819 from, or where to change it to send as 1208 by default.
Once again, lads, thank you all very much.
(Jeff, I think I owe you a few pints at this stage...) |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Jun 14, 2007 3:10 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| paulgroo wrote: |
Okay, I have one last question on this (well...one or two)...
This is purely from an MQ perspective.
When the iSeries queue manager (CCSID 37) posts a messages to the Windows Queue manager (CCSID 1208), the message that lands on the Windows queue has a CCSID of 819.
1.
Is this because it's picking up some sort of CCSID setting from the OS? |
No this is because it got the CCSID of the sending qmgr... or because the CCSID on the message was set to 819.
| paulgroo wrote: |
2.
In order for it to post the message in 1208, do I have to do a 'message convert' on the iSeries sender channel? |
No, all you should need to do is set the CCSID on the message in the MQMD before sending the message. Of course the content of the message should then be in the CCSID specified on the MQMD.
| paulgroo wrote: |
(By using WMQTool, when I send the message from iSeries and specify 1208, it arrives perfectly on the Windows side).
I just can't figure out where it's getting 819 from, or where to change it to send as 1208 by default.
Once again, lads, thank you all very much.
(Jeff, I think I owe you a few pints at this stage...) |
To send the message as 1208 as default you need to put the text to the message in CCSID 1208 and set the CCSID on the MQMD to 1208 before the put.
Using JMS there is also the possibility to set a queue's CCSID in JNDI.
The message will then be converted by the qmgr to the CCSID specified in the JNDI before the put is done and the JNDI CCSID will be set on the MQMD (just before the put.) This is internal to the send operation....
There is of course also the possibility of setting the CCSID on your qmgr / qcf object to signify to the qmgr the default CCSID your text will be in. You may want to check the manuals but I believe in this case the qmgr will translate from the given CCSID to it's own CCSID during the put operation...
Last and not least to be entirely clear this only applies to messages having their format set to MQFMT_STRING.
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
