ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » IBM MQ Installation/Configuration Support » Coded Character Set ID for QM - Keep default or change UTF-8

Post new topic  Reply to topic
 Coded Character Set ID for QM - Keep default or change UTF-8 « View previous topic :: View next topic » 
Author Message
mkza.swe
PostPosted: Thu Jun 28, 2012 8:43 am    Post subject: Coded Character Set ID for QM - Keep default or change UTF-8 Reply with quote

Novice

Joined: 04 Jun 2012
Posts: 16

Hi!

I'm using English(UK) windows 2008 server and WMQ V7.

When creating a queue manager there is no option to set the Coded Character Set ID (CCSID). The one that was set by default was 850 (LATIN-1).

We are trying to use UTF-8 for all messages, as that should support most (all?) characters. There could be text (addresses and stuff) to people in most of the world, so there is a need for a comprehensive support of characters.

So, most messages received are UTF-8 (ccsid 1208), but some are ISO8859-1 (ccsid 819), sent by a Unix QM. We send queries in 1208 and receive answer in 819.

So to the questions:
[b]1. [/b]Is there any danger of receiving messages in 819 instead of 1208? The supplier seem a bit reluctant to change to 1208. I can't find out if it is possible to "lose" data because of characters not possible to represent when converting. Is there a list or tool or something where we can find out what characters are not convertable? Convertion need to work both ways since we are sending data in 1208 that are changed to 819 at the receiver, and then back to 1208 when we receive it.

[b]2. [/b]Does it matter what CCSID the Queue Manager is set to? There is no risk to lose characters cause the QM is set to the wrong CCSID? As long as the MQMD Codepage field is set, my understanding is that the QM setting doesn't matter. Or does this also apply to the encoding that log files and error logs etc are written in, so that some characters may not be visible there? It somehow feels better to set the QM to 1208, but could that cause other problems? The infocenter tells me to use something that is "appropriate for my platform", which doesn't really help me at all.

BTW, Good reference for CCSID to understandable names:
http://www-01.ibm.com/software/globalization/ccsid/ccsid_registered.html
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Jun 28, 2012 8:56 am    Post subject: Re: Coded Character Set ID for QM - Keep default or change U Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

mkza.swe wrote:
When creating a queue manager there is no option to set the Coded Character Set ID (CCSID).


So why not change it post-creation?

It also has nothing to do with the CCSID used for any given message unless the putting application chooses to use it.


We are trying to use UTF-8 for all messages, as that should support most (all?) characters. There could be text (addresses and stuff) to people in most of the world, so there is a need for a comprehensive support of characters.

mkza.swe wrote:
So, most messages received are UTF-8 (ccsid 1208), but some are ISO8859-1 (ccsid 819), sent by a Unix QM. We send queries in 1208 and receive answer in 819.


Messages sent from other queue managers can't reference the CCSID setting on the target queue manager. I repeat my comment above regarding the explicit use of CCSID, especially important if you're using .NET.

mkza.swe wrote:
Is there any danger of receiving messages in 819 instead of 1208? The supplier seem a bit reluctant to change to 1208. I can't find out if it is possible to "lose" data because of characters not possible to represent when converting.


If the supplier is confident that everything they send fits into the 819 code page, then no, there's no danger.

mkza.swe wrote:
Is there a list or tool or something where we can find out what characters are not convertable?


The Internet is full of code page tables. Mr Google will find them for you.

mkza.swe wrote:
Does it matter what CCSID the Queue Manager is set to?


Only if it's used. See my comments above.

mkza.swe wrote:
Or does this also apply to the encoding that log files and error logs etc are written in, so that some characters may not be visible there?


What's going to be written into error or log files from the messages? Clearly if you've set the "nationality" of the queue manager to be a language you don't have a code page for that's going to be an issue, but why would you do that?

mkza.swe wrote:
The infocenter tells me to use something that is "appropriate for my platform", which doesn't really help me at all.


I think that's very good advice.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Thu Jun 28, 2012 1:48 pm    Post subject: Re: Coded Character Set ID for QM - Keep default or change U Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

mkza.swe wrote:
The infocenter tells me to use something that is "appropriate for my platform", which doesn't really help me at all.

BTW, Good reference for CCSID to understandable names:
http://www-01.ibm.com/software/globalization/ccsid/ccsid_registered.html

Read this as set the CCSID of the qmgr to a value that you have a code page installed for on your system......
As for UTF-8... If you want to read all your messages in UTF-8, supply CCSID 1208 to the connection factory / message header before the get and do a get with convert...
Note that this only works on text messages (MQFMT_STRING)...

For the rest read the excellent advice given by Vitor
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
rekarm01
PostPosted: Sun Jul 01, 2012 6:18 pm    Post subject: Re: Coded Character Set ID for QM - Keep default or change U Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 1415

The primary purpose of the qmgr ccsid is to read/write the character data in the MQMD header itself. The applications that put/get messages are free to choose any other supported ccsid to read/write the message data.

Vitor wrote:
The Internet is full of code page tables. Mr Google will find them for you.

Though be aware that some other vendors number some of their code pages differently, and even IBM distinguishes between its own ccsids and codepage ids, and sometimes numbers them differently too.
Back to top
View user's profile Send private message
mkza.swe
PostPosted: Mon Jul 09, 2012 6:19 am    Post subject: Re: Coded Character Set ID for QM - Keep default or change U Reply with quote

Novice

Joined: 04 Jun 2012
Posts: 16

I just noticed something weird... totally by accident.

I am sending messages from one queue manager to another. The message that I put on the remote-queue-definition is encoded in UTF-8 (ccsid 1208) and the MQMD codepage is set to 1208.

Channels between the servers are stopped, and when browsing the messages on the transmission queue I see that they are set as MQMD codepage 850 (same as the queue manager is created with). It seems like they arrive as 1208 on the other queue manager though.

This makes me confused however. Is MQ converting the encodings by itself in the channel transport? I thought the encoding would me maintained... What if I would have had a foreign character not supported by 850 (which is why I elected to use UTF-8 to start with), would there be trouble then? This kind of suggests that the queue manager ccsid might not be unimportant?
Back to top
View user's profile Send private message
Vitor
PostPosted: Mon Jul 09, 2012 6:22 am    Post subject: Re: Coded Character Set ID for QM - Keep default or change U Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

mkza.swe wrote:
Channels between the servers are stopped, and when browsing the messages on the transmission queue I see that they are set as MQMD codepage 850 (same as the queue manager is created with). It seems like they arrive as 1208 on the other queue manager though.


Which MQMD are you looking at? The one associated with your data (after the transmission header) or the one the MCA added when it created the transmission header?

I put it to you that you're looking at the MCA's MQMD not yours.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
mkza.swe
PostPosted: Mon Jul 09, 2012 6:40 am    Post subject: Re: Coded Character Set ID for QM - Keep default or change U Reply with quote

Novice

Joined: 04 Jun 2012
Posts: 16

Vitor wrote:
Which MQMD are you looking at? The one associated with your data (after the transmission header) or the one the MCA added when it created the transmission header?

I put it to you that you're looking at the MCA's MQMD not yours.


I see. I was not aware that the MCA added an extra MQMD, but I guess it makes sence.

I am using browse functionality in either MQ Explorer (Coded character set id) or RFHUtil (MQMD tab, Codepage field). I'm not sure how I could see the different MQMD headers, if there are two? I see some additional data beeing added to my body data for the message, like destination queue and queue manager, put application, date etc, but can't find anything that looks like codepage, neither 1208 or 4B8 (which is 1208 hexadecimal).

Anyway... Then I guess it's all in order. No data is re-encoded?!

Cheers!
Back to top
View user's profile Send private message
mqjeff
PostPosted: Mon Jul 09, 2012 6:44 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

Here's the real point.

If the data on the receiving queue at the final destination is the same data as the data you sent, then stop worrying about it.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Mon Jul 09, 2012 6:55 am    Post subject: Re: Coded Character Set ID for QM - Keep default or change U Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

mkza.swe wrote:
I was not aware that the MCA added an extra MQMD, but I guess it makes sense.

The XQH is well documented.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
mkza.swe
PostPosted: Mon Jul 09, 2012 7:00 am    Post subject: Re: Coded Character Set ID for QM - Keep default or change U Reply with quote

Novice

Joined: 04 Jun 2012
Posts: 16

bruce2359 wrote:
The XQH is well documented.


Thank you, sorry for taking up your time...
Back to top
View user's profile Send private message
bruce2359
PostPosted: Mon Jul 09, 2012 7:32 am    Post subject: Re: Coded Character Set ID for QM - Keep default or change U Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

mkza.swe wrote:
bruce2359 wrote:
The XQH is well documented.


Thank you, sorry for taking up your time...

Sarcasm??

I hadn't intended any sarcasm. Rather, I was suggesting further reading/research.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
mkza.swe
PostPosted: Mon Jul 09, 2012 8:40 am    Post subject: Re: Coded Character Set ID for QM - Keep default or change U Reply with quote

Novice

Joined: 04 Jun 2012
Posts: 16

bruce2359 wrote:
mkza.swe wrote:
bruce2359 wrote:
The XQH is well documented.


Thank you, sorry for taking up your time...

Sarcasm??

I hadn't intended any sarcasm. Rather, I was suggesting further reading/research.


Nope, no sarcasm. I understood what you meant :)
Back to top
View user's profile Send private message
mahashan13
PostPosted: Fri Mar 04, 2016 8:39 am    Post subject: CCSID changed from one queue manager to another Reply with quote

Newbie

Joined: 15 Feb 2016
Posts: 4

Hi all,

My problem seems to be related to this thread Source application sends data with CCSID 850, and first flow in WMB does transformation moves the message to another queue manager , where i see message is not properly formed and its failing. I noticed one thing here, CCSID is set to 819 where message is failing, how this CCSID is getting changed when message moved to another q manager ?
Back to top
View user's profile Send private message
mqjeff
PostPosted: Fri Mar 04, 2016 8:51 am    Post subject: Re: CCSID changed from one queue manager to another Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

mahashan13 wrote:
Hi all,

My problem seems to be related to this thread Source application sends data with CCSID 850, and first flow in WMB does transformation moves the message to another queue manager , where i see message is not properly formed and its failing. I noticed one thing here, CCSID is set to 819 where message is failing, how this CCSID is getting changed when message moved to another q manager ?


The CCSID of a message is always set by the application putting the message. This could include the application failing to set the CCSID or using the CCSID of the queue manager rather than of the data in the message.
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
smdavies99
PostPosted: Fri Mar 04, 2016 11:12 am    Post subject: Reply with quote

Jedi Council

Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.

if you explicitly set the CCSID of the message YOU the flow designer/coder are in control. If you don't then you are at the mercy of whoever setup the Queue Manager in the first place. I've even been quizzed by IBM Global Services as to why I felt that I should set the Queue Manager CCSID to say 1208 on an installation. Their reason was that it wasn't in their standard.

A 'Doh!' moment.

I also explicitly set the output CCSID on ALL my flows that go into production. Then I know that the flow will work no matter what 'expert' setup the queue manager in the first place.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995

Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » IBM MQ Installation/Configuration Support » Coded Character Set ID for QM - Keep default or change UTF-8
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.