| Author |
Message
|
| speediii |
 Posted: Tue Aug 15, 2006 12:40 am Post subject: EBCDIC/ASCII Conversion |
 |
|
Novice
Joined: 11 Apr 2006
Posts: 19
|
We have a number of MQ Lines sending SWIFT messages that are in both ASCII & EBCIDIC format. I recently wrote a Java broker for routing messages and this works fine, it reads messages and moves them on in the original format (I am not using the MQGMO_CONVERT option and pass on the Encoding and Format fields from the original MQ message).
We have however an external company supplying a component that needs to be in the "mq workflow", but we've found that their component changes the EBCDIC messages into ASCII. We'd like the messages to be left untouched in everyway.
One solution is to put EBCDIC conversion on the MQ Channels from the Maineframe - but I'd like to know what they've done.
This is what they tell me: "...The MQ Connector always makes MQGET with the convert option (MQGMO_CONVERT), so he always get the message in ASCII. He will then put the message in ASCII into the output queues. Your application that needs to receive EBCDIC can then issue a MQGET with the convert option to translate the message in EBCDIC...."
Can someone please explain to me what should be done here. Personally I think they should not be doing CONVERT at all and use readString or similar and put the message back with the original Format and Encoding fields set. They are using the C or C++ interface I think.
Many thanks in advance
Lee |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Tue Aug 15, 2006 1:29 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
It's best to separate sender apps from receiver apps. This means that receiver apps should not make any assumptions about the data - like that the data is EBCDIC or ASCII or anything else.
That is, any app that needs to process text data should always issue the Get with Convert to ensure that the text data is in it's local CCSID.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| speediii |
| Posted: Tue Aug 15, 2006 1:53 am Post subject: |
|
|
Novice
Joined: 11 Apr 2006
Posts: 19
|
Unfortunately the receiver apps in question require the data to be in a specific format (EBCDIC) and this is something we can't change. Therefore, if we want to put applications inbetween in the middleware they must not change the format of the MQ message. So in this case not all receivers use the CONVERT get option.
If this third party software is doing an MQ CONVERT on the get which you say is correct, is the only answer to convert the data back to EBCDIC in my broker program, or use an exit routine or similar?
All I really want is for them to put the message in it's original format like I do with the router program I wrote...
Can't they just move the Format and Encoding fields from the original MQ message when they create the mq message for the PUT?
I totally agree that each program in the MQ flow shouldn't care about the data encoding but if legacy receiving applications expect a certain format this is something we need to maintain. Adding applications inbetween needs to be as transparent as possible and making to many changes increases the chances that we make a mistake during deployment !! |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Aug 15, 2006 2:07 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| speediii wrote: |
| Unfortunately the receiver apps in question require the data to be in a specific format (EBCDIC) |
But EBCDIC is not a "format" - it's text. So if the receiver apps require in in that code page fine, do a get with convert to put it in EBCDIC. If it happens to be in EBCDIC already it's a really quick and simple conversion! 
If this is not done, and you say you have both ASCII & EBCDIC (your original post) then surely it's better for the receiving apps to convert into the code page they're actually using than have a routing program that attempts to ensure the right message in the right code page gets to the right program. This is one of the strength of MQ, and aids the transparent adding of new applications.
Don't be using exit routines to do EBCDIC to ASCII conversions - this is a sledgehammer tied to an anvil to crack a nut. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Aug 15, 2006 2:15 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
If all your legacy apps require that the data be in EBCDIC, then setting the channels to convert incoming messages is not a bad idea.
Their app isn't necessarily doing anything wrong, either.
It sounds like you didn't discuss the interface properly.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Aug 15, 2006 5:58 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
And it looks that you do not have a good understanding of the meaning of get with convert when the message is a text message (format=MQSTR).
On an EBCDIC platform this means that an incoming TEXT message in ASCII is transformed to the EBCDIC code format. The receiving application needs to specify get with convert and notify the qmgr (if it is not in ccsid 37 or 500) that the client is requesting this format. (by setting the ccsid of the qmgr object at instanciation in the program?)
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| tleichen |
| Posted: Wed Aug 16, 2006 7:21 am Post subject: |
|
|
Yatiri
Joined: 11 Apr 2005
Posts: 663
Location: Center of the USA
|
The philosophy of MQSeries, with regard to text type messages is "receiver makes good on the message". Thus, conversion is done if necessary, based on the encoding and codeset of both the message and the receiver. If you go against the grain on this, you're on your own, and you will need to do something that is not entirely seemless. 
_________________
IBM Certified MQSeries Specialist
IBM Certified MQSeries Developer |
|
| Back to top |
|
|
| speediii |
| Posted: Thu Aug 17, 2006 2:48 am Post subject: |
|
|
Novice
Joined: 11 Apr 2006
Posts: 19
|
If I had a good understanding of this then I wouldn't need to ask these questions on an MQ Forum 
Maybe I didn't explain this very well but what I was trying to understand is: How is it that my program manages to read MQ messages OK and put them back (route them forwards) in their original format so the destinations don't complain (I don't use CONVERT on the get, just Java readString & writeString), and yet the third party app we've just plugged in is transforming all messages it PUTs forward into ASCII?"
If I've understood what's been said then MQ takes care of the conversion for you on the GET, so what the third party app is doing wrong is that they are not moving the CharacterSet & Encoding fields from the original MQ message to the new MQ message they PUT forwards (Input & Output queue workflow)...
I presume also that as the format is MQSTR providing the characterSet and Encoding is set the receiving app will convert the message "if necessary". |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Aug 17, 2006 3:00 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| speediii wrote: |
If I've understood what's been said then MQ takes care of the conversion for you on the GET, so what the third party app is doing wrong is that they are not moving the CharacterSet & Encoding fields from the original MQ message to the new MQ message they PUT forwards (Input & Output queue workflow)...
|
What they should be doing is setting the CharacterSet & Encoding fields to match those used to create the message. So if they're writing ASCII messages they should quote ASCII values.
I also believe (Paging the Grand PooBah or other Java guru) that the readString method has an implicit conversion because it's a string. Someone?
| speediii wrote: |
I presume also that as the format is MQSTR providing the characterSet and Encoding is set the receiving app will convert the message "if necessary". |
Only if the receiving application specifies it via the get options or (if I'm correct) the readString method.
(Sure I've seen that someplace.....  )
For the record, not a Java person, might be talking rubbish regarding readString.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Aug 17, 2006 3:36 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
It will certainly do the conversion if the CCSID and format are set correctly using JMS. And you have no controll over it doing so using JMS.
So if they are receiving a text message and sending a text message it is up to you in your receiving application to specify the convert option on the get.
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| tleichen |
| Posted: Thu Aug 17, 2006 7:44 am Post subject: |
|
|
Yatiri
Joined: 11 Apr 2005
Posts: 663
Location: Center of the USA
|
Of course if you just want to pass messages back and forth with no conversion at all, they can always be passed as MQFMT_NONE. But then, you need to deal with the consequences (if there are any).
_________________
IBM Certified MQSeries Specialist
IBM Certified MQSeries Developer |
|
| Back to top |
|
|
| rmpinchback |
| Posted: Thu Aug 17, 2006 10:17 am Post subject: |
|
|
Newbie
Joined: 17 Aug 2006
Posts: 6
|
Speediii, I have an idea about the problem. I'm going from memory here, but this sounds like something I ran into about six months ago. One of the gurus may scream "no this is wrong"... I just know that I went from messages not being accepted by a mainframe to having them accepted.
If what I think is happening is happening, then focusing only on the mainframe end might not help you. Yes the mainframe should be doing a convert on the get, but they may already be doing a convert on the get. It isn't enough. The mainframe knows what it is converting TO but it also needs to know what it is converting FROM; how you put messages on the queue in the first place influences the FROM.
The problem is that unless you control otherwise, the default CCSID for client mode comes from the queue manager. That means if the client shoves some ASCII variant into a message and puts it on the queue, the queue manager (and hence the receiving application) might think it is something else.... like EBCDIC (if the QM is running on a mainframe or deliberately configured to have that CCSID). That's fine if it the text in the message was already in EBCDIC, but definitely no help if it wasn't.
The solution I used, and you may need, is to specify the code page for the kind of platform that this vendor's application is executing on. In other words, the vendor may have to fix their code. Its something that, if you ever had to track it down once, I doubt you'd ever forget it. The fact that they described what they are using as "ASCII" makes me suspect they haven't had to do this. The magic values come from page 564 and pages 603-623 of the App Programming Ref (not the Java one).
Since its been a bit, I can't remember 100% if I specified this as a message property, or fed it to MQEnvironment when getting the queue manager connection, but one way or another they need to get it right or you are hosed. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Aug 17, 2006 11:48 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| rmpinchback wrote: |
| One of the gurus may scream "no this is wrong"... |
Can I just say "not quite I don't think" quite loudly? 
Starting with the disclaimer, it's been a while since I used Client Attach on a mainframe but any MQ message should have the CCSID used to write it coded in the MQMD. If a get with convert is specified and the message is formatted as a string then the queue manager seeks a code conversion table to take that CCSID into the queue manager's CCSID. Queue Manger's default CCSID is just an attribute, normally the CCSID of the underlying platform.
This behavour can be modified using an exit or other methods. CCSIDs can be set. You can get problems if the conversion is unrecognised or is not as solid as you'd hope.
| rmpinchback wrote: |
| The problem is that unless you control otherwise, the default CCSID for client mode comes from the queue manager. That means if the client shoves some ASCII variant into a message and puts it on the queue, the queue manager (and hence the receiving application) might think it is something else.... like EBCDIC (if the QM is running on a mainframe or deliberately configured to have that CCSID). |
IMHO if the incoming message has an ASCII variant and the queue manager defaults to EBCDIC (logically enough for a mainframe) then the conversion should still take place even through Client Attach. Now I accept if the message is in weird-ascii you could be a conversion short & this could give the result described, but I think you could fix it by updating the conversion, holding to the "receiver makes good" principle, as well as changing the code page used on the send.
My 2 cents. Other, confilicting, view from better gurus welcomed.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Aug 18, 2006 2:20 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
I think what he was referring to is setting to XXX the CCSID on the qmgr instance created by the program that will put the message. This tells the qmgr that by default the messages I am posting are in CCSID XXX. Now the content of my message may not be in CCSID XXX and as such I would be introducing errors into the system...
Same thing if you set the message CCSID to YYY while the content is really in CCSID XXX
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| rmpinchback |
| Posted: Fri Aug 18, 2006 5:09 am Post subject: |
|
|
Newbie
Joined: 17 Aug 2006
Posts: 6
|
| Yes this sounds like it describes the situation I was seeing. My role in the problem was probably similar to the vendor that speediii mentioned. We had MQ stuff that worked just fine... added in support for a new service to a mainframe in another company accessed over MQ, and hit the ASCII-EBCDIC conversion. Suspect the material difference was that instead of telling the mainframe guys to make the problem go away I was actually hunting around to see if there was something we should be fixing ourselves. |
|
| Back to top |
|
|
|
|
