MQSeries.net :: View topic - CSV file input, JSON msg output, charset encoding problem

cevans · Posted: Thu Jul 18, 2013 8:14 am Post subject:

Apologies if I am asking a dumb question but I have found the IBM documentation or web search to have no clear explanation or example that provides clarity to the problem I am experiencing. Broker is quite new to me so I am only familiar with how it works for what I have experienced in training and the projects I have worked on. Anyway â€¦

I take a csv file as input. The file is win-1252, encoded and can contain extended characters such as Â£.

I successfully parse the input using a xsd and can view the message tree in debug with Â£ characters correctly displayed.

I copy the message to environment so that I can use it later.

I then read an MQ Queue to retrieve a JSON, utf-8, message which must be updated with data from the csv file.

When I update the file by replacing part of the input message tree with the saved csv tree (including Â£) the new output message looks fine in debug.

The updated JSON, utf-8, message is written to an MQ Queue.

If I call the service that reads the updated JSON message from the Queue I see the Â£ symbol represented as Ã‚Â£. I have researched this and believe the underlying coding for the character is still windows-1252 as this is the correct representation of that code when seen as utf-8.

So it looks like I need to convert the csv message to utf-8 before I use the content to update a utf-8 file. Sounds reasonable.

The problem is what is the best way to do this? The Help and searching has not identified a clear approach and certainly no suitable or similar example. Or maybe I am not looking in the right place?

I have tried the following with no success.
Change the CodedCharSetId Property
Create a bit stream from the parsed csv and then create an output tree from this bit stream using the CCSID for UTF (see below)

DECLARE inEncoding INT InputProperties.Encoding;
DECLARE inCCSID INT InputProperties.CodedCharSetId;
DECLARE inBitStream BLOB ASBITSTREAM(InputRoot.DFDL,inEncoding,inCCSID);

DECLARE outEncoding INT inEncoding;
DECLARE outCCSID INT 437;
CREATE LASTCHILD OF OutputRoot DOMAIN('DFDL') PARSE (inBitStream,outEncoding,outCCSID,'IncentiveCodes','IncentiveCodes');

So having tried a simple and more complex solution without success I wanted to ask the community for suggestions on an approach and if someone has achieved this a working example would be fantastic. I have seen many response here in relation to similar code page issue so if you want to tell me to RTFM then please donâ€™t bother posting as that would not be helpful.

Many thanks