ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » Encoding used while writing to user trace log file

Post new topic  Reply to topic
 Encoding used while writing to user trace log file « View previous topic :: View next topic » 
Author Message
prasadpav
PostPosted: Thu May 05, 2016 7:05 am    Post subject: Encoding used while writing to user trace log file Reply with quote

Centurion

Joined: 03 Oct 2004
Posts: 142

Hi Gurus,
I'm using Message Broker v7 and Fix pack 5 (or may be above) on AIX. We have come across a small problem recently where the message from SAP has international characters (Chinese characters) and it is said that the source system (SAP) has sent the data to message broker using UTF-8 encoding. But when I used the usertrace to capture the data, I can see the message payload in UTF-16 hex encoding but not UTF-8. I was told by someone within my team that they have seen few instances where message broker when logs the message payload to a file (using mqsichangetrace -u), it sometimes uses UTF-16. Anyone knows of this behaviour and confirm if it is correct or wrong please and under what circumstances it uses UTF-16 as opposed to UTF-8? Any pointers towards the broker documentation will also help if such behaviour is documented.

Many thanks in advance.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Thu May 05, 2016 8:33 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

The logical message tree is in CCSID 1200.
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
prasadpav
PostPosted: Thu May 05, 2016 9:25 am    Post subject: Reply with quote

Centurion

Joined: 03 Oct 2004
Posts: 142

Thanks for the swift reply mqjeff. Much appreciated.

If you don't mind clarifying my follow up questions pls:

1) If the source system sends encoded UTF-8 byte stream, does the broker convert it to UTF-16 by the time I reference the logical tree part (let's say InputRoot.BLOB.BLOB)?

We have a common code to determine the length of the whole message including the headers, so we have something like this:

Code:
LENGTH(ASBITSTREAM(InputRoot CCSID COALESCE(InputRoot.Properties.CodedCharSetId, 1208) ENCODING(InputRoot.Properties.Encoding, MQENC_NATIVE))


If I run the above code, after SAPInput node, I had hoped that the ASBITSTREAM() function would convert data from codepage 1200 (Logical tree using codepage 1200 as per your response) to 1208 (as specified in the COALESCE() statement, remember InputRoot.Properties.CodedCharSetId is NULL after SAPInput node, hence using 1208 from COALESCE function). But I guess there are some intermediate codepage conversions that are happening by ASBITSTREAM() function because we are seeing the following error while it converting from unicode to 819. But we fail to understand why it is converting to 819 in the first place:

Code:
2016-04-11 09:55:15.084167    10796   ParserException  BIP5010E: XML Writing Errors have occurred.
                                       Errors have occurred during writing of XML.
                                       Review further error messages for an indication to the cause of the errors.
2016-04-11 09:55:15.084183    10796   RecoverableException  BIP2136E: Source character ''[b]4e2d[/b]'' in field ''007300650020004c0061006e0067007500610067006500200074006500
780074003a000a4e2d5171524d91cd5e865e0259d44e668bb085847199676553d78d3f30018d2a6c6130016ee57528804c67436848661f671f65e5ff08
003967080032003265e5ff094e0a534857286d4e53574e2d96625ba35224ff0c85847199676588ab5224590465e0671f5f925211ff0c5265593a653f6cbb
674352297ec88eabff0c5e7659046ca165364e2a4eba516890e88d224ea73002000a000a0045003c002f0054004500580054005f004c0049004e0045003
e003c002f005300610070005a00690057007400790063006c006d0031005a00320063006c00610069006d0074007800740031003000300030003e00
3c005300610070005a00690057007400790063006c006d0031005a00320063006c00610069006d0074007800740031003000300030003e003c0052
00450046005f00480041004e0044004c0045003e0030003000300030003000300030003000300031003c002f005200450046005f00480041004e0044
004c0045003e003c004c0041004e00470055003e0045003c002f004c0041004e00470055003e003c004c0041004e00470055005f00490053004f003e00
5a0048003c002f004c0041004e00470055005f00490053004f003e003c0046004f0052004d00410054005f0043004f004c003e003c002f0046004f00520
04d00410054005f0043004f004c003e003c0054004500580054005f004c0049004e0045'' cannot be [b]converted from Unicode to codepage '819'[/b].

The source character is an invalid code point within the given codepage.


Any thoughts on why it would convert to 819 code page first?

Thanks in advance.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Thu May 05, 2016 9:32 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

The BLOB domain doesn't get converted.

You have used COALESCE to decide which CCSID to use.
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
timber
PostPosted: Thu May 05, 2016 1:03 pm    Post subject: Reply with quote

Grand Master

Joined: 25 Aug 2015
Posts: 1280

This is getting confusing, so let's get some facts out in the open.
- When WMB/IIB parses a message, it uses the encoding specified in the node properties or in the transport header ( MQMD header, HTTP header etc )
- All character data in the message tree is stored as UTF-16. Parsers will convert character data from its source encoding to UTF-16 if they parse it at all ( the default setting for parsers is 'on-demand' )
- Parsers have to believe what they are told about the character encoding of the input data. There is no way for a parser to auto-detect the encoding.

Now to answer the points raised:
Quote:
We have come across a small problem recently where the message from SAP has international characters (Chinese characters) and it is said that the source system (SAP) has sent the data to message broker using UTF-8 encoding. But when I used the usertrace to capture the data, I can see the message payload in UTF-16 hex encoding but not UTF-8.
User trace is a great debugging tool, but it's not the correct tool for investigating this problem. You need to look at the actual bytes ( not characters ) of the input message. That's the only way to know for sure whether the sender has labelled it with the correct encoding.
Quote:
I was told by someone within my team that they have seen few instances where message broker when logs the message payload to a file (using mqsichangetrace -u), it sometimes uses UTF-16. Anyone knows of this behaviour and confirm if it is correct or wrong please and under what circumstances it uses UTF-16 as opposed to UTF-8? Any pointers towards the broker documentation will also help if such behaviour is documented.
I'm not at all surprised that user trace is encoded in UTF-16. It seems like an excellent design decision. It doesn't need to be documented because user trace is a diagnostic output, and not something that affects the functional behaviour of the message flow.

You all need to take a step back from the problem, and think again about the best way to diagnose it.

The problem with conversion to 819 is probably happening like this:
1. The sender labels the message as 819 in the header
2. WMB receives the bytes, but does not attempt to parse the message because Parse Timing is set to the default, On Demand.
3. Your logging subflow unconditionally attempts to serialize the message using ASBITSTREAM. It correctly specifies InputRoot.Properties.CodedCharSetId. But that value is incorrect, because the encoding is not, in fact, 819. Your logging subflow just happens to be the first place the problem is discovered.

If you had actually *read* the user trace instead of speculating about its character encoding then it would probably have shown you all of this...unless my guess is completely wrong, of course.
Back to top
View user's profile Send private message
prasadpav
PostPosted: Fri May 06, 2016 2:45 am    Post subject: Reply with quote

Centurion

Joined: 03 Oct 2004
Posts: 142

Thanks everyone for taking time to reply.

@timber

Quote:
If you had actually *read* the user trace instead of speculating about its character encoding then it would probably have shown you all of this...unless my guess is completely wrong, of course.


I wasn't speculating anything and I've read the trace and understood it very well (this is not my first time going through the usertrace). But something still is not clear, hence the question.
Back to top
View user's profile Send private message
timber
PostPosted: Fri May 06, 2016 3:06 am    Post subject: Reply with quote

Grand Master

Joined: 25 Aug 2015
Posts: 1280

Yes, apologies if that sounded snarky. The point is that user trace doesn't claim to reflect the character encoding of whatever is passing through the execution group. It's a text description of what happened, not a trace of the message data.

I'm not 100% clear on whether all of your questions are now answered. Do you think my guess was correct?
Back to top
View user's profile Send private message
prasadpav
PostPosted: Fri May 06, 2016 4:47 am    Post subject: Reply with quote

Centurion

Joined: 03 Oct 2004
Posts: 142

@timber - Your guess is not correct in this case i'm afraid.

But I did a little more digging and the problem is very clear for me now, but still not sure why it is happening though!! Allow me to explain my problem again:

1) SAP system is sending data in UTF-8 (definitely)
2) SAPInput node in Message Broker is receiving it as DataObject and converting it to Logical Tree as per the adapter message set definition. Based on the response from "mqjeff", the logical tree uses UTF-16, hence code page conversion happened from UTF-8 to UTF-16
3) Immediately after SAPInput node, InputRoot.Properties.CodedCharSetId & Encoding are NULL, and the following piece of code which calculates the LENGTH of the whole message is failing:

Code:
LENGTH(ASBITSTREAM(InputRoot CCSID COALESCE(InputRoot.Properties.CodedCharSetId, 1208) ENCODING(InputRoot.Properties.Encoding, MQENC_NATIVE))


When I checked the usertrace, the reason for failure is this:

Code:
2016-04-11 09:55:15.084167    10796   ParserException  BIP5010E: XML Writing Errors have occurred.
                                       Errors have occurred during writing of XML.
                                       Review further error messages for an indication to the cause of the errors.
2016-04-11 09:55:15.084183    10796   RecoverableException  BIP2136E: Source character ''[b]4e2d[/b]'' in field ''007300650020004c0061006e0067007500610067006500200074006500
780074003a000a4e2d5171524d91cd5e865e0259d44e668bb085847199676553d78d3f30018d2a6c6130016ee57528804c67436848661f671f65e5ff08
003967080032003265e5ff094e0a534857286d4e53574e2d96625ba35224ff0c85847199676588ab5224590465e0671f5f925211ff0c5265593a653f6cbb
674352297ec88eabff0c5e7659046ca165364e2a4eba516890e88d224ea73002000a000a0045003c002f0054004500580054005f004c0049004e0045003
e003c002f005300610070005a00690057007400790063006c006d0031005a00320063006c00610069006d0074007800740031003000300030003e00
3c005300610070005a00690057007400790063006c006d0031005a00320063006c00610069006d0074007800740031003000300030003e003c0052
00450046005f00480041004e0044004c0045003e0030003000300030003000300030003000300031003c002f005200450046005f00480041004e0044
004c0045003e003c004c0041004e00470055003e0045003c002f004c0041004e00470055003e003c004c0041004e00470055005f00490053004f003e00
5a0048003c002f004c0041004e00470055005f00490053004f003e003c0046004f0052004d00410054005f0043004f004c003e003c002f0046004f00520
04d00410054005f0043004f004c003e003c0054004500580054005f004c0049004e0045'' cannot be [b]converted from Unicode to codepage '819'[/b].

The source character is an invalid code point within the given codepage.


Interesting bit is at the end of that byte string - "cannot be converted from Unicode to codepage '819'"

This was my original question - The ASBITSTREAM() function CCSID parameters clearly instructed that the resultant byte string from this function to be in UTF-8, which means broker has to convert data from logical tree which is in UTF-16 to UTF-8, but it is rather doing conversion from UTF-16 to '819' (Aix default code page). Why this conversion is happening to code page '819'?

Hope the question & my problem is clear now.

4) Here's another interesting fact. After the above failure, we then modified the code to use InputBody and OPTIONS as FolderBitStream, it worked as expected. Here's the piece of code that works:

Code:
LENGTH(ASBITSTREAM(InputBody OPTIONS BITOR(FolderBitStream, ValidateNone)
      CCSID COALESCE(InputRoot.Properties.CodedCharSetId, 1208)
      ENCODING COALESCE(InputRoot.Properties.Encoding, MQENC_NATIVE) ));


As usual, many thanks in advance for taking time in going through this.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Fri May 06, 2016 4:59 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

prasadpav wrote:
This was my original question - The ASBITSTREAM() function CCSID parameters clearly instructed that the resultant byte string from this function to be in UTF-8, which means broker has to convert data from logical tree which is in UTF-16 to UTF-8, but it is rather doing conversion from UTF-16 to '819' (Aix default code page). Why this conversion is happening to code page '819'?


It's instructed to use EITHER the codepage of the incoming message OR UTF-8. That's what COALESCE does - it says "return the first value that IS NOT NULL".

So your InputRoot...CodedCharSetID is 819, not NULL.
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
prasadpav
PostPosted: Fri May 06, 2016 5:15 am    Post subject: Reply with quote

Centurion

Joined: 03 Oct 2004
Posts: 142

Sorry, I did not included that part of usertrace which says that is NULL. Here it is:


Code:
2016-04-11 09:55:15.056272    10796   UserTrace   BIP2537I: Node 'SI_AD_SAP_TRBGB_ALE_MT_InboundV2.SF_SAP_Input.SF_SetBusinessServicePropertiesV2.Set BSID and Logging': Executing statement   ''SET logRef.LengthOfEntireMessage = LENGTH(ASBITSTREAM(InputRoot ENCODING COALESCE(InputRoot.Properties.Encoding, MQENC_NATIVE) CCSID COALESCE(InputRoot.Properties.CodedCharSetId, 1208)));'' at ('.SF_SetBusinessServicePropertiesV2_Set_BSID_and_Logging.Main', '37.3').
2016-04-11 09:55:15.056292    10796   UserTrace   BIP2543I: Node 'SI_AD_SAP_TRBGB_ALE_MT_InboundV2.SF_SAP_Input.SF_SetBusinessServicePropertiesV2.Set BSID and Logging': ('.SF_SetBusinessServicePropertiesV2_Set_BSID_and_Logging.Main', '38.22') : Failed to navigate to path element number '3' because it does not exist.
2016-04-11 09:55:15.056318    10796   UserTrace   BIP2539I: Node 'SI_AD_SAP_TRBGB_ALE_MT_InboundV2.SF_SAP_Input.SF_SetBusinessServicePropertiesV2.Set BSID and Logging': Evaluating expression ''InputRoot.Properties.Encoding'' at ('.SF_SetBusinessServicePropertiesV2_Set_BSID_and_Logging.Main', '38.22'). This resolved to ''InputRoot.Properties.Encoding''. The result was ''NULL''.
2016-04-11 09:55:15.056334    10796   UserTrace   BIP2539I: Node 'SI_AD_SAP_TRBGB_ALE_MT_InboundV2.SF_SAP_Input.SF_SetBusinessServicePropertiesV2.Set BSID and Logging': Evaluating expression ''MQENC_NATIVE'' at ('.SF_SetBusinessServicePropertiesV2_Set_BSID_and_Logging.Main', '38.53'). This resolved to ''MQENC_NATIVE''. The result was ''273''.
2016-04-11 09:55:15.056351    10796   UserTrace   BIP2539I: Node 'SI_AD_SAP_TRBGB_ALE_MT_InboundV2.SF_SAP_Input.SF_SetBusinessServicePropertiesV2.Set BSID and Logging': Evaluating expression ''COALESCE(InputRoot.Properties.Encoding, MQENC_NATIVE)'' at ('.SF_SetBusinessServicePropertiesV2_Set_BSID_and_Logging.Main', '38.13'). This resolved to ''COALESCE(NULL, 273)''. The result was ''273''.
2016-04-11 09:55:15.056364    10796   UserTrace   BIP2543I: Node 'SI_AD_SAP_TRBGB_ALE_MT_InboundV2.SF_SAP_Input.SF_SetBusinessServicePropertiesV2.Set BSID and Logging': ('.SF_SetBusinessServicePropertiesV2_Set_BSID_and_Logging.Main', '37.83') : Failed to navigate to path element number '3' because it does not exist.
2016-04-11 09:55:15.056390    10796   UserTrace   BIP2539I: Node 'SI_AD_SAP_TRBGB_ALE_MT_InboundV2.SF_SAP_Input.SF_SetBusinessServicePropertiesV2.Set BSID and Logging': Evaluating expression ''InputRoot.Properties.CodedCharSetId'' at ('.SF_SetBusinessServicePropertiesV2_Set_BSID_and_Logging.Main', '37.83'). This resolved to ''InputRoot.Properties.CodedCharSetId''. The result was ''NULL''.
2016-04-11 09:55:15.056408    10796   UserTrace   BIP2539I: Node 'SI_AD_SAP_TRBGB_ALE_MT_InboundV2.SF_SAP_Input.SF_SetBusinessServicePropertiesV2.Set BSID and Logging': Evaluating expression ''COALESCE(InputRoot.Properties.CodedCharSetId, 1208)'' at ('.SF_SetBusinessServicePropertiesV2_Set_BSID_and_Logging.Main', '37.74'). This resolved to ''COALESCE(NULL, 1208)''. The result was ''1208''.
2016-04-11 09:55:15.084129    10796   UserTrace   BIP2231E: Error detected whilst processing a message in node 'SI_AD_SAP_TRBGB_ALE_MT_InboundV2.IDOC_AsyncInboundAdapter_Operations1'.
                                       The message broker detected an error whilst processing a message in node 'SI_AD_SAP_TRBGB_ALE_MT_InboundV2.IDOC_AsyncInboundAdapter_Operations1'. The message has been augmented with an exception list and has been propagated to the node's failure terminal for further processing.
                                       See the following messages for details of the error.
2016-04-11 09:55:15.084142    10796   RecoverableException  BIP2230E: Error detected whilst processing a message in node 'SI_AD_SAP_TRBGB_ALE_MT_InboundV2.SF_SAP_Input.SF_SetBusinessServicePropertiesV2.Set BSID and Logging'.
                                       The message broker detected an error whilst processing a message in node 'SI_AD_SAP_TRBGB_ALE_MT_InboundV2.SF_SAP_Input.SF_SetBusinessServicePropertiesV2.Set BSID and Logging'. An exception has been thrown to cut short the processing of the message.
                                       See the following messages for details of the error.
2016-04-11 09:55:15.084148    10796   RecoverableException  BIP2488E:  ('.SF_SetBusinessServicePropertiesV2_Set_BSID_and_Logging.Main', '37.3') Error detected whilst executing the SQL statement ''SET logRef.LengthOfEntireMessage = LENGTH(ASBITSTREAM(InputRoot ENCODING COALESCE(InputRoot.Properties.Encoding, MQENC_NATIVE) CCSID COALESCE(InputRoot.Properties.CodedCharSetId, 1208)));''.
                                       The message broker detected an error whilst executing the given statement. An exception has been thrown to cut short the SQL program.
                                       See the following messages for details of the error.
2016-04-11 09:55:15.084167    10796   ParserException  BIP5010E: XML Writing Errors have occurred.
                                       Errors have occurred during writing of XML.
                                       Review further error messages for an indication to the cause of the errors.
2016-04-11 09:55:15.084183    10796   RecoverableException  BIP2136E: Source character ''4e2d'' in field
 ''003c004e00530031003a005300610070005a00690057007400790063006c006d003100200078006d006c006e0073003a004e00530031003d00220068007400740070003a002f002f007700770077002e00690062006d002e0063006
f006d002f0078006d006c006e0073002f00700072006f0064002f007700650062007300700068006500720065002f006a003200630061002f007300610070002f007300610070007a00690077007400790063006c006d0031002200
3e003c005300410050005400720061006e00730061006300740069006f006e00490044003e003000410045003800320030003000450039003000450030003500370030004200370034003700410030004300360046003c002f00530
0410050005400720061006e00730061006300740069006f006e00490044003e003c005300610070005a00690057007400790063006c006d003100490044006f00630042004f003e003c00530061007000490044006f00630043006f
006e00740072006f006c005200650063006f00720064003e003c0052004300560050004f0052003e0041003000300030003000300030003000320039003c002f0052004300560050004f0052003e003c0053004e004400500052005
4003e004c0053003c002f0053004e0044005000520054003e003c0053004e0044005000460043003e003c002f0053004e0044005000460043003e003c0054004500530054003e003c002f0054004500530054003e003c004d004500
530043004f0044003e003c002f004d004500530043004f0044003e003c00490044004f0043005400590050003e005a0049005f0057005400590043004c004d0031003c002f00490044004f0043005400590050003e003c0053004e0
04400500052004e003e004a004500510043004c004e0054003300320032003c002f0053004e004400500052004e003e003c005300540044003e003c002f005300540044003e003c004100520043004b00450059003e003c002f0041
00520043004b00450059003e003c00430052004500540049004d003e003100300035003500300034003c002f00430052004500540049004d003e003c0044004f0043005400590050003e003c002f0044004f0043005400590050003
e003c005400410042004e0041004d003e004500440049005f0044004300340030003c002f005400410042004e0041004d003e003c004300520045004400410054003e00320030003100360030003400310031003c002f0043005200
45004400410054003e003c005200450046004700520050003e003c002f005200450046004700520050003e003c004f00550054004d004f0044003e0032003c002f004f00550054004d004f0044003e003c004d0041004e004400540
03e003300320032003c002f004d0041004e00440054003e003c005200430056005300410044003e003c002f005200430056005300410044003e003c004d00450053005400590050003e005a004d005f005700540059005f00500041
004900440043004c00410049004d003c002f004d00450053005400590050003e003c00520043005600500052004e003e0045005300420057005400590041004c0045003c002f00520043005600500052004e003e003c00530054004
4005600520053003e003c002f005300540044005600520053003e003c005200430056004c00410044003e003c002f005200430056004c00410044003e003c004400490052004500430054003e0031003c002f004400490052004500
430054003e003c0053004e0044004c00410044003e003c002f0053004e0044004c00410044003e003c004500580050005200530053003e003c002f004500580050005200530053003e003c0053004e0044005300410044003e003c0
02f0053004e0044005300410044003e003c0044004f004300520045004c003e003700330031003c002f0044004f004300520045004c003e003c00530045005200490041004c003e0032003000310036003000330033003000310031
0031003600300033003c002f00530045005200490041004c003e003c0053004e00440050004f0052003e005300410050004a00450051003c002f0053004e00440050004f0052003e003c0044004f0043004e0055004d003e0030003
000300030003000300030003000390034003000340032003100370032003c002f0044004f0043004e0055004d003e003c005300540044004d00450053003e005a004d005f005700540059003c002f005300540044004d0045005300
3e003c0052004500460049004e0054003e003c002f0052004500460049004e0054003e003c005200430056005000460043003e003c002f005200430056005000460043003e003c005200450046004d00450053003e003c002f00520
0450046004d00450053003e003c004d00450053004600430054003e003c002f004d00450053004600430054003e003c00430049004d005400590050003e003c002f00430049004d005400590050003e003c00530054004100540055
0053003e00330030003c002f005300540041005400550053003e003c005200430056005000520054003e004c0053003c002f005200430056005000520054003e003c002f00530061007000490044006f00630043006f006e0074007
2006f006c005200650063006f00720064003e003c005300610070005a00690057007400790063006c006d00310044006100740061005200650063006f00720064003e003c005300610070005a00690057007400790063006c006d00
31005a003200660069006c00650068006400720031003000300030003e003c0043004c004d004e004f003e003000300030003000300030003000300037003000310034003c002f0043004c004d004e004f003e003c005a005a00530
05500420043004f0055004e0054003e00300031003c002f005a005a0053005500420043004f0055004e0054003e003c005a005a004400450041004c00450052005f00490044003e00320031003000330036003c002f005a005a0044
00450041004c00450052005f00490044003e003c005a005a004400490053005400520049004200550054004f0052005f00490044003e004d0030003000330034003c002f005a005a004400490053005400520049004200550054004
f0052005f00490044003e003c005a005a004d004f00440045004c003e0048004400560048003c002f005a005a004d004f00440045004c003e003c005a005a0041004c005400560049004e003e00530041004c004c00410041004100
31003400420041003500370033003300390034003c002f005a005a0041004c005400560049004e003e003c005a005a0045004e00470049004e004e004f003e003c002f005a005a0045004e00470049004e004e004f003e003c005a0
05a00560049004e005f0044004500530043003e0044006900730063006f007600650072007900200032002e0037004c00200054004400560036002000440069006500730065006c00200034005700440020004100750074006f0020
003500200044006f003c002f005a005a00560049004e005f0044004500530043003e003c005a005a0041004e004e004f003e0032003000310031003c002f005a005a0041004e004e004f003e003c005a005a00530041004c0045003
e00320030003100310030003400310038003c002f005a005a00530041004c0045003e003c005a005a004c0041004e00440031003e003c002f005a005a004c0041004e00440031003e003c005a005a005200450043004f0052004400
450044005f00560041004c00550045003e00380031003500300030003c002f005a005a005200450043004f0052004400450044005f00560041004c00550045003e003c005a005a005200450043004f0052004400450044005f00550
04e00490054003e004b004d003c002f005a005a005200450043004f0052004400450044005f0055004e00490054003e003c0050004e005400580043004e0054003e003c002f0050004e005400580043004e0054003e003c0043004c
004d00540059003e005a004a004c004d003c002f0043004c004d00540059003e003c00520045004600440054003e00320030003100360030003300320038003c002f00520045004600440054003e003c005200450046004e004f003
e0035003600340032003000300054003c002f005200450046004e004f003e003c004f004c00440043004e003e003c002f004f004c00440043004e003e003c004100550054004e004f003e003c002f004100550054004e004f003e00
3c00520045004c00540059003e00560045004c004f003c002f00520045004c00540059003e003c00520045004c004f0054003e003c002f00520045004c004f0054003e003c00520045004c004f0042005f004500580054003e00530
041004c0041004e003200310034003400420041003500370033003300390034003c002f00520045004c004f0042005f004500580054003e003c005000410052004e0052003e0030003000340030003000300033003500380035003c
002f005000410052004e0052003e003c00520045004c00440054003e003c002f00520045004c00440054003e003c004100530054004100540045003e005a003000310036003c002f004100530054004100540045003e003c0057004
50052004b0053003e0031003000310030003c002f005700450052004b0053003e003c0043004c004d004700520050003e003c002f0043004c004d004700520050003e003c0052004500530055004c0054005f004f00550054005f00
4d005700540059003e00320034003c002f0052004500530055004c0054005f004f00550054005f004d005700540059003e003c0052004500530055004c0054005f0049004e005f004d005700540059003e00390038003c002f00520
04500530055004c0054005f0049004e005f004d005700540059003e003c00520045004c005400590032003e003c002f00520045004c005400590032003e003c004300420041004300540049004f004e00440054003e003c002f0043
00420041004300540049004f004e00440054003e003c0043004c004f00530045005f0044004100540045003e003c002f0043004c004f00530045005f0044004100540045003e003c005a005a005000470043004f00440045003e005
60057003c002f005a005a005000470043004f00440045003e003c005a005a004300430043004f00440045003e005300300032005600330037004100440031003c002f005a005a004300430043004f00440045003e003c005a005a00
430052004d0052003e003c002f005a005a00430052004d0052003e003c005a005a0043004f004e0054004100430054003e003c002f005a005a0043004f004e0054004100430054003e003c005a005a00500048005f004e0055004d0
03e003c002f005a005a00500048005f004e0055004d003e003c005a005a0053004f0052003e003c002f005a005a0053004f0052003e003c005a005a005300450052005f00570052005f00490044003e003c002f005a005a00530045
0052005f00570052005f00490044003e003c005a005a0054004500430048005f00490044003e003800350032003100300030003c002f005a005a0054004500430048005f00490044003e003c005a005a00530055005000450052005
f00490044003e003c002f005a005a00530055005000450052005f00490044003e003c005a005a00440054005f0052004f005f0043004c004f00530045003e00320030003100360030003300320038003c002f005a005a0044005400
5f0052004f005f0043004c004f00530045003e003c005a005a00440054005f0052004f005f004f00500045004e003e003c002f005a005a00440054005f0052004f005f004f00500045004e003e003c005a005a0043004f0055004e0
05400520059003e0043004e003c002f005a005a0043004f0055004e005400520059003e003c005a005a0052004c0054005f00440041004d004100470045003e003c002f005a005a0052004c0054005f00440041004d004100470045
003e003c005a005a004600530041005f00540059005000450053003e003c002f005a005a004600530041005f00540059005000450053003e003c005a005a0049004e0044005f0043004c004d005f0045005400520059003e003c002
f005a005a0049004e0044005f0043004c004d005f0045005400520059003e003c005a005a00410041005f0043004f00440045003e003c002f005a005a00410041005f0043004f00440045003e003c005a005a00530052005f004800
540059005f0049004e0044003e0058003c002f005a005a00530052005f004800540059005f0049004e0044003e003c005a005a004d0049004c005f0049004e0044003e003c002f005a005a004d0049004c005f0049004e0044003e0
03c005a005a005200500052005f004f0050004e005f00440054003e00320030003100360030003300320038003c002f005a005a005200500052005f004f0050004e005f00440054003e003c004d0041004e00550041004c003e003c
002f004d0041004e00550041004c003e003c005a005a005200500052005f0043004c0047005f00440054003e00320030003100360030003300320038003c002f005a005a005200500052005f0043004c0047005f00440054003e003
c0043004c004f005300450044003e003c002f0043004c004f005300450044003e003c005a005a004e0041004d0045005f0041005500540048003e003c002f005a005a004e0041004d0045005f0041005500540048003e003c005a00
5a004200520041004e0044003e004c003c002f005a005a004200520041004e0044003e003c0043005200450041004400410054003e00320030003100360030003300330030003c002f0043005200450041004400410054003e003c0
04300520045004100420059003e004a004c005200420043004d0033003c002f004300520045004100420059003e003c00430048004e0047004400410054003e00320030003100360030003300330030003c002f00430048004e0047
004400410054003e003c00430048004e004700420059003e005400410052004f0057004f004c0031003c002f00430048004e004700420059003e003c004c004f0047005300590053003e004a004500510043004c004e00540033003
20032003c002f004c004f0047005300590053003e003c005a005a0054004500430048005f004100500050003e003c002f005a005a0054004500430048005f004100500050003e003c005a005a0043004f004d004d005f0041005000
500052004f00560041004c003e003c002f005a005a0043004f004d004d005f0041005000500052004f00560041004c003e003c005a005a00560049004e0052004500530054003e003c002f005a005a00560049004e0052004500530
054003e003c005a005a0056004500480052004500530054005f00500041005200540049004c003e003c002f005a005a0056004500480052004500530054005f00500041005200540049004c003e003c005a005a005300440044005f
0049004e0044003e003c002f005a005a005300440044005f0049004e0044003e003c005a005a0054004500430048005f004e0041004d0045003e003c002f005a005a0054004500430048005f004e0041004d0045003e003c005a005
a005300450052005f00570052005f004e0041004d0045003e003c002f005a005a005300450052005f00570052005f004e0041004d0045003e003c005a005a004a004f0042005f0043004100520044005f004e0055004d003e003500
34003500320030003c002f005a005a004a004f0042005f0043004100520044005f004e0055004d003e003c005a005a004d004f004e005f0053004500520056003e00300030003000350039003c002f005a005a004d004f004e005f0
053004500520056003e003c005a005a005200450047004e004f003e003c002f005a005a005200450047004e004f003e003c005a005a004400530048004f0050003e003000300030003c002f005a005a004400530048004f0050003e
003c005a005a004400450041004c00450052003e00320031003000330036003c002f005a005a004400450041004c00450052003e003c005a005a005000410059004d0045004e0054005f00440054003e003c002f005a005a0050004
10059004d0045004e0054005f00440054003e003c005a005a00410043004300450050005400450044005f00440054003e00320030003100360030003300330030003c002f005a005a00410043004300450050005400450044005f00
440054003e003c005a005a005300550042004d0049005300530049004f004e005f00440054003e00320030003100360030003300330030003c002f005a005a005300550042004d0049005300530049004f004e005f00440054003e0
03c005a005a005200450051005f0046004f0052003e003c002f005a005a005200450051005f0046004f0052003e003c005a005a005000520054004300490050004100540049004e005f005400590050003e00470031003c002f005a
005a005000520054004300490050004100540049004e005f005400590050003e003c005a005a00420052004b005f00440057004e005f0043004e005400520059003e003c002f005a005a00420052004b005f00440057004e005f004
3004e005400520059003e003c005a005a004c00470043005f0043004c00410049004d0054005900500045003e00310031003c002f005a005a004c00470043005f0043004c00410049004d0054005900500045003e003c005a005a00
500041005200540043004c004d005f005200450046003e003c002f005a005a00500041005200540043004c004d005f005200450046003e003c005a005a00440045004d004f0044004100540045003e003c002f005a005a004400450
04d004f0044004100540045003e003c005000440044004100540055003e00320030003100310030003200310031003c002f005000440044004100540055003e003c005a005a005400520041004e0053003e00560051003c002f005a
005a005400520041004e0053003e003c005a005a0045004e00470043004f00440045003e00510031003c002f005a005a0045004e00470043004f00440045003e003c005a005a005200450056004300430043004f00440045003e003
c002f005a005a005200450056004300430043004f00440045003e003c005a005a0046004300430041005500540048003e003c002f005a005a0046004300430041005500540048003e003c005a005a00540052004e0053004d004e00
5f005400590050003e0041003c002f005a005a00540052004e0053004d004e005f005400590050003e003c005a005a00460043004300420049004e003e003c002f005a005a00460043004300420049004e003e003c005a005a00460
04300430052004500470049004f004e003e0040003c002f005a005a0046004300430052004500470049004f004e003e003c005a005a00460049005200530054005f00410050005000520056004c005f00440054003e003200300031
00360030003300330030003c002f005a005a00460049005200530054005f00410050005000520056004c005f00440054003e003c004c0041004e00470055005f00490053004f003e005a0048003c002f004c0041004e00470055005
f00490053004f003e003c004b0044004700520050003e00310031003c002f004b0044004700520050003e003c005300610070005a00690057007400790063006c006d0031005a00320063006c00610069006d007400780074003100
3000300030003e003c005200450046005f00480041004e0044004c0045003e0030003000300030003000300030003000300031003c002f005200450046005f00480041004e0044004c0045003e003c004c0041004e00470055003e0
045003c002f004c0041004e00470055003e003c004c0041004e00470055005f00490053004f003e005a0048003c002f004c0041004e00470055005f00490053004f003e003c0046004f0052004d00410054005f0043004f004c003e
003c002f0046004f0052004d00410054005f0043004f004c003e003c0054004500580054005f004c0049004e0045003e005b005700650064002c0020003300300020004d0061007200200032003000310036002000310030003a003
00032003a0031003200200047004d0054005d000a004300680069006e0065007300650020004c0061006e0067007500610067006500200074006500780074003a000a4e2d5171524d91cd5e865e0259d44e668bb085847199676553
d78d3f30018d2a6c6130016ee57528804c67436848661f671f65e5ff08003967080032003265e5ff094e0a534857286d4e53574e2d96625ba35224ff0c85847199676588ab5224590465e0671f5f925211ff0c5265593a653f6cbb6
74352297ec88eabff0c5e7659046ca165364e2a4eba516890e88d224ea73002000a000a0045003c002f0054004500580054005f004c0049004e0045003e003c002f005300610070005a00690057007400790063006c006d0031005a
00320063006c00610069006d0074007800740031003000300030003e003c005300610070005a00690057007400790063006c006d0031005a00320063006c00610069006d0074007800740031003000300030003e003c00520045004
6005f00480041004e0044004c0045003e0030003000300030003000300030003000300031003c002f005200450046005f00480041004e0044004c0045003e003c004c0041004e00470055003e0045003c002f004c0041004e004700
55003e003c004c0041004e00470055005f00490053004f003e005a0048003c002f004c0041004e00470055005f00490053004f003e003c0046004f0052004d00410054005f0043004f004c003e003c002f0046004f0052004d00410
054005f0043004f004c003e003c0054004500580054005f004c0049004e0045003e006e0067006c0069007300680020005400720061006e0073006c006100740069006f006e003a000a000a0046006f0072006d0065007200200073
006500630072006500740061007200790020006f006600200074006800650020004300500043002000430068006f006e006700710069006e00670020004d0075006e00690063006900700061006c00200042006f002000580069006
c0061006900200062007200690062006500720079002c00200063006f007200720075007000740069006f006e002c002'' cannot be converted from Unicode to codepage '819'.
                                       The source character is an invalid code point within the given codepage.
                                       Correct the application or message flow that generated the message to ensure that all data within the message can be represented in the target codepage.



Back to top
View user's profile Send private message
timber
PostPosted: Fri May 06, 2016 8:11 am    Post subject: Reply with quote

Grand Master

Joined: 25 Aug 2015
Posts: 1280

Thanks. For the first time in this thread we have all of the facts available.

The user trace is hard to explain based on the observed facts. ASBITSTREAM is clearly being told to use 1208 as the CCSID. I'm not sure why it would use 819. I don't usually suggest service trace for debugging message flows, but this may be an exceptional case where it will supply the missing information. Or, you could open a PMR and ask IBM.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Fri May 06, 2016 7:26 pm    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

Could it be that the broker's qmgr CCSID is 819?
What happens if you change the CCSID of the queue manager to 1208 and bounce broker and queue manager? Did you have an MQMD header on the message and did it have CCSID 0 or native on it?
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
rekarm01
PostPosted: Mon May 09, 2016 5:52 pm    Post subject: Re: Encoding used while writing to user trace log file Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 1415

prasadpav wrote:
I was told by someone within my team that they have seen few instances where message broker when logs the message payload to a file (using mqsichangetrace -u), it sometimes uses UTF-16.

The usertrace may use some form of Unicode internally, but the end result (such as the mqsiformatlog output) probably depends on the locale. As timber pointed out, this doesn't affect the message flow behavior, but it might be useful to set the user's locale to specify utf-8 (or other suitable value), if, for example, the formatted usertrace doesn't display Chinese characters correctly.

Also, try putting a Trace node after the SAPInput node, (before the ASBITSTREAM() call), to display the ${Root} logical tree; if XML writing errors occur in the Trace node, then the problem is more likely with the input message, and not with ASBITSTREAM().

timber wrote:
ASBITSTREAM is clearly being told to use 1208 as the CCSID. I'm not sure why it would use 819.

ASBITSTREAM() might be told to use 1208 for the message body, but it could still use other ccsids for character data in the message headers.
Back to top
View user's profile Send private message
timber
PostPosted: Mon May 09, 2016 11:24 pm    Post subject: Reply with quote

Grand Master

Joined: 25 Aug 2015
Posts: 1280

Quote:
it could still use other ccsids for character data in the message headers
I agree. I think a service trace may be the way to find out, but your suggestion of inserting a Trace node could yield the vital clue. And it would be a whole lot easier to read.
Back to top
View user's profile Send private message
mayheminMQ
PostPosted: Fri May 13, 2016 6:27 am    Post subject: Reply with quote

Voyager

Joined: 04 Sep 2012
Posts: 77
Location: UK beyond the meadows of RocknRoll

Looking at the trace, both places where InputRoot.Properties were being used, they both returned NULL which means we can assume that the Properties is not being passed through or the values are empty.

Under ASBITSTREAM function, since this is the first time a code page check is taking place and there are no default values from Properties folder, is there a small chance that IIB ends up taking the OS level default CCSID first?
_________________
A Colorblind man may appear disadvantaged but he always sees more than just colors...
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » Encoding used while writing to user trace log file
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.