| Author |
Message
|
| msy_corrado |
 Posted: Wed Mar 11, 2015 10:19 am Post subject: IIB 9.0.0.2 Integration Service CCSID |
 |
|
Newbie
Joined: 05 Mar 2015
Posts: 4
|
Hi all.
My first post, I hope it is correct.
IIB 9.0.0.2 on Windows Server 2012 Standard, MQ 7.5.0.1
We have an integration service with
| Code: |
| <?xml version="1.0" encoding="UTF-8"?> |
that I think it's the default (is it possibile to change this default ?).
We receive a message that has a "Ç" character in a string field, I think it is an allowed character in UTF-8.
The message is correctly processed and then we put it to a Mq Output queue.
In the second flow we read this queue , with an MQInput node, and we receive an XML parsing error
| Quote: |
| An invalid XML character (Unicode: 0xffffffff) was found in the element content of the document. |
This is the Property of the message we receive:
| Code: |
( ['SOAPRoot' : 0xf9ca7f50]
(0x01000000:Name ):Properties = ( ['SOAPPROPERTYPARSER' : 0xe4d20070]
(0x03000000:NameValue):MessageSet = '' (CHARACTER)
(0x03000000:NameValue):MessageType = '' (CHARACTER)
(0x03000000:NameValue):MessageFormat = '' (CHARACTER)
(0x03000000:NameValue):Encoding = 546 (INTEGER)
(0x03000000:NameValue):CodedCharSetId = 1208 (INTEGER)
(0x03000000:NameValue):Transactional = FALSE (BOOLEAN)
(0x03000000:NameValue):Persistence = FALSE (BOOLEAN)
(0x03000000:NameValue):CreationTime = GMTTIMESTAMP '2015-03-11 16:53:52.330002' (GMTTIMESTAMP)
(0x03000000:NameValue):ExpirationTime = -1 (INTEGER)
(0x03000000:NameValue):Priority = 0 (INTEGER)
(0x03000000:NameValue):ReplyIdentifier = X'000000000000000000000000000000000000000000000000' (BLOB)
(0x03000000:NameValue):ReplyProtocol = 'SOAP-AXIS2' (CHARACTER)
(0x03000000:NameValue):Topic = NULL
(0x03000000:NameValue):ContentType = 'text/xml;charset=UTF-8' (CHARACTER)
(0x03000000:NameValue):IdentitySourceType = '' (CHARACTER)
(0x03000000:NameValue):IdentitySourceToken = '' (CHARACTER)
(0x03000000:NameValue):IdentitySourcePassword = '' (CHARACTER)
(0x03000000:NameValue):IdentitySourceIssuedBy = '' (CHARACTER)
(0x03000000:NameValue):IdentityMappedType = '' (CHARACTER)
(0x03000000:NameValue):IdentityMappedToken = '' (CHARACTER)
(0x03000000:NameValue):IdentityMappedPassword = '' (CHARACTER)
(0x03000000:NameValue):IdentityMappedIssuedBy = '' (CHARACTER)
) |
This is the Property of the message read from the queue:
| Code: |
( ['MQROOT' : 0xee0ac2c0]
(0x01000000:Name ):Properties = ( ['MQPROPERTYPARSER' : 0xe080df30]
(0x03000000:NameValue):MessageSet = '' (CHARACTER)
(0x03000000:NameValue):MessageType = '' (CHARACTER)
(0x03000000:NameValue):MessageFormat = '' (CHARACTER)
(0x03000000:NameValue):Encoding = 546 (INTEGER)
(0x03000000:NameValue):CodedCharSetId = 1208 (INTEGER)
(0x03000000:NameValue):Transactional = TRUE (BOOLEAN)
(0x03000000:NameValue):Persistence = TRUE (BOOLEAN)
(0x03000000:NameValue):CreationTime = GMTTIMESTAMP '2015-03-11 16:54:17.910' (GMTTIMESTAMP)
(0x03000000:NameValue):ExpirationTime = -1 (INTEGER)
(0x03000000:NameValue):Priority = 0 (INTEGER)
(0x03000000:NameValue):ReplyIdentifier = X'000000000000000000000000000000000000000000000000' (BLOB)
(0x03000000:NameValue):ReplyProtocol = 'MQ' (CHARACTER)
(0x03000000:NameValue):Topic = NULL
(0x03000000:NameValue):ContentType = '' (CHARACTER)
(0x03000000:NameValue):IdentitySourceType = '' (CHARACTER)
(0x03000000:NameValue):IdentitySourceToken = '' (CHARACTER)
(0x03000000:NameValue):IdentitySourcePassword = '' (CHARACTER)
(0x03000000:NameValue):IdentitySourceIssuedBy = '' (CHARACTER)
(0x03000000:NameValue):IdentityMappedType = '' (CHARACTER)
(0x03000000:NameValue):IdentityMappedToken = '' (CHARACTER)
(0x03000000:NameValue):IdentityMappedPassword = '' (CHARACTER)
(0x03000000:NameValue):IdentityMappedIssuedBy = '' (CHARACTER)
)
(0x01000000:Name ):MQMD = ( ['MQHMD' : 0xf1c77b20]
(0x03000000:NameValue):SourceQueue = 'ISA2MSY_CONTRATTO' (CHARACTER)
(0x03000000:NameValue):Transactional = TRUE (BOOLEAN)
(0x03000000:NameValue):Encoding = 546 (INTEGER)
(0x03000000:NameValue):CodedCharSetId = 1208 (INTEGER)
(0x03000000:NameValue):Format = 'MQSTR ' (CHARACTER)
(0x03000000:NameValue):Version = 2 (INTEGER)
(0x03000000:NameValue):Report = 0 (INTEGER)
(0x03000000:NameValue):MsgType = 8 (INTEGER)
(0x03000000:NameValue):Expiry = -1 (INTEGER)
(0x03000000:NameValue):Feedback = 0 (INTEGER)
(0x03000000:NameValue):Priority = 0 (INTEGER)
(0x03000000:NameValue):Persistence = 1 (INTEGER)
(0x03000000:NameValue):MsgId = X'414d5120494239514d444556202020203e85f8542017d702' (BLOB)
(0x03000000:NameValue):CorrelId = X'000000000000000000000000000000000000000000000000' (BLOB)
(0x03000000:NameValue):BackoutCount = 0 (INTEGER)
(0x03000000:NameValue):ReplyToQ = ' ' (CHARACTER)
(0x03000000:NameValue):ReplyToQMgr = 'IB9QMDEV ' (CHARACTER)
(0x03000000:NameValue):UserIdentifier = 'SYSTEM ' (CHARACTER)
(0x03000000:NameValue):AccountingToken = X'060101120000000000000000000000000000000000000000000000000000000b' (BLOB)
(0x03000000:NameValue):ApplIdentityData = ' ' (CHARACTER)
(0x03000000:NameValue):PutApplType = 11 (INTEGER)
(0x03000000:NameValue):PutApplName = '0.0.2\bin\DataFlowEngine.exe' (CHARACTER)
(0x03000000:NameValue):PutDate = DATE '2015-03-11' (DATE)
(0x03000000:NameValue):PutTime = GMTTIME '16:54:17.910' (GMTTIME)
(0x03000000:NameValue):ApplOriginData = ' ' (CHARACTER)
(0x03000000:NameValue):GroupId = X'000000000000000000000000000000000000000000000000' (BLOB)
(0x03000000:NameValue):MsgSeqNumber = 1 (INTEGER)
(0x03000000:NameValue):Offset = 0 (INTEGER)
(0x03000000:NameValue):MsgFlags = 0 (INTEGER)
(0x03000000:NameValue):OriginalLength = -1 (INTEGER)
) |
In the trace I find this error:
| Quote: |
2015-03-11 17:54:17.977934 6196 UserTrace BIP5004E: Un errore XML ''An invalid XML character (Unicode: 0xffffffff) was found in the element content of the document.'' si è verificato alla riga 1, colonna 1226, durante l'analisi dell'elemento ''/Root/XMLNSC/backendContratto/contratto/isv:indirizzo''.
I codici di errore interni sono '1502' e '2'. |
Out Queue Manager (7.5.0.1) has Coded Character Set Id = 850.
Does anyone have any suggestions?
Thanks a lot |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Wed Mar 11, 2015 10:46 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Stop the second flow and dump the message that has been written to the queue. us a utility such as amsqbcg. This will browse the message andoutput the conetents of the message and the MQMD. The output will be in HEX and Text. This way you can see if there are incorrect characters in the message,
Then you can read it with a utility such as rfhutil and it to the second flow and see what happens.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Mar 11, 2015 11:08 am Post subject: Re: IIB 9.0.0.2 Integration Service CCSID |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| msy_corrado wrote: |
We have an integration service with
| Code: |
| <?xml version="1.0" encoding="UTF-8"?> |
that I think it's the default (is it possibile to change this default ?). |
Yes
| msy_corrado wrote: |
| We receive a message that has a "Ç" character in a string field, I think it is an allowed character in UTF-8. |
It might be allowed in UTF-8, it might not be allowed in XML. Check it out here
| msy_corrado wrote: |
| The message is correctly processed and then we put it to a Mq Output queue. |
Does the processing include actually parsing the string in question? If not, the on-demand parsing may not notice it.
| msy_corrado wrote: |
In the second flow we read this queue , with an MQInput node, and we receive an XML parsing error
| Quote: |
| An invalid XML character (Unicode: 0xffffffff) was found in the element content of the document. |
|
Which implies that code point is outside the acceptable range.
| msy_corrado wrote: |
In the trace I find this error:
| Quote: |
2015-03-11 17:54:17.977934 6196 UserTrace BIP5004E: Un errore XML ''An invalid XML character (Unicode: 0xffffffff) was found in the element content of the document.'' si è verificato alla riga 1, colonna 1226, durante l'analisi dell'elemento ''/Root/XMLNSC/backendContratto/contratto/isv:indirizzo''.
I codici di errore interni sono '1502' e '2'.
Out Queue Manager (7.5.0.1) has Coded Character Set Id = 850.
|
|
Which node actually throws the error? I accept it's caught by the MQInput node, but where is it thrown from? Another possibility is that whatever that character is in UTF-8 is that it's valid in XML but can't be converted into code page 850 for output.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Mar 11, 2015 11:27 am Post subject: Re: IIB 9.0.0.2 Integration Service CCSID |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| Another possibility is that whatever that character is in UTF-8 is that it's valid in XML but can't be converted into code page 850 for output. |
The CCSID on the message being read is 1208, not 850.
Otherwise, your points about legal in UTF-8 but not XML, and where is the exception being thrown are valid.
Also double-check the byte contents of the field /Root/XMLNSC/backendContratto/contratto/isv:indirizzo to make sure that there aren't any extraneous characters in there that are bad. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Mar 11, 2015 11:30 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| msy_corrado wrote: |
| In the second flow we read this queue , with an MQInput node, and we receive an XML parsing error |
When you get the message from the MQInput node do you have the convert flag set on the node? If yes, can you please turn it off? 
xffffffff look a little bit suspicious to me. Don't remember what the write utf puts on the message these days... Haven't done that in ages...
If you put a message on the queue make sure the program that wrote used writeText or some such method and NOT write-UTF. If write-UTF was used you can ONLY retrieve the correct information using read-UTF
You did set MQFMT_STRING on the message. Make sure the whole payload is TEXT. No numbers in hex like -1 (0xffffffff).
Hope it helps
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Mar 11, 2015 11:37 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| IIB doesn't use writeUTF. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Mar 11, 2015 11:43 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
I also would like to believe that the message in question has actually been put with IIB on the queue and not with any other tool made to emulate an IIB message... but without the OP's assurance I cannot be certain... I would like to believe that the message in question has all been serialized by XMLSNC, but then what is the xFFFFFFFF doing there? Is it in a binary field that is not recognized as such by the schema / parser on the other side? Something is afoot and we need the OP to investigate...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Mar 11, 2015 11:55 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| fjb_saper wrote: |
| I also would like to believe that the message in question has actually been put with IIB on the queue and not with any other tool made to emulate an IIB message... but without the OP's assurance I cannot be certain... |
| msy_corrado wrote: |
| The message is correctly processed and then we put it to a Mq Output queue. |
|
|
| Back to top |
|
|
| kimbert |
| Posted: Wed Mar 11, 2015 12:34 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| We receive a message that has a "Ç" character in a string field, I think it is an allowed character in UTF-8. |
UTF-8 is a Unicode encoding, not a 'character set'. So UTF-8 includes *all* characters that are supported by Unicode. Which means the same as 'all characters' for almost any practical purpose.
Same goes for UTF-16 and ITF-32. And a couple of other UTF-* encodings that are hardly ever used.
It is still possible that the input document contains a badly-formed UTF-8 character. Find the bytes that represent the offending character and compare them to the UTF-8 specification ( or use one of the online tools to check, if you want an easier way )
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| msy_corrado |
| Posted: Wed Mar 11, 2015 2:22 pm Post subject: |
|
|
Newbie
Joined: 05 Mar 2015
Posts: 4
|
Hi all.
I read the message in the queue with RFHUTIL
This is the XML
| Quote: |
| lla libe rtÇ 5</i |
This is the HEX
| Quote: |
| 6C6C6120 6C696265 7274C720 353C2F69 |
The char is the exadecimal C7 , character I think allowed in UTF8
| Quote: |
| U+00C7 Ç c3 87 LATIN CAPITAL LETTER C WITH CEDILLA |
as listed in http://www.utf8-chartable.de.
Then I tried to put a map before the MQOutput on the first flow, setting che Charset Id to 1200 (I think UTF16).
The message now is correctly readed from the second flow.
But, if after that I put it on a new queue, then the third flow has the same problem.
[/quote] |
|
| Back to top |
|
|
| kimbert |
| Posted: Thu Mar 12, 2015 4:46 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
So your bytes are
| Code: |
| 6C6C6120 6C696265 7274C720 353C2F69 |
...and you have specified the encoding as UTF-8.
Please read this : http://en.wikipedia.org/wiki/UTF-8#Description
...and let us know when you have spotted the problem 
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| msy_corrado |
| Posted: Thu Mar 12, 2015 6:12 am Post subject: |
|
|
Newbie
Joined: 05 Mar 2015
Posts: 4
|
| kimbert wrote: |
So your bytes are
| Code: |
| 6C6C6120 6C696265 7274C720 353C2F69 |
...and you have specified the encoding as UTF-8.
Please read this : http://en.wikipedia.org/wiki/UTF-8#Description
...and let us know when you have spotted the problem |
Well....I think the reason can be that this char require 2 byte. Is it ?
How can I set a default UTF-16 in my integration services ? Is it correct or is it better to have UTF8 in input ?
Thanks. |
|
| Back to top |
|
|
| kimbert |
| Posted: Thu Mar 12, 2015 1:25 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
For you, UTF-8 is probably the best choice because most of the characters will be single-byte or 2-byte.
However, your first problem is to find out which character encoding the sending application used when it wrote this message. It obviously was not UTF-8 despite what the XML header is claiming.
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| msy_corrado |
| Posted: Thu Mar 26, 2015 8:12 am Post subject: |
|
|
Newbie
Joined: 05 Mar 2015
Posts: 4
|
| kimbert wrote: |
For you, UTF-8 is probably the best choice because most of the characters will be single-byte or 2-byte.
However, your first problem is to find out which character encoding the sending application used when it wrote this message. It obviously was not UTF-8 despite what the XML header is claiming. |
Hi, I'm here again because we already have this problem.
SOAP input message Properties: Encoding 546 , CCSID 1208
Message is than put in a queue , queue manager has CCSID 850 (but also tried to modify to 1208 or 1252)
A second flow extract the message from the queue
If the message contains an è character (I think allowed in 1208 UTF- the first flow, SOAP integration service, has no problem and process the message , convert in XMLNSC and put in the queue.
The message is correctly added to the queue
The second flow , using XMLNSC , extract the message with an exception "An invalid XML character (Unicode: 0xffffffff) was found in the element content of the document." , in the properties I have Encoding 546 , CCSID 1208 and also in the MQMD.
What's the problem in the second flow? I attach the debug variables.
| Code: |
Message
Properties
MessageSet
MessageType
MessageFormat
Encoding 546
CodedCharSetId 1208
Transactional true
Persistence true
CreationTime 2015-03-26 14:50:26.610
ExpirationTime -1
Priority 0
ReplyIdentifier 000000000000000000000000000000000000000000000000
ReplyProtocol MQ
Topic
ContentType
IdentitySourceType
IdentitySourceToken
IdentitySourcePassword
IdentitySourceIssuedBy
IdentityMappedType
IdentityMappedToken
IdentityMappedPassword
IdentityMappedIssuedBy
MQMD
SourceQueue INPUT
Transactional true
Encoding 546
CodedCharSetId 1208
Format MQSTR
Version 2
Report 0
MsgType 8
Expiry -1
Feedback 0
Priority 0
Persistence 1
MsgId 414d5120494239514d444556202020205017145520007e02
CorrelId 000000000000000000000000000000000000000000000000
BackoutCount 0
ReplyToQ
ReplyToQMgr IB9QMDEV
UserIdentifier SYSTEM
AccountingToken 060101120000000000000000000000000000000000000000000000000000000b
ApplIdentityData
PutApplType 11
PutApplName 0.0.2\\bin\\DataFlowEngine.exe
PutDate 2015-03-26
PutTime 14:50:26.610
ApplOriginData
GroupId 000000000000000000000000000000000000000000000000
MsgSeqNumber 1
Offset 0
MsgFlags 0
OriginalLength -1
XMLNSC
ExceptionList
RecoverableException
File F:\\build\\slot1\\S900_P\\src\\DataFlowEngine\\MessageServices\\ImbDataFlowNode.cpp
Line 1155
Function ImbDataFlowNode::createExceptionList
Type ComIbmMQInputNode
Name ContrattoBackendProcessing#FCMComposite_1_1
Label ContrattoBackendProcessing.MSGFLOW
Catalog BIPmsgs
Severity 3
Number 2230
Text Node throwing exception
Insert
Type 14
Text ContrattoBackendProcessing.MSGFLOW
RecoverableException
File F:\\build\\slot1\\S900_P\\src\\DataFlowEngine\\SQLNodeLibrary\\ImbTraceNode.cpp
Line 360
Function ImbTraceNode::evaluate
Type ComIbmTraceNode
Name ContrattoBackendProcessing#FCMComposite_1_27
Label ContrattoBackendProcessing.Trace
Catalog BIPmsgs
Severity 3
Number 2230
Text Caught exception and rethrowing
Insert
Type 14
Text ContrattoBackendProcessing.Trace
ParserException
File F:\\build\\slot1\\S900_P\\src\\MTI\\MTIforBroker\\GenXmlParser4\\ImbXMLNSCParser.cpp
Line 1139
Function ImbXMLNSCParser::parseRightSibling
Type
Name
Label
Catalog BIPmsgs
Severity 3
Number 5009
Text XML Parsing Errors have occurred
ParserException
File F:\\build\\slot1\\S900_P\\src\\MTI\\MTIforBroker\\GenXmlParser4\\ImbXMLNSCDocHandler.cpp
Line 702
Function ImbXMLNSCDocHandler::handleParseErrors
Type ComIbmMQInputNode
Name ContrattoBackendProcessing#FCMComposite_1_1
Label ContrattoBackendProcessing.MSGFLOW
Catalog BIPmsgs
Severity 3
Number 5004
Text An XML parsing error has occurred while parsing the XML document
Insert
Type 2
Text 1502
Insert
Type 2
Text 2
Insert
Type 2
Text 1
Insert
Type 2
Text 1212
Insert
Type 5
Text An invalid XML character (Unicode: 0xffffffff) was found in the element content of the document.
Insert
Type 5
Text /Root/XMLNSC/backendContratto/contratto/http://www.xxx.com/ns:indirizzo
|
|
|
| Back to top |
|
|
| zpat |
| Posted: Thu Mar 26, 2015 12:20 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
You can label a message as CCSID 1208, but that doesn't help if the actual message data is not valid for that code page.
Remember MQPUT does not convert the message data. No, never, not even if you ask nicely and no matter what value you place in the MQMD.CCSID field.
Mismatching the advertised codepage with the actual one used is a very common error.
If you use the correct CCSID (maybe it is 1252 for example) then MQGET or WMB has a chance to convert the data correctly into UTF.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
|
|
