| Author |
Message
|
| rohank84 |
 Posted: Tue Sep 17, 2013 3:26 am Post subject: Handle Junk character in message flow |
 |
|
Centurion
Joined: 31 Dec 2008
Posts: 109
|
Hi Guys,
WMB 6.1
MQ v6
I have a message coming from an unknown source. The message contains a junk character. The code is supposed to split the message and insert into JD Edwards database. The code parses the message through a message set. The code works fine and does not throw any errors but the records are not inserted into the table.
When i opened the file in textpad or notepad nothing seemed wrong but when i opened it in notepad++ i could see a junk character.
There are 2 message flow to do it.
1. Splitter --- this splits the message based on an indicator. This is working fine.
2. DB Insert --- This code is causing an issue.
The message flow has a file input node where the message set is used to parse the message.
I am not able to upload the file hence could not show you the incoming file.
Any idea how can I handle this .... |
|
| Back to top |
|
 |
| lancelotlinc |
| Posted: Tue Sep 17, 2013 3:29 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
|
| Back to top |
|
|
| rohank84 |
| Posted: Tue Sep 17, 2013 4:26 am Post subject: |
|
|
Centurion
Joined: 31 Dec 2008
Posts: 109
|
The code never reaches the database...it fails when it tries to parse the file...... I will get the user trace but the question here is how i can handle this scenario..... I did one thing wherein I removed the junk and padded a white space to it. The code worked fine and records were inserted.
Is there a way I can handle it.......can some change in encoding can help .... |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Tue Sep 17, 2013 4:31 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| rohank84 wrote: |
The code never reaches the database...it fails when it tries to parse the file...... I will get the user trace but the question here is how i can handle this scenario..... I did one thing wherein I removed the junk and padded a white space to it. The code worked fine and records were inserted.
Is there a way I can handle it.......can some change in encoding can help .... |
So you are saying that there is a data quality issue with the producer of the information. Did you contact them to find the root cause for the junk data ?
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| rohank84 |
| Posted: Tue Sep 17, 2013 10:37 pm Post subject: |
|
|
Centurion
Joined: 31 Dec 2008
Posts: 109
|
Yes ...its the issue with the source system. I had a discussion and they are also clueless in why that character is coming. They are investigating on it. In the mean while, i am trying to handle it in message flow itself. The messages are generated by the Mainframe system. When I open the file in Notepad++ i see a character "nul" in it.
I guess there may be some encoding that I can do to remove these but I am not sure.....
Also when I am trying to get the user trace I get this message when i hit mqsireadlog.....Any idea why this is coming ....
BIP8217S: Trace file D:\Documents and Settings\All Users\Application Data\IBM\MQ
SI/common/log/WBRK61_DEFAULT_BROKER.bfc193a7-3b01-0000-0080-8609c87d559d.userTra
ce.bin.0 is corrupt, and has not been fully read.
The trace file contains data other than complete trace records for the current p
roduct version, and the trace reader command was not able to process the file.
Ensure that the correct product version is used to read trace files. If the corr
ect version is being used and this error still occurs, ensure that enough disk s
pace is available to store trace data, and that none of the files are limited by
other means (e.g., ulimit on UNIX platforms). |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Sep 18, 2013 12:06 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
when you enabled trace did you use the -r option?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| rohank84 |
| Posted: Wed Sep 18, 2013 1:13 am Post subject: |
|
|
Centurion
Joined: 31 Dec 2008
Posts: 109
|
yes ...these are the commands I fired ....
mqsichangetrace WBRK61_DEFAULT_BROKER -u -e egname -f flowname -l debug -r -c 50000
mqsireadlog WBRK61_DEFAULT_BROKER -u -e egname -f -o D:\SplitterError\flowtrace.xml
mqsiformatlog -i D:\SplitterError\flowtrace.xml -o D:\SplitterError\usertrace.txt
mqsichangetrace WBRK61_DEFAULT_BROKER -u -e egname -f flowname -l none
I got the trace for the first time and then I fired the last command ......Yesterday when i tried ...the first command was hit successfully ....but then the error came for mqsireadlog ........
I deleted the file and tried again but it failed again for the same error ....apparently the filename remains the same .... |
|
| Back to top |
|
|
| rohank84 |
| Posted: Wed Sep 18, 2013 1:29 am Post subject: |
|
|
Centurion
Joined: 31 Dec 2008
Posts: 109
|
This is what I get in Trace node........
( ['MQROOT' : 0xa793c18]
(0x01000000:Name):RecoverableException = (
(0x03000000:NameValue):File = 'F:\build\S610_P\src\DataFlowEngine\ImbComputeNode.cpp' (CHARACTER)
(0x03000000:NameValue):Line = 464 (INTEGER)
(0x03000000:NameValue):Function = 'ImbComputeNode::evaluate' (CHARACTER)
(0x03000000:NameValue):Type = 'ComIbmComputeNode' (CHARACTER)
(0x03000000:NameValue):Name = 'IJD_SG_WMB_PUB00#FCMComposite_1_6' (CHARACTER)
(0x03000000:NameValue):Label = 'IJD_SG_WMB_PUB00.LogReceived' (CHARACTER)
(0x03000000:NameValue):Catalog = 'BIPv610' (CHARACTER)
(0x03000000:NameValue):Severity = 3 (INTEGER)
(0x03000000:NameValue):Number = 2230 (INTEGER)
(0x03000000:NameValue):Text = 'Caught exception and rethrowing' (CHARACTER)
(0x01000000:Name ):RecoverableException = (
(0x03000000:NameValue):File = 'F:\build\S610_P\src\DataFlowEngine\ImbRdl\ImbRdlStatementGroup.cpp' (CHARACTER)
(0x03000000:NameValue):Line = 602 (INTEGER)
(0x03000000:NameValue):Function = 'SqlStatementGroup::execute' (CHARACTER)
(0x03000000:NameValue):Type = 'ComIbmComputeNode' (CHARACTER)
(0x03000000:NameValue):Name = 'IJD_SG_WMB_PUB00#FCMComposite_1_6' (CHARACTER)
(0x03000000:NameValue):Label = 'IJD_SG_WMB_PUB00.LogReceived' (CHARACTER)
(0x03000000:NameValue):Catalog = 'BIPv610' (CHARACTER)
(0x03000000:NameValue):Severity = 3 (INTEGER)
(0x03000000:NameValue):Number = 2488 (INTEGER)
(0x03000000:NameValue):Text = 'Error detected, rethrowing' (CHARACTER)
(0x01000000:Name ):Insert = (
(0x03000000:NameValue):Type = 5 (INTEGER)
(0x03000000:NameValue):Text = '.IJD_WMB_PROCESSINV_PUB00_001_LogReceived.Main' (CHARACTER)
)
(0x01000000:Name ):Insert = (
(0x03000000:NameValue):Type = 5 (INTEGER)
(0x03000000:NameValue):Text = '9.3' (CHARACTER)
)
(0x01000000:Name ):Insert = (
(0x03000000:NameValue):Type = 5 (INTEGER)
(0x03000000:NameValue):Text = 'SET Environment.Variables.NativeID = COALESCE(NULLIF(TRIMBOTH(BOTH FROM inRef.EUROFACTINVOICERECORD.EF_10_INVOICE_NUM), ''), ' ') || ';' || COALESCE(NULLIF(TRIMBOTH(BOTH FROM inRef.EUROFACTINVOICERECORD.EF_10_AVIEXP_BORDEREAU), ''), ' ') || ';' || COALESCE(NULLIF(TRIMBOTH(BOTH FROM inRef.EUROFACTINVOICERECORD.EF_10_SELLER_CNTRY), ''), ' ') || ';' || COALESCE(NULLIF(TRIMBOTH(BOTH FROM inRef.EUROFACTINVOICERECORD.EF_10_BUYER_COUNTRY), ''), ' ') || ';' || COALESCE(NULLIF(TRIMBOTH(BOTH FROM inRef.EUROFACT_ROOT_20.EUROFACTINVOICELINE1.EF_20_PRD_SUB_TYPE), ''), ' ');' (CHARACTER)
)
(0x01000000:Name ):RecoverableException = (
(0x03000000:NameValue):File = 'F:\build\S610_P\src\DataFlowEngine\ImbRdl\ImbRdlFieldRef.cpp' (CHARACTER)
(0x03000000:NameValue):Line = 2704 (INTEGER)
(0x03000000:NameValue):Function = 'SqlFieldReference::evaluate' (CHARACTER)
(0x03000000:NameValue):Type = 'ComIbmComputeNode' (CHARACTER)
(0x03000000:NameValue):Name = 'IJD_SG_WMB_PUB00#FCMComposite_1_6' (CHARACTER)
(0x03000000:NameValue):Label = 'IJD_SG_WMB_PUB00.LogReceived' (CHARACTER)
(0x03000000:NameValue):Catalog = 'BIPv610' (CHARACTER)
(0x03000000:NameValue):Severity = 3 (INTEGER)
(0x03000000:NameValue):Number = 2498 (INTEGER)
(0x03000000:NameValue):Text = 'Navigation error' (CHARACTER)
(0x01000000:Name ):Insert = (
(0x03000000:NameValue):Type = 5 (INTEGER)
(0x03000000:NameValue):Text = '.IJD_WMB_PROCESSINV_PUB00_001_LogReceived.Main' (CHARACTER)
)
(0x01000000:Name ):Insert = (
(0x03000000:NameValue):Type = 5 (INTEGER)
(0x03000000:NameValue):Text = '13.34' (CHARACTER)
)
(0x01000000:Name ):Insert = (
(0x03000000:NameValue):Type = 2 (INTEGER)
(0x03000000:NameValue):Text = '2' (CHARACTER)
)
(0x01000000:Name ):ParserException = (
(0x03000000:NameValue):File = 'F:\build\S610_P\src\MTI\MTIforBroker\GenXmlParser2\XmlImbParser.cpp' (CHARACTER)
(0x03000000:NameValue):Line = 703 (INTEGER)
(0x03000000:NameValue):Function = 'XmlImbParser::parseRightSibling' (CHARACTER)
(0x03000000:NameValue):Type = '' (CHARACTER)
(0x03000000:NameValue):Name = '' (CHARACTER)
(0x03000000:NameValue):Label = '' (CHARACTER)
(0x03000000:NameValue):Catalog = 'BIPv610' (CHARACTER)
(0x03000000:NameValue):Severity = 3 (INTEGER)
(0x03000000:NameValue):Number = 5009 (INTEGER)
(0x03000000:NameValue):Text = 'XML Parsing Errors have occurred' (CHARACTER)
(0x01000000:Name ):ParserException = (
(0x03000000:NameValue):File = 'F:\build\S610_P\src\MTI\MTIforBroker\GenXmlParser2\XmlBrokerAsgardParser.cpp' (CHARACTER)
(0x03000000:NameValue):Line = 723 (INTEGER)
(0x03000000:NameValue):Function = 'XmlBrokerAsgardParser::error' (CHARACTER)
(0x03000000:NameValue):Type = '' (CHARACTER)
(0x03000000:NameValue):Name = '' (CHARACTER)
(0x03000000:NameValue):Label = '' (CHARACTER)
(0x03000000:NameValue):Catalog = 'BIPv610' (CHARACTER)
(0x03000000:NameValue):Severity = 3 (INTEGER)
(0x03000000:NameValue):Number = 5004 (INTEGER)
(0x03000000:NameValue):Text = 'An error has been reported by the BIPXML4C component.' (CHARACTER)
(0x01000000:Name ):Insert = (
(0x03000000:NameValue):Type = 2 (INTEGER)
(0x03000000:NameValue):Text = '196' (CHARACTER)
)
(0x01000000:Name ):Insert = (
(0x03000000:NameValue):Type = 5 (INTEGER)
(0x03000000:NameValue):Text = '' (CHARACTER)
)
(0x01000000:Name ):Insert = (
(0x03000000:NameValue):Type = 2 (INTEGER)
(0x03000000:NameValue):Text = '1' (CHARACTER)
)
(0x01000000:Name ):Insert = (
(0x03000000:NameValue):Type = 2 (INTEGER)
(0x03000000:NameValue):Text = '6761' (CHARACTER)
)
(0x01000000:Name ):Insert = (
(0x03000000:NameValue):Type = 5 (INTEGER)
(0x03000000:NameValue):Text = 'Invalid character (Unicode: 0x0) ' (CHARACTER)
)
(0x01000000:Name ):Insert = (
(0x03000000:NameValue):Type = 5 (INTEGER)
(0x03000000:NameValue):Text = 'EF_16_SEAL_NUMBER' (CHARACTER)
)
)
)
)
)
)
) |
|
| Back to top |
|
|
| Tibor |
| Posted: Wed Sep 18, 2013 1:59 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
|
| Back to top |
|
|
| rohank84 |
| Posted: Wed Sep 18, 2013 2:22 am Post subject: |
|
|
Centurion
Joined: 31 Dec 2008
Posts: 109
|
Lets not divert from the main topic here of handling the junk ...
I have posted the exception list of the error .....
| Code: |
(0x01000000:Name ):Insert = (
(0x03000000:NameValue):Type = 5 (INTEGER)
(0x03000000:NameValue):Text = 'Invalid character (Unicode: 0x0) ' (CHARACTER)
)
(0x01000000:Name ):Insert = (
(0x03000000:NameValue):Type = 5 (INTEGER)
(0x03000000:NameValue):Text = 'EF_16_SEAL_NUMBER' (CHARACTER)
) |
I tried to below 2 things in esql to change the ccsid and encoding but none helped ....
1. CCSID 1208 and Encoding 546
2. CCSID 500 Encoding 785
Is there a way around for this ..... |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Sep 18, 2013 2:30 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
There are plenty of posts here with a similar theme.
If the data you get is wrong then there is not a lot you can do about it.
you could read the data as a blob, checn for a null at the start and substring the blob to remove it but that is total BODGE/HACK and should not be regarded as a long term solution.
you really need to get the data you receive fixed.
otherwise it is GIGO
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Sep 18, 2013 2:35 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
The error about reading the internal trace file is troublesome.
either it's somehow completely empty - which shouldn't throw this error, or something's gone wrong somewhere.
I'd suggest a broker restart, followed by another changetrace -r. If that doesn't solve that error, then open a PMR.
As a rule, you should never manually manipulate the internal trace files. |
|
| Back to top |
|
|
| Tibor |
| Posted: Wed Sep 18, 2013 2:42 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
smdavies99 is right, but just for my curiosity... could you send us a message dump in hexadecimal format?
Besides, I guess this is an XML parser error, because Unicode+00 is not allowed inside an XML. |
|
| Back to top |
|
|
| rohank84 |
| Posted: Wed Sep 18, 2013 2:50 am Post subject: |
|
|
Centurion
Joined: 31 Dec 2008
Posts: 109
|
Yes there are many posts here for this but none has helped me so far .... I was thinking of handling it here somehow but I guess there is no proper solution to it.
Can you guys tell me how can i identify this junk in my message....I can then think of doing a search and replace .....
Tibor ...i will post the blob of the message ..... |
|
| Back to top |
|
|
| Tibor |
| Posted: Wed Sep 18, 2013 2:56 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
You can use REPLACE() function for changing U+00 to space char:
| Code: |
| SET OutputRoot.BLOB.BLOB = REPLACE(InputRoot.BLOB.BLOB, x'00', x'20'); |
If your input parser is BLOB. |
|
| Back to top |
|
|
|
|
