| Author |
Message
|
| LazyBoy |
 Posted: Wed Jul 08, 2009 6:14 pm Post subject: varying length copybook message |
 |
|
Voyager
Joined: 04 May 2006
Posts: 78
|
Hi,
I need to parse a messsage from mainframe applicaton in WMB 6.1 , it is sending copybook messages. The problem with this message is if the message field does not have any value other than padding character it will truncate the message and send it.
For ex: if the message length is 500 and has 10 fields each field of length 50, if only first five fields have value then it would send just 250 bytes.
can I model this message in CWF?
I tried to use End Of Bitstream property , but this doesn't help my requirement. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu Jul 09, 2009 12:10 am Post subject: Re: varying length copybook message |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| LazyBoy wrote: |
can I model this message in CWF?
|
How would you interpret such a message with a normal application? What indicates how many fields have actually been sent? If it's simply the length of the message (i.e. 250 byte messages have 50 fields) then you need to model it as such a varying sturcture. If there's a number or other indicator, use that.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| kimbert |
| Posted: Thu Jul 09, 2009 2:45 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| can I model this message in CWF? |
No - CWF will complain that it has run out of bit stream.
However, as you are using v6.1 you may well find that TDS can cope with your message format. TDS now supports a wide range of physical types which were previously only available in CWF. And TDS doesn't mind if the bit stream is truncated. You may need to make the trailing fields optional ( minOccurs="0" ) but you ought to be doing that anyway if they can be omitted. |
|
| Back to top |
|
|
| LazyBoy |
| Posted: Thu Jul 09, 2009 6:07 am Post subject: |
|
|
Voyager
Joined: 04 May 2006
Posts: 78
|
Thanks for the reply.
As suggested I used TDS format with physical type as fixed length, made the element nillable and minoccurs to 0 , but I am still not able to parse the message.
The problem with the message is there is no field which indicates the data length and there are no tags as well.
| Quote: |
| What indicates how many fields have actually been sent? |
The length of messsage is the only way. |
|
| Back to top |
|
|
| kimbert |
| Posted: Thu Jul 09, 2009 6:18 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| I am still not able to parse the message. |
I have good reason to believe that this is possible. However, it's hard to help unless you list the error messages that you received. Please list them in full - take a user trace if necessary. |
|
| Back to top |
|
|
| LazyBoy |
| Posted: Thu Jul 09, 2009 12:58 pm Post subject: |
|
|
Voyager
Joined: 04 May 2006
Posts: 78
|
The incoming message length is 529 bytes but the total length of all the fields in message definition is 531, the last 2 fields are of length 1 byte.
Here are the properties I have configured:
Message Set: TDS User Defined Mixed.
Trim Fix Len String
For the last two fields I have following properties:
Logical property: minoccurs = 0
Nillable = Yes
Physical property:
Encoding Null : Nulllitervalue
Encoding Null Value: blank
Here is the Exception message I am getting.
<RecoverableException timestamp='2009-07-09 14:24:02.683094' thread='4372' function='SqlRoutine::invoke' type='ComIbmComputeNode'
name='MYMESSAGEFLOWL#FCMComposite_1_2.gen/MAINFRAME/M_REVERSEFLOW_MAINFRAMEService_UPDATE_INITIALL#FCMComposite_1_4.ComIbmMapping#FCMComposite_1_1'
label='MAINFRAME.UPDATE_INITIALL.M_REVERSEFLOW_MAINFRAMEService.Mapping.ComIbmCompute' text=''Error occured in procedure'' catalog='BIPv610'
number='2934' file='F:\build\S610_P\src\DataFlowEngine\ImbRdl\ImbRdlRoutine.cpp' line='548'><Insert
type='string'>'M_REVERSEFLOW_MAINFRAMEService_UPDATE_INITIALL_Mapping'</Insert><Insert
type='string'>MAINFRAME.UPDATE_INITIALL.M_REVERSEFLOW_MAINFRAMEService.Mapping.ComIbmCompute</Insert></RecoverableException><RecoverableException
timestamp='2009-07-09 14:24:02.683110' thread='4372' function='SqlStatementGroup::execute' type='ComIbmComputeNode'
name='MYMESSAGEFLOWL#FCMComposite_1_2.gen/MAINFRAME/M_REVERSEFLOW_MAINFRAMEService_UPDATE_INITIALL#FCMComposite_1_4.ComIbmMapping#FCMComposite_1_1'
label='MAINFRAME.UPDATE_INITIALL.M_REVERSEFLOW_MAINFRAMEService.Mapping.ComIbmCompute' text=''Error detected, rethrowing'' catalog='BIPv610'
number='2488' file='F:\build\S610_P\src\DataFlowEngine\ImbRdl\ImbRdlStatementGroup.cpp' line='602'><Insert
type='string'>'gen.MAINFRAME.M_REVERSEFLOW_MAINFRAMEService_UPDATE_INITIALL_Mapping'</Insert><Insert
type='string'>'762.2'</Insert><Insert type='string'>'PROPAGATE FINALIZE DEFAULT DELETE DEFAULT;'</Insert><Insert
type='string'>MAINFRAME.UPDATE_INITIALL.M_REVERSEFLOW_MAINFRAMEService.Mapping.ComIbmCompute</Insert></RecoverableException><RecoverableException
timestamp='2009-07-09 14:24:02.683128' thread='4372' function='ImbTraceNode::evaluate' type='ComIbmTraceNode'
name='MYMESSAGEFLOWL#FCMComposite_1_2.gen/MAINFRAME/M_REVERSEFLOW_MAINFRAMEService_UPDATE_INITIALL#FCMComposite_1_13'
label='MAINFRAME.UPDATE_INITIALL.M_REVERSEFLOW_MAINFRAMEService.Trace' text=''Caught exception and rethrowing'' catalog='BIPv610' number='2230'
file='F:\build\S610_P\src\DataFlowEngine\ImbTraceNode.cpp' line='341'><Insert
type='string'>MAINFRAME.UPDATE_INITIALL.M_REVERSEFLOW_MAINFRAMEService.Trace</Insert></RecoverableException><ParserException timestamp='2009-07-09
14:24:02.683146' thread='4372' function='MtiImbParser::parseRightSibling' type='ComIbmSOAPInputNode' name='MYMESSAGEFLOWL#FCMComposite_1_1'
label='MAINFRAME.UPDATE_INITIALL.SOAP Input' text=''ImbRecoverableException caught from worker->parseNext.'' catalog='BIPv610' number='5285'
file='F:\build\S610_P\src\MTI\MTIforBroker\MtiImbParser2\MtiImbParser.cpp' line='731'><Insert type='string'>'M_PERMIT_INITIAL_CC'</Insert><Insert
type='integer'>1</Insert><Insert type='string'>'Text1'</Insert><Insert
type='string'>'/msg_RESPONSERECORD/HAZMAT_EXT_DATE'</Insert><Insert type='string'>MAINFRAME.UPDATE_INITIALL.SOAP
Input</Insert></ParserException><ParserException timestamp='2009-07-09 14:24:02.683178' thread='4372' function='NXDWorker::parseNext' type='' name=''
label='' text=''TDS General Error'' catalog='BIPv610' number='5421' file='F:\build\S610_P\src\cpi\pwf\nxd\nxdworker.cpp' line='462'><Insert
type='string'>'msg_RESPONSERECORD'</Insert><Insert type='string'>'/msg_RESPONSERECORD/LICENSE_TERM'</Insert><Insert
type='integer'>529</Insert><Insert type='string'></Insert></ParserException><ParserException timestamp='2009-07-09 14:24:02.683196' thread='4372'
function='NXDScanner::extractFixedLengthData' type='' name='' label='' text=''Not enough data in bitstream'' catalog='BIPv610' number='5426'
file='F:\build\S610_P\src\cpi\pwf\nxd\nxdscanner.cpp' line='107'><Insert type='string'></Insert></ParserException><UserTrace timestamp='2009-07-09
14:24:02.683711' thread='4372' function='ImbSOAPInputNode::onNotPropagated' type='ComIbmSOAPInputNode' name='MYMESSAGEFLOWL#FCMComposite_1_1'
label='MAINFRAME.UPDATE_INITIALL.SOAP Input' text=''Sending SOAP fault'' catalog='BIPv610' number='3724'
file='F:\build\S610_P\src\WebServices\WSLibrary\ImbSOAPInputNode.cpp' line='909'><Insert type='string'>'MAINFRAME.UPDATE_INITIALL.SOAP
Input'</Insert><Insert type='string'>MAINFRAME.UPDATE_INITIALL.SOAP Input</Insert></UserTrace></UserTraceLog> |
|
| Back to top |
|
|
| kimbert |
| Posted: Thu Jul 09, 2009 2:36 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Sorry - I refuse to ruin my eyesight trying to read that. Please either format it so that I can read it, or else take a user trace and post the relevant part. |
|
| Back to top |
|
|
| LazyBoy |
| Posted: Thu Jul 09, 2009 3:27 pm Post subject: |
|
|
Voyager
Joined: 04 May 2006
Posts: 78
|
Sorry about that, I had taken user trace and posted a part of the trace.
Here is the message in event log:
BIP5426:
The bitstream of a TDS message contains less data than expected.
The TDS parser could not complete parsing because the bitstream contains less data than expected. This could be caused by either an incorrect value for the Length property in the message definition, or by an inconsistent value inside a LengthRef field.
Make sure that the incoming message is a consistent message under the TDS message definition.
BIP5421:
Tagged/Delimited String Format (TDS) parsing error
Current message : ''msg_RESP''
Path to current element : ''/msg_RESP/TERM''
Offset from start of message : 529
See following errors for more details.
BIP5285:
( WBRK61BROKER.default ) Parsing errors have occurred.
Message set name: 'MSETNAME'
Message format: 'Text1'
Message type path: '/msg_RESP/EXT_DATE' |
|
| Back to top |
|
|
| kimbert |
| Posted: Fri Jul 10, 2009 1:20 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| I had taken user trace and posted a part of the trace |
Then you're doing something wrong! Taking a user trace is a three-step process
1. switch on user trace ( usually with the -r flag to discard old trace output )
[put message through flow]
2. mqsireadlog
3. mqsiformatlog
I suspect that you're missing out step 3 and trying to read the raw XML.
No point in trying to diagnose the problem based on BIP5426 - that's just telling us what we already suspected. However, v6.1 TDS writes a trace of the parser's activities into the user trace. Please do the steps above, and post the relevant part of the (formatted) user trace here. Please use [code] tags around the user trace fragment. |
|
| Back to top |
|
|
| LazyBoy |
| Posted: Fri Jul 10, 2009 7:36 am Post subject: |
|
|
Voyager
Joined: 04 May 2006
Posts: 78
|
Hi Kimbert,
Thank You for suggestions on Trace.
Here is the user trace, I didn't find anything valuable that could help me in debugging. I hope you can figure out something.
| Code: |
2009-07-10 09:21:19.691925 5048 UserTrace BIP2658I: MQGetNode 'MQ.UPD_INIT.MYAPP_FLOW_MQService.MYQUEUE1' is propagating a message to the 'Output' Terminal.
None
None
2009-07-10 09:21:19.691993 5048 UserTrace BIP2539I: Node 'MQ.UPD_INIT.MYAPP_FLOW_MQService.Trace': Evaluating expression ''Root'' at ('', '1.3'). This resolved to ''Root''. The result was ''ROW... Root Element Type=16777216 NameSpace='' Name='Root' Value=NULL''.
2009-07-10 09:21:20.093772 5048 Error BIP2628E: Exception condition detected on input node 'MQ.UPD_INIT.SOAP Input'.
The input node 'MQ.UPD_INIT.SOAP Input' detected an error whilst processing a message. The message flow has been rolled-back and, if the message was being processed in a unit of work, it will remain on the input queue to be processed again. Following messages will indicate the cause of this exception.
Check the error messages which follow to determine why the exception was generated, and take action as described by those messages.
2009-07-10 09:21:20.093862 5048 RecoverableException BIP2230E: Error detected whilst processing a message in node 'MQ.UPD_INIT.MYAPP_FLOW_MQService.Mapping.ComIbmCompute'.
The message broker detected an error whilst processing a message in node 'MQ.UPD_INIT.MYAPP_FLOW_MQService.Mapping.ComIbmCompute'. An exception has been thrown to cut short the processing of the message.
See the following messages for details of the error.
2009-07-10 09:21:20.093912 5048 RecoverableException BIP2488E: ('gen.MQ.IBM_WBIMB_MYAPP_FLOW_MQService_UPD_INIT_Mapping.MAIN', '3.1') Error detected whilst executing the SQL statement ''gen.MQ.MYAPP_FLOW_MQService_UPD_INIT_Mapping(InputRoot, OutputRoot, InputLocalEnvironment, OutputLocalEnvironment);''.
The message broker detected an error whilst executing the given statement. An exception has been thrown to cut short the SQL program.
See the following messages for details of the error.
2009-07-10 09:21:20.093959 5048 RecoverableException BIP2934E: Error detected whilst executing the function or procedure ''MYAPP_FLOW_MQService_UPD_INIT_Mapping''.
The message broker detected an error whilst executing the function or procedure ''MYAPP_FLOW_MQService_UPD_INIT_Mapping''. An exception has been thrown to cut short the processing of the message.
See the following messages for details of the error.
2009-07-10 09:21:20.094003 5048 RecoverableException BIP2488E: ('gen.MQ.MYAPP_FLOW_MQService_UPD_INIT_Mapping', '762.2') Error detected whilst executing the SQL statement ''PROPAGATE FINALIZE DEFAULT DELETE DEFAULT;''.
The message broker detected an error whilst executing the given statement. An exception has been thrown to cut short the SQL program.
See the following messages for details of the error.
2009-07-10 09:21:20.094055 5048 RecoverableException BIP2230E: Error detected whilst processing a message in node 'MQ.UPD_INIT.MYAPP_FLOW_MQService.Trace'.
The message broker detected an error whilst processing a message in node 'MQ.UPD_INIT.MYAPP_FLOW_MQService.Trace'. An exception has been thrown to cut short the processing of the message.
See the following messages for details of the error.
2009-07-10 09:21:20.094100 5048 ParserException BIP5285E: Parsing errors have occurred.
Message set name: 'MYAPP_PINIT_CC'
Message format: 'Text1'
Message type path: '/msg_RESPD/DATE'
Review other error messages to find the cause of the errors.
2009-07-10 09:21:20.094144 5048 ParserException BIP5421S: Tagged/Delimited String Format (TDS) parsing error
Current message : ''msg_RESP'
Path to current element : ''/msg_RESPD/TERM''
Offset from start of message : 529
See following errors for more details.
2009-07-10 09:21:20.094186 5048 ParserException BIP5426E: The bitstream of a TDS message contains less data than expected.
The TDS parser could not complete parsing because the bitstream contains less data than expected. This could be caused by either an incorrect value for the Length property in the message definition, or by an inconsistent value inside a LengthRef field.
Make sure that the incoming message is a consistent message under the TDS message definition.
2009-07-10 09:21:20.177471 5048 UserTrace BIP3724I: Node 'MQ.UPD_INIT.SOAP Input' sending a SOAP fault message to the originating client.
See subsequent messages for success or failure messages relating to this reply, and for any transport-specific messages.
No action required.
2009-07-10 09:24:02.843230 784 UserTrace BIP2632I: Message received and propagated to 'out' terminal of MQ input node 'ConfigurationMessageFlow.InputNode'.
2009-07-10 09:24:02.843318 784 UserTrace BIP6060I: Parser type ''Properties'' created on behalf of node 'ConfigurationMessageFlow.InputNode' to handle portion of incoming message of length 0 bytes beginning at offset '0'.
2009-07-10 09:24:02.843348 784 UserTrace BIP6061I: Parser type ''MQMD'' created on behalf of node 'ConfigurationMessageFlow.InputNode' to handle portion of incoming message of length '364' bytes beginning at offset '0'. Parser type selected based on value ''MQHMD'' from previous parser.
2009-07-10 09:24:02.843508 784 UserTrace BIP6061I: Parser type ''XMLS'' created on behalf of node 'ConfigurationMessageFlow.InputNode' to handle portion of incoming message of length '235' bytes beginning at offset '364'. Parser type selected based on value ''XMLS'' from previous parser. |
|
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jul 10, 2009 7:53 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Are you sure the fields that the mainframe did not include are marked as optional?
Are you sure the mainframe set the proper length to indicate that the optional fields were truncated? |
|
| Back to top |
|
|
| LazyBoy |
| Posted: Fri Jul 10, 2009 9:11 am Post subject: |
|
|
Voyager
Joined: 04 May 2006
Posts: 78
|
Hi mqjeff,
| Quote: |
| Are you sure the fields that the mainframe did not include are marked as optional? |
Yes, I have made those fields optional.
| Quote: |
| Are you sure the mainframe set the proper length to indicate that the optional fields were truncated? |
The mainframe is not sending the datalength in the message.
There is no one on mainframe system who knows how this works, they are sending us the cobol source code, I need to look the program to understand their system works. |
|
| Back to top |
|
|
| kimbert |
| Posted: Fri Jul 10, 2009 12:35 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| Here is the user trace, I didn't find anything valuable that could help me in debugging |
Ah! That's because I failed to provide another important user trace tip. You need to set the tracing level to 'debug' in order to get the trace messages from the parser.
I always forget to mention that because I never use User Trace in 'normal' mode. |
|
| Back to top |
|
|
| LazyBoy |
| Posted: Fri Jul 10, 2009 2:48 pm Post subject: |
|
|
Voyager
Joined: 04 May 2006
Posts: 78
|
| Quote: |
Ah! That's because I failed to provide another important user trace tip. You need to set the tracing level to 'debug' in order to get the trace messages from the parser.
I always forget to mention that because I never use User Trace in 'normal' mode |
The trace I have posted is an debug level trace. |
|
| Back to top |
|
|
| kimbert |
| Posted: Fri Jul 10, 2009 4:14 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| The trace I have posted is an debug level trace |
I can assure you that v6.1 TDS writes detailed trace output to debug-level user trace. I know because I put most of it in myself, and I have used it regularly.
Either you omitted it from your post, or you didn't set the trace level properly, or you're not on v6.1.
...or I'm going crazy. Mustn't discount that possibility  |
|
| Back to top |
|
|
|
|
