| Author |
Message
|
| chirayu |
 Posted: Tue Feb 07, 2012 4:44 am Post subject: DFDL in message broker v8 |
 |
|
Apprentice
Joined: 04 May 2011
Posts: 30
|
We are trying to develop a new DFDL parser for binary type of message where the elements are repeating themselves and the repeat count is dynamic which can be calculated from an arithmatic calculation of two fields.
eg
73ABCXYZ
where
7 = F1,3 = F2,
A = F3[1],B = F4[1], C = F5[1]
X = F3[2],Y = F4[2], Z = F5[2]
Repeating number is calculated with F1 and F2 as (F1-1)/F2 = (7-1)/3 = 2
Hence {F3,F4,F5} will repeate themselves only twice in same sequence
Can i do this arithmetic calulation with Xpath or DFDL expession in Occurs count kind (or anywhere else)? |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Tue Feb 07, 2012 5:46 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I'm not the expert here, but the dfdl parser should support this kind of structure.
The "Occurs Count Kind", if you set it to 'expression', will then allow you to input an expression into 'Occurs Count'.
So you would put a DFDL function expression in that represented (F1-1)/F2 into Occurs Count and it should do the right thing. |
|
| Back to top |
|
|
| exerk |
| Posted: Tue Feb 07, 2012 6:45 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
Moving this to the Broker forum...
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| kimbert |
| Posted: Tue Feb 07, 2012 7:50 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
Yes, DFDL is designed for exactly this type of thing.
| Code: |
element name="message"
element name="totalLength" type="xs:int"
element name="recordLength" type="xs:int"
element name="record" minOccurs="0" maxOccurs="unbounded"
occursCountKind="expression"
occursCount="{xs:unsignedInt((../totalLength - 1)/(../recordLength))}"
|
..and please note:
- I did not test this before posting it
- DFDL trace should reveal where things are going wrong, if they do. |
|
| Back to top |
|
|
| chirayu |
| Posted: Tue Feb 14, 2012 2:57 pm Post subject: |
|
|
Apprentice
Joined: 04 May 2011
Posts: 30
|
Thanks all for your help.
I could create a DFDL parser using occurKind as expression and got dynamically repeated data parsed.
now I am trying to implement it with a simple message broker flow with MQinput - Compute - MQOutput nodes.
Compute node has simple functionality to change the format of the message.
But while testing that flow, I have got following error in debugger
BIP5835
A DFDL serialization error occurred. A NameValue element was found to contain one or more child elements.
The element causing the problem is: <insert_1>1
Severity
20 : Error
Explanation
While writing the DFDL message, a NameValue element was found to contain one or more child elements.
A NameValue element in a DFDL message tree cannot contain child elements.
Response
Remove the child elements from the NameValue elements.
and the element which is pointed by debugger is the one with repeating fields.
Let me know your views over this. |
|
| Back to top |
|
|
| kimbert |
| Posted: Tue Feb 14, 2012 4:55 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| I could create a DFDL parser using occurKind as expression and got dynamically repeated data parsed. |
That's good.
| Quote: |
now I am trying to implement it with a simple message broker flow with MQinput - Compute - MQOutput nodes.
Compute node has simple functionality to change the format of the message. |
Please state what the input and output formats are. I *guess* you are receiving an XML message and writing using DFDL, Correct?
The error looks a bit strange. Please can you do this:
- disconnect the debugger
- add a Trace node in your flow after the Compute node
- Set the pattern in the Trace node to ${Root}
- Take a user trace
- Post the relevant part of the user trace |
|
| Back to top |
|
|
| chirayu |
| Posted: Wed Feb 15, 2012 1:46 am Post subject: |
|
|
Apprentice
Joined: 04 May 2011
Posts: 30
|
my input format is normal text message
eg. 73ABCXYZ
A = F3[1],B = F4[1], C = F5[1]
X = F3[2],Y = F4[2], Z = F5[2]
and i am trying to convert that to TLV format as
F3;1;A;F4;1;B;F5;1;C;F3;1;X;F4;1;Y;F5;1;Z;
I have a function written for the same.
I could convert till the last field, and at the end of the field I get following error in exception list. I am running with debugger. |
|
| Back to top |
|
|
| kimbert |
| Posted: Wed Feb 15, 2012 1:52 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Please can you supply the Trace node output, as requested. Without that, it's not possible to diagnose the problem accurately. |
|
| Back to top |
|
|
| chirayu |
| Posted: Wed Feb 15, 2012 2:22 am Post subject: |
|
|
Apprentice
Joined: 04 May 2011
Posts: 30
|
Trace output for ${Root}
( ['MQROOT' : 0x26806c78]
(0x01000000:Name):Properties = ( ['MQPROPERTYPARSER' : 0x21bd0680]
(0x03000000:NameValue):MessageSet = '' (CHARACTER)
(0x03000000:NameValue):MessageType = '{}:Test' (CHARACTER)
(0x03000000:NameValue):MessageFormat = '' (CHARACTER)
(0x03000000:NameValue):Encoding = 273 (INTEGER)
(0x03000000:NameValue):CodedCharSetId = 850 (INTEGER)
(0x03000000:NameValue):Transactional = TRUE (BOOLEAN)
(0x03000000:NameValue):Persistence = FALSE (BOOLEAN)
(0x03000000:NameValue):CreationTime = GMTTIMESTAMP '2012-02-14 19:48:53.950' (GMTTIMESTAMP)
(0x03000000:NameValue):ExpirationTime = -1 (INTEGER)
(0x03000000:NameValue):Priority = 0 (INTEGER)
(0x03000000:NameValue):ReplyIdentifier = X'000000000000000000000000000000000000000000000000' (BLOB)
(0x03000000:NameValue):ReplyProtocol = 'MQ' (CHARACTER)
(0x03000000:NameValue):Topic = NULL
(0x03000000:NameValue):ContentType = '' (CHARACTER)
(0x03000000:NameValue):IdentitySourceType = '' (CHARACTER)
(0x03000000:NameValue):IdentitySourceToken = '' (CHARACTER)
(0x03000000:NameValue):IdentitySourcePassword = '' (CHARACTER)
(0x03000000:NameValue):IdentitySourceIssuedBy = '' (CHARACTER)
(0x03000000:NameValue):IdentityMappedType = '' (CHARACTER)
(0x03000000:NameValue):IdentityMappedToken = '' (CHARACTER)
(0x03000000:NameValue):IdentityMappedPassword = '' (CHARACTER)
(0x03000000:NameValue):IdentityMappedIssuedBy = '' (CHARACTER)
)
(0x01000000:Name):MQMD = ( ['MQHMD' : 0x21c4ac60]
(0x03000000:NameValue):SourceQueue = 'TESTIN.Q' (CHARACTER)
(0x03000000:NameValue):Transactional = TRUE (BOOLEAN)
(0x03000000:NameValue):Encoding = 273 (INTEGER)
(0x03000000:NameValue):CodedCharSetId = 850 (INTEGER)
(0x03000000:NameValue):Format = ' ' (CHARACTER)
(0x03000000:NameValue):Version = 2 (INTEGER)
(0x03000000:NameValue):Report = 0 (INTEGER)
(0x03000000:NameValue):MsgType = 8 (INTEGER)
(0x03000000:NameValue):Expiry = -1 (INTEGER)
(0x03000000:NameValue):Feedback = 0 (INTEGER)
(0x03000000:NameValue):Priority = 0 (INTEGER)
(0x03000000:NameValue):Persistence = 0 (INTEGER)
(0x03000000:NameValue):MsgId = X'414d512053414d504c452e514d202020eba53a4f20005104' (BLOB)
(0x03000000:NameValue):CorrelId = X'000000000000000000000000000000000000000000000000' (BLOB)
(0x03000000:NameValue):BackoutCount = 0 (INTEGER)
(0x03000000:NameValue):ReplyToQ = ' ' (CHARACTER)
(0x03000000:NameValue):ReplyToQMgr = 'SAMPLE.QM ' (CHARACTER)
(0x03000000:NameValue):UserIdentifier = ' ' (CHARACTER)
(0x03000000:NameValue):AccountingToken = X'0000000000000000000000000000000000000000000000000000000000000000' (BLOB)
(0x03000000:NameValue):ApplIdentityData = ' ' (CHARACTER)
(0x03000000:NameValue):PutApplType = 0 (INTEGER)
(0x03000000:NameValue):PutApplName = ' ' (CHARACTER)
(0x03000000:NameValue):PutDate = DATE '2012-02-14' (DATE)
(0x03000000:NameValue):PutTime = GMTTIME '19:48:53.950' (GMTTIME)
(0x03000000:NameValue):ApplOriginData = ' ' (CHARACTER)
(0x03000000:NameValue):GroupId = X'000000000000000000000000000000000000000000000000' (BLOB)
(0x03000000:NameValue):MsgSeqNumber = 1 (INTEGER)
(0x03000000:NameValue):Offset = 0 (INTEGER)
(0x03000000:NameValue):MsgFlags = 0 (INTEGER)
(0x03000000:NameValue):OriginalLength = -1 (INTEGER)
)
(0x01000000:Name):DFDL = |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Feb 15, 2012 2:39 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
to quote Kimbert
| Quote: |
Post the relevant part of the user trace
|
There should be some messages output by the DFDL Parser when it tries to do its work.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| kimbert |
| Posted: Wed Feb 15, 2012 2:42 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
Thanks for the trace output...but there are two problems
- the output only shows the Properties folder and the MQMD header. The interesting part is the message body under 'DFDL' but that appears to be either missing or truncated.
- you should always use [c o d e] tags when posting log files or other formatted text. Otherwise it's really hard to read the text.
Did you disconnect the debugger before taking that trace? If not, that might be the reason for the empty body. If you can't find out what's going on, please take a user trace ( debug-level ) as I advised earlier - that will show the full text of any parsing errors that may be occurring. |
|
| Back to top |
|
|
| chirayu |
| Posted: Wed Feb 15, 2012 6:26 am Post subject: |
|
|
Apprentice
Joined: 04 May 2011
Posts: 30
|
| Code: |
( ['MQROOT' : 0x26ff1e18]
(0x01000000:Name):Properties = ( ['MQPROPERTYPARSER' : 0x21c12018]
(0x03000000:NameValue):MessageSet = '' (CHARACTER)
(0x03000000:NameValue):MessageType = '{}:Test' (CHARACTER)
(0x03000000:NameValue):MessageFormat = '' (CHARACTER)
(0x03000000:NameValue):Encoding = 273 (INTEGER)
(0x03000000:NameValue):CodedCharSetId = 850 (INTEGER)
(0x03000000:NameValue):Transactional = TRUE (BOOLEAN)
(0x03000000:NameValue):Persistence = FALSE (BOOLEAN)
(0x03000000:NameValue):CreationTime = GMTTIMESTAMP '2012-02-15 13:06:42.300' (GMTTIMESTAMP)
(0x03000000:NameValue):ExpirationTime = -1 (INTEGER)
(0x03000000:NameValue):Priority = 0 (INTEGER)
(0x03000000:NameValue):ReplyIdentifier = X'000000000000000000000000000000000000000000000000' (BLOB)
(0x03000000:NameValue):ReplyProtocol = 'MQ' (CHARACTER)
(0x03000000:NameValue):Topic = NULL
(0x03000000:NameValue):ContentType = '' (CHARACTER)
(0x03000000:NameValue):IdentitySourceType = '' (CHARACTER)
(0x03000000:NameValue):IdentitySourceToken = '' (CHARACTER)
(0x03000000:NameValue):IdentitySourcePassword = '' (CHARACTER)
(0x03000000:NameValue):IdentitySourceIssuedBy = '' (CHARACTER)
(0x03000000:NameValue):IdentityMappedType = '' (CHARACTER)
(0x03000000:NameValue):IdentityMappedToken = '' (CHARACTER)
(0x03000000:NameValue):IdentityMappedPassword = '' (CHARACTER)
(0x03000000:NameValue):IdentityMappedIssuedBy = '' (CHARACTER)
)
(0x01000000:Name):MQMD = ( ['MQHMD' : 0x2843a8f0]
(0x03000000:NameValue):SourceQueue = 'TESTIN.Q' (CHARACTER)
(0x03000000:NameValue):Transactional = TRUE (BOOLEAN)
(0x03000000:NameValue):Encoding = 273 (INTEGER)
(0x03000000:NameValue):CodedCharSetId = 850 (INTEGER)
(0x03000000:NameValue):Format = ' ' (CHARACTER)
(0x03000000:NameValue):Version = 2 (INTEGER)
(0x03000000:NameValue):Report = 0 (INTEGER)
(0x03000000:NameValue):MsgType = 8 (INTEGER)
(0x03000000:NameValue):Expiry = -1 (INTEGER)
(0x03000000:NameValue):Feedback = 0 (INTEGER)
(0x03000000:NameValue):Priority = 0 (INTEGER)
(0x03000000:NameValue):Persistence = 0 (INTEGER)
(0x03000000:NameValue):MsgId = X'414d512053414d504c452e514d202020eba53a4f2000bb03' (BLOB)
(0x03000000:NameValue):CorrelId = X'000000000000000000000000000000000000000000000000' (BLOB)
(0x03000000:NameValue):BackoutCount = 0 (INTEGER)
(0x03000000:NameValue):ReplyToQ = ' ' (CHARACTER)
(0x03000000:NameValue):ReplyToQMgr = 'SAMPLE.QM ' (CHARACTER)
(0x03000000:NameValue):UserIdentifier = ' ' (CHARACTER)
(0x03000000:NameValue):AccountingToken = X'0000000000000000000000000000000000000000000000000000000000000000' (BLOB)
(0x03000000:NameValue):ApplIdentityData = ' ' (CHARACTER)
(0x03000000:NameValue):PutApplType = 0 (INTEGER)
(0x03000000:NameValue):PutApplName = ' ' (CHARACTER)
(0x03000000:NameValue):PutDate = DATE '2012-02-15' (DATE)
(0x03000000:NameValue):PutTime = GMTTIME '13:06:42.300' (GMTTIME)
(0x03000000:NameValue):ApplOriginData = ' ' (CHARACTER)
(0x03000000:NameValue):GroupId = X'000000000000000000000000000000000000000000000000' (BLOB)
(0x03000000:NameValue):MsgSeqNumber = 1 (INTEGER)
(0x03000000:NameValue):Offset = 0 (INTEGER)
(0x03000000:NameValue):MsgFlags = 0 (INTEGER)
(0x03000000:NameValue):OriginalLength = -1 (INTEGER)
)
(0x01000000:Name):DFDL = ( ['dfdl' : 0x27017ec8]
(0x03000000:NameValue):Test = 'field1;3;TRM;field2;3;128;field3;2;62;field1;20;11001A12345ABC01FERN;field2;20;AND123456781234567.8;field3;22;91234567.891234567.89Y;' (CHARACTER)
(
(0x03000000:NameValue):field1 = 'TRM' (CHARACTER)
(0x03000000:NameValue):field2 = 128 (INTEGER)
(0x03000000:NameValue):field3 = 62 (INTEGER)
(0x01000000:Name ):field5 = (
(0x03000000:NameValue):field1 = '11001A12345ABC01FERN' (CHARACTER)
(0x03000000:NameValue):field2 = 'AND123456781234567.8' (CHARACTER)
(0x03000000:NameValue):field3 = '91234567.891234567.89Y' (CHARACTER)
)
(0x01000000:Name ):field5 = (
(0x03000000:NameValue):field1 = '11001A12345ABC01FERN' (CHARACTER)
(0x03000000:NameValue):field2 = 'AND123456781234567.8' (CHARACTER)
(0x03000000:NameValue):field3 = '91234567.891234567.89Y' (CHARACTER)
)
)
)
)
|
this is the complete trace node output.. and following is the snapshot in user trace
| Code: |
2012-02-15 11:20:46.332519 5600 UserTrace BIP5841I: ''Offset: 138. Finished processing element 'Test'.''
The DFDL component has written an informational diagnostic message.
Refer to the appropriate message in the embedded component's documentation.
2012-02-15 11:21:47.995395 5600 Error BIP2628E: Exception condition detected on input node 'Test_Flow.MQ Input'.
The input node 'Test_Flow.MQ Input' detected an error whilst processing a message. The message flow has been rolled-back and, if the message was being processed in a unit of work, it will remain on the input queue to be processed again. Following messages will indicate the cause of this exception.
Check the error messages which follow to determine why the exception was generated, and take action as described by those messages.
2012-02-15 11:21:47.995418 5600 RecoverableException BIP2230E: Error detected whilst processing a message in node 'Test_Flow.MQ Output'.
The message broker detected an error whilst processing a message in node 'Test_Flow.MQ Output'. An exception has been thrown to cut short the processing of the message.
See the following messages for details of the error.
2012-02-15 11:21:47.995433 5600 ParserException BIP5835E: A DFDL serialization error occurred. A NameValue element was found to contain one or more child elements.
The element causing the problem is: '/DFDL/Test'
While writing the DFDL message, a NameValue element was found to contain one or more child elements.
A NameValue element in a DFDL message tree cannot contain child elements.
Remove the child elements from the NameValue elements.
2012-02-15 11:21:48.990772 5600 Error BIP2648E: Message backed out to a queue; node 'Test_Flow.MQ Input'. |
|
|
| Back to top |
|
|
| kimbert |
| Posted: Wed Feb 15, 2012 7:07 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
A couple of questions:
- Why is the entire input bitstream copied into OutputRoot.DFDL.Test?
- Why is the entire output tree nested under OutputRoot.DFDL.Test?
- Do you have a global element in your DFDL xsd with name 'Test'? If not, your OutputRoot will not match the XSD ( I'm guessing that your global element is actually 'TRM' but you cannot expect the DFDL parser to search through OutputRoot until it finds something that matches the xsd ). |
|
| Back to top |
|
|
| chirayu |
| Posted: Wed Feb 15, 2012 8:40 am Post subject: |
|
|
Apprentice
Joined: 04 May 2011
Posts: 30
|
In CopyEntireMessage, I am copying entire inputroot tree to outputroot
OutputRoot = InputRoot
and then OutputRoot.DFDL.Test = make_TLV(InputRoot.DFDL.Test)
make_TLV is a function takes tree reference and returns characters |
|
| Back to top |
|
|
| kimbert |
| Posted: Wed Feb 15, 2012 8:47 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
In CopyEntireMessage, I am copying entire inputroot tree to outputroot
OutputRoot = InputRoot |
OK so far...
| Quote: |
and then OutputRoot.DFDL.Test = make_TLV(InputRoot.DFDL.Test)
make_TLV is a function takes tree reference and returns characters |
Please explain
a) why you are doing this second step
b) how the entire InputRoot tree has become the *children* ( nested inside ) of OutputRoot.DFDL.Test. |
|
| Back to top |
|
|
|
|
