| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| IIB : MRM with Parsed Record Sequence Query |
« View previous topic :: View next topic » |
| Author |
Message
|
| wmbv7newbie |
 Posted: Mon Apr 11, 2016 3:10 pm Post subject: IIB : MRM with Parsed Record Sequence Query |
 |
|
Centurion
Joined: 13 May 2014
Posts: 121
|
Hi,
I have a requirement where I get a bunch of records as repeatig blocks in a file. I need to process only the valid records from the file and ignore the corrupt ones. The structure of file looks like -
| Code: |
0SOC00001 GROS201506223
22029938276Mauricio Silva QUEEN
4000112920184020000001 1
80000001 0001
22029948276Richard Connolly DEPT
4000112920724310000001 1
80000001 0001
22029968276PAOLO CAVALIERE UNIVERSITY
4000114082216160000001 1
80000002 0002
90000098 0098
|
The line starting with 0 is file header and will occur once. The block having lines starting with 2, 4 and 8 will have 1 to -1 (unbounded) occurrences with 2 having 1 occurrence, 4 having 1 to -1 (unbounded) and 8 having 1 occurrence. I need to read the repeating block of 2,4,8 and output as one message from the file node. I think I need to do a Parsed Record Sequence to achieve this. I have created a MessageSet as follows -
| Quote: |
Root
-- LocalComplexType (Fixed length)
----Header (Tag 0) 1 to 1
------Field1
------Field2...
----Lines 1 to -1
-----sequence (Tagged delimited)
--------LineType2
----------sequence (Tag 2, Fixed length, 1 to 1)
-----------Field1
-----------Field2...
--------LineType4
----------sequence(Tag 4, Fixed length, 1 to -1)
-----------Field1
-----------Field2...
--------LineType8
----------sequence(Tag 8, Fixed length, 1 to 1)
-----------Field1
-----------Field2...
----Trailer (Tag 9) 1 to 1
------Field1
------Field2...
|
but I am getting different kinds of errors like -
| Quote: |
Element 'OrderBody' has unbounded maxOccurs when the enclosing structure has Data Element Separation 'FixedLength'. It must be the last member of the structure. Physical format: 'Text1'.
Missing a group indicator for complex type or group
|
I have read through all the posts around this issue on the forum but unfortunately, not many ended with a solution. Can someone guide me about resolving this. Will put the user trace also in a while.
Thanks!
Last edited by wmbv7newbie on Tue Apr 12, 2016 6:11 am; edited 1 time in total |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Mon Apr 11, 2016 10:20 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
You need to set up a choice:
- header
- record sequence occurs unbounded
- trailer
This should allow you to parse. Don't. forget to use the adequate discriminators.
And i would suggest you use DFDL instead of MRM
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Apr 12, 2016 12:30 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Please OBFUSCATE any data that you put out on this forum.
Perhaps the people you identified in your poast might not want to have their address and phone number posted on the internet?
Please go back and edit any data that allows people to be identified.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| timber |
| Posted: Tue Apr 12, 2016 12:57 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1280
|
| Quote: |
| I have a requirement where I get a bunch of records as repeatig blocks in a file. I need to process only the valid records from the file and ignore the corrupt ones. |
Parsed Record Sequence uses the parser ( MRM in this case ) to work out where the record boundariese are. If the MRM parser throws an exception on any record then the processing of the file will stop.
I think you need 2 models. The first model splits the file on lines that begin with '0'. This is the model that you specify on the FileInput node. The second model is a properly-structured model that re-parses the BLOB propagated by the FileInput node. You can put a ResetContentDescriptor node after the FileInput node to do that. Catch exceptions from the RCD node to process 'corrupt' records.
You can use the MRM parser to do this, but DFDL would probably be a lot easier. DFDL allows you to test the model in the toolkit, and it gives *far* better diagnostics than the MRM parser. |
|
| Back to top |
|
|
| wmbv7newbie |
| Posted: Tue Apr 12, 2016 6:35 am Post subject: |
|
|
Centurion
Joined: 13 May 2014
Posts: 121
|
@smdavies99 - Although thats dummy data, i have removed most of the content now.
@timber and @fjb_saper - Line 0 and 9 will come only once in one file. The block with lines 2, 4 and 8 will repeat unbounded. Inside that too, line 4 will repeat unbounded.
I have managed to create a DFDL that correctly parses the file and gives desired output but it is processing only as whole file. I might not have been clear before, so the requirement exactly is -
Each record block with 2,4 and 8 lines (this block repeats unbounded times, and each time inside it, line 4 also repeats unbounded times), which has corrupt records needs to be skipped and we should continue processing the next block.
I wanted to create separate messages out of the file node itself for each block of 2, 4 and 8 lines so that each can be processed at a time, skipped if corrupt and control will go back to the file node to pick next block.
But looks like this is challenging to do in DFDL. I am now thinking to do in esql itself.
It would be great if you could suggest a way to achieve this using DFDL. Thanks! |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Apr 12, 2016 6:49 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Are 'corrupt records' records with invalid values (by business logic, not by model restrictions - i.e. field length) ? Or are they records that don't meet model restrictions?
If the records contain invalid values, you can create a model used by the FileInput node that knows about the field level structures of the records.
If the records are structurally wrong, then you will have to create a model used by the FileInput node that knows only about the high level record structure. Something like Indicator:2, data: string,delimiter \n
Do the blocks of 2 or 4 records always come as a set? or do you need to deal with a block like {2-record,4-record,2-record,2-record,2-record,4-record} ?
You will have to construct the model to handle that, and do either some kind of aggregation of individual records, or parse the whole file at the FileInput node.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| wmbv7newbie |
| Posted: Tue Apr 12, 2016 6:54 am Post subject: |
|
|
Centurion
Joined: 13 May 2014
Posts: 121
|
| Quote: |
| Are 'corrupt records' records with invalid values (by business logic, not by model restrictions - i.e. field length) ? Or are they records that don't meet model restrictions? |
Which will give parsing errors like lenght mismatch, invalid characters etc.
| Quote: |
| Do the blocks of 2 or 4 records always come as a set? or do you need to deal with a block like {2-record,4-record,2-record,2-record,2-record,4-record} ? |
Yes. They always come in same order. 2data, 4data, 4data, 8data, 2data, 4data,8data....and so on
| Quote: |
| You will have to construct the model to handle that, and do either some kind of aggregation of individual records, or parse the whole file at the FileInput node. |
Yes. Parse at whole file level is working for me. But in esql, i'll have to loop through all the records one by one and skip if any record gives an exaception. I wanted to save this and do the bit at DFDL parser itself. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Apr 12, 2016 6:58 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| wmbv7newbie wrote: |
| Quote: |
| Are 'corrupt records' records with invalid values (by business logic, not by model restrictions - i.e. field length) ? Or are they records that don't meet model restrictions? |
Which will give parsing errors like lenght mismatch, invalid characters etc. |
Use a model at the FileInput that doesn't throw exceptions for those kinds of things.
| wmbv7newbie wrote: |
| Quote: |
| Do the blocks of 2 or 4 records always come as a set? or do you need to deal with a block like {2-record,4-record,2-record,2-record,2-record,4-record} ? |
Yes. They always come in same order. 2data, 4data, 4data, 8data, 2data, 4data,8data....and so on |
So you don't get all of the 2-records followed by all of the 4-records followed by all of the 8-records.
If each 2-record has a set of 4-records and a set of 8-records in some order, you can construct a model that knows that and can propagate a single 2-record, including all children, out of the FileInput node. Again, using a model that doesn't know about the structure inside those records.
| wmbv7newbie wrote: |
| Quote: |
| You will have to construct the model to handle that, and do either some kind of aggregation of individual records, or parse the whole file at the FileInput node. |
Yes. Parse at whole file level is working for me. But in esql, i'll have to loop through all the records one by one and skip if any record gives an exaception. I wanted to save this and do the bit at DFDL parser itself. |
Use a second model, that knows about the internal records. If a given 2-record coming out of the FileInput fails, then just catch the exception and return true.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| timber |
| Posted: Tue Apr 12, 2016 8:12 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1280
|
Just to confirm; mqjeff is 100% correct. You have exactly 2 choices here:
1. Set Records and Elements to 'Delimited' and propagate individual lines from the FileInput node to the rest of the flow. Write some logic in the message flow to check the first character of each line and assemble lines into valid groups. Re-parse/validate each group using the MRM message set.
2. Create a message set that defines an optional Header, followed by a repeating sequence group ( not a repeating element ) containing the 2-record, 4-record and 8-record, followed by an optional trailer. This will allow some invalid file contents ( it will allow no header, and/or a header with every 2,4,8 group, and/or a trailer with every 2,4,8 group and/or no trailer at all ). That's unavoidable when you use Parsed Record Sequence to parse a file with a header/trailer. You could always add some logic in the message flow to check for the illegal combinations.
If I was doing it, I'd probably opt for 1. unless validation of the header/trailer was unnecessary. |
|
| Back to top |
|
|
| wmbv7newbie |
| Posted: Tue Apr 12, 2016 10:42 am Post subject: |
|
|
Centurion
Joined: 13 May 2014
Posts: 121
|
Thanks.
| Quote: |
wmbv7newbie wrote:
Quote:
Do the blocks of 2 or 4 records always come as a set? or do you need to deal with a block like {2-record,4-record,2-record,2-record,2-record,4-record} ?
Yes. They always come in same order. 2data, 4data, 4data, 8data, 2data, 4data,8data....and so on
So you don't get all of the 2-records followed by all of the 4-records followed by all of the 8-records.
If each 2-record has a set of 4-records and a set of 8-records in some order, you can construct a model that knows that and can propagate a single 2-record, including all children, out of the FileInput node. Again, using a model that doesn't know about the structure inside those records. |
This is exactly what I want to achieve at this point. So I went with creating a DFDL as below -
Order
-- sequence occurs exactly once
---- Header occurs exactly once
-------sequence occurs exactly once
----------field1, field 2 etc. all fixed length
---- Record occurs unbounded
-------sequence occurs exactly once
----------field1, initiator 2 but not sure how to end this to include n no. of 4s and 8s until next 2 occurs (delimiter?)
---- Trailer occurs exactly once
-------sequence occurs exactly once
----------field1, field 2 etc. all fixed length
After this, I will create a second DFDL which will have clear declaration of 2, 4 and 8 so that I can individually read each block of record. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Apr 12, 2016 10:47 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| wmbv7newbie wrote: |
---- Record occurs unbounded
-------sequence occurs exactly once
----------field1, initiator 2 but not sure how to end this to include n no. of 4s and 8s until next 2 occurs (delimiter?)
------------- <choice> |
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| wmbv7newbie |
| Posted: Tue Apr 12, 2016 11:49 am Post subject: |
|
|
Centurion
Joined: 13 May 2014
Posts: 121
|
thanks a ton @mqjeff!!! I am so close now.
Input -
| Quote: |
0SOC00001
23036178280Gerard van der
40001184776138
40001184776138
40001184776138
80000001 0001
23036178280Gerard van der
40001184776138
80000001 0001
90000098 |
DFDL -
| Quote: |
Order
-- sequence occurs exactly once
---- Header occurs exactly once, initiator 0
-------sequence occurs exactly once
----------field1, field 2 etc. all fixed length
---- OrderBody occurs unbounded
-------sequence occurs exactly once
----------choice, initiator 2, delimited
------------field1
---- Trailer occurs exactly once, initiator 9
-------sequence occurs exactly once
----------field1, field 2 etc. all fixed length |
Issue 1 is the exception -
| Quote: |
| CTDP3060E: Initiator '9' not found at offset '191' for sequence or choice within element '/NewDFDL[1]/Trailer[1]'. |
I think I am missing some termination setting in Choice.
Issue 2 is that although I have set Record Parsed Sequence, I am not getting separate records for the 2s block. It is all coming as -
| Quote: |
OrderBody
field1:CHARACTER:3036178280Gerard van der
40001184776138
40001184776138
40001184776138
80000001 0001
23036178280Gerard van der
40001184776138
40001184776138
40001184776138
80000001 0001
90000098
|
|
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Apr 12, 2016 11:59 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
record type 2 is not the choice.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| wmbv7newbie |
| Posted: Tue Apr 12, 2016 12:03 pm Post subject: |
|
|
Centurion
Joined: 13 May 2014
Posts: 121
|
| 2 is initiator and record type 4 and 8 are choices? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Apr 12, 2016 12:05 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| wmbv7newbie wrote: |
| 2 is initiator and record type 4 and 8 are choices? |
There is a child of 2 that contains a choice of 4 and 8. This repeats unbounded. 4 and 8 repeat once.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
|
|
|
  |
Goto page 1, 2 Next |
Page 1 of 2 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
