| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Parsing the Industry standard files using DFDL parser |
« View previous topic :: View next topic » |
| Author |
Message
|
| hiitsjam |
 Posted: Sat Nov 03, 2012 5:51 am Post subject: Parsing the Industry standard files using DFDL parser |
 |
|
Newbie
Joined: 20 May 2012
Posts: 4
|
Hello everyone,
I am exploring new features of WMB 8.0 especially the DFDL parser. I have successfully parsed a CSV file and transformed it into a COBOL binary file using DFDL parser (WMB 8.0) and this works like a magic. Pretty cool 
As I understand, DFDL can be used to represent virtually any file format. I am trying to take my R&D to the next level where I want to parse and transform an industry standard file (it could be any industry standard file for example, EDI, SWIFT etc) into some other format (lets say XML /CSV / COBOL binary etc) using the DFDL parser available in WMB 8.0
I have seen and gone through the samples that are shipped through WMB 8.0 which uses DFDL parsers but none of those samples are using industry standard formats with DFDL parsers. IBM has demonstrated DFDL parsers only for CSV files.
When I try to create a DFDL message model for industry standard formats (for example, EDI format, SWIFT format), the new message model dialog box shows the following options :
XML
SOAP XML
Other XML
Text and binary
CSV text
Record Oriented Text
COBOL
Other text or binary
Enterprise Information System
SAP
Siebel
PeopleSoft
JD Edwards
Other
CORBA IDL
Database Record
MIME
IBM Supplied
Not sure which option to select here for industry standard formats. It must be something (either Record Oriented Text or Other text or binary) from Text and Binary section. When I proceed with Record Oriented Text option, I get to see an screen where it asks for header , body and trailer fields. I am just not getting how to do all these settings for an industry standard format like EDI formats (837 electronic claims, 835 claim payment advice file etc), SWIFT formats etc.
If some one has already done such kind of exercise before, would love to hear their experience on this.
Pls note that, I do not want to use WTX for transformation here. Also, I have already tried this exercise with MRM parsers (IBM samples and they work as expected), but here I want to do the same using DFDL parsers.
Any help here shall highly be appreciated and sorry for the long post.
Thanks
Jameel |
|
| Back to top |
|
 |
| Vitor |
| Posted: Sat Nov 03, 2012 7:07 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Things like SWIFT, EDI, HL7 are all text based formats. In my view you'd start with the Record Orientated Text but others with more experience of WMBv8 may legitimately disagree.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sat Nov 03, 2012 1:22 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
I know for a fact that DFDL can handle IATA Type B messages. These are sometimes know as 'SitaTex' messages.
These are most certainly not CVS of XML types.
There is an example (from wikipeda}
| Code: |
QD AAAABBB. - IATA message to AAAABBB (priority=deferred)
.XXXXYYY 111301 - from (origin code + timestamp)
ASM - IATA type (keyword). Identifies type of message
UTC - Time mode (Coordinated Universal Time). UTC or LOCAL
27SEP03899E001/TSTF DL Y - Message Reference line
NEW - ASM subtype (Action Identifier)
BA667/13APR - Flight and date of flight;
J 319 C1M25VVA4C26 - fleet & equipment information
LHR1340 BCN1610 - Station/time for depart & arrival
LHRQQQ 99/1 - Route information. 99 is the passenger departure terminal code
QQQBCN 98/A - Route information. 98 is the passenger arrival terminal code
QQQQQQ 906/PAYDIV B - Route information.
LHRQQQ 999/1 - Route information. 999 is the aircraft arrival terminal code
QQQBCN 998/A - Route information. 998 is the aircraft departure terminal code
SI
|
I'm part way through developing a message model that handles AFTN messages.
http://origin.www.faa.gov/about/office_org/headquarters_offices/ato/service_units/enroute/flight_plan_filing/filing/Reference_Guide/Message_ACK_REJ/message_examples/index.cfm?print=go
This shows that essentiall the same message can be different according to where it originates.
Both of these (STX and AFTN) are really contact admin formats. For example, there is one Operator in Montreal that always puts extra spaces in the Sitatex messages and will often transpose the digits of the Flight Number. You have to be able to handle the extra spaces in the parser. DFDL can do that in ways that were almost impossible before.
I could not have developed these models with the MRM parser and I have to give my thanks to Steve Hanson (IBM Hursley) for showing me how these message types could be modelled.
It is early days for the DFDL Parser. I can see its advantages in that tools other than message broker in that you will be able to share the xsd that describes the message between very different applications (n the future)
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| kimbert |
| Posted: Sat Nov 03, 2012 2:12 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| I have successfully parsed a CSV file and transformed it into a COBOL binary file using DFDL parser (WMB 8.0) and this works like a magic. Pretty cool |
Thanks for the feedback - we like it a lot too, but it sounds better coming from the users
| Quote: |
| As I understand, DFDL can be used to represent virtually any file format. |
Correct - it's just a description language for data formats.
| Quote: |
| I am trying to take my R&D to the next level where I want to parse and transform an industry standard file (it could be any industry standard file for example, EDI, SWIFT etc) into some other format (lets say XML /CSV / COBOL binary etc) using the DFDL parser available in WMB 8.0 |
Great idea. DFDL can certainly do those formats, and learning how to do it will teach you a lot about DFDL.
| Quote: |
| I have seen and gone through the samples that are shipped through WMB 8.0 which uses DFDL parsers but none of those samples are using industry standard formats with DFDL parsers. IBM has demonstrated DFDL parsers only for CSV files. |
Last sentence is not quite true but your point stands - more of the samples need to be converted to use DFDL. And will be - it's just a matter of time.
My advice is to pick 'Other text or binary' and start from scratch. All of the other options will take you down a route which is aimed at a particular family of data format. As you have already realised, we don't yet have options for the industry standard formats, so there is no other good choice.
'Other text and binary' will effectively give you an empty xsd that contains a set of defaults for all of the DFDL properties. That prevents the Problems view from lighting up like a Christmas tree as soon as you start modelling anything - but take care, because the defaults may well not be exactly what you need.
I'm happy to provide help via this forum if and when further questions arise. |
|
| Back to top |
|
|
| mgk |
| Posted: Sat Nov 03, 2012 3:45 pm Post subject: |
|
|
Padawan
Joined: 31 Jul 2003
Posts: 1642
|
Also, WMB 8.0.0.1 includes a new ISO 8583 DFDL sample. ISO 8583 is a widely used binary standard for transmission of ATM and credit card transactions and the sample includes a DFDL schema for the 1987 version of the standard plus message flows to transform to/from XML.
Kind regards,
_________________
MGK
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
