ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » DFDL from XML

Post new topic  Reply to topic Goto page 1, 2  Next
 DFDL from XML « View previous topic :: View next topic » 
Author Message
CUrquhartHP
PostPosted: Wed Jul 10, 2013 5:49 am    Post subject: DFDL from XML Reply with quote

Newbie

Joined: 10 Jul 2013
Posts: 6

Hi
I read a tutorial showing how to create DFDL schema from CSV in WebSphere Message Broker Toolkit and assumed (based on the screenshots) I could also create a DFDL schema from XML, but after a play it appears I can't.

I do:

"New Message Model", select "Other XML" then press Next. At this stage I expect an option for DFDL but it is not offered.

Am I correct that it's not possible with WebSphere? (And in fact with no other software too it appears).
Back to top
View user's profile Send private message
lancelotlinc
PostPosted: Wed Jul 10, 2013 5:52 am    Post subject: Reply with quote

Jedi Knight

Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA

DFDL

http://publib.boulder.ibm.com/infocenter/wmbhelp/v8r0m0/index.jsp?topic=%2Fcom.ibm.etools.mft.doc%2Fbc40130_.htm

Quote:
The following samples use the DFDL parser to process messages:
COBOL
Comma Separated Value (CSV)


1. Read the InfoCentre.

2. Fully investigate all samples.

3. Take the training.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER
Back to top
View user's profile Send private message Send e-mail
smdavies99
PostPosted: Wed Jul 10, 2013 6:12 am    Post subject: Re: DFDL from XML Reply with quote

Jedi Council

Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.

CUrquhartHP wrote:
Hi
I could also create a DFDL schema from XML, but after a play it appears I can't.



The DFDL Parser is not for XML messages. It is for things like TDS, CSV or CobolCopybook (flat file) formats. The only bit of XML that it uses is the DSDL message description is stored in an XML Schema file (.xsd).
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995

Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions.
Back to top
View user's profile Send private message
CUrquhartHP
PostPosted: Wed Jul 10, 2013 6:29 am    Post subject: Reply with quote

Newbie

Joined: 10 Jul 2013
Posts: 6

Thanks.

Our requirement is to convert from XML to formatted binary (formatted in that it conforms to an ICD), so DFDL is the chosen way to convert from XML to binary and from binary to XML (which can be done via Daffodil). However, DFDL schema is cumbersome to write for a large data set so I was wanting to find a tool to ease the burden.

Thanks for your replies.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Wed Jul 10, 2013 6:46 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

DFDL does not convert anything.

DFDL models your complex binary data. You can then use Broker code to convert from a logical message tree that matches your XML into a logical message tree that matches the DFDL document.

The DFDL tooling in Broker Toolkit comes with a number of importers and a number of predefined formats that may make it easier to develop a model for your specific data.

The DFDL model editor, particularly in v8.0.0.2 and later, is very easy to use to construct message models from scratch. You can then use the DFDL Test tools in Toolkit to parse and serialize sample messages, without writing any broker code or deploying anything to runtime. This will allow you to develop your model and test it completely independently of the code that needs to use the model.

It's not clear what an ICD is. If this is an existing message format, there may be existing machine-consumable definitions that can be used. For example, there may be COBOL copybooks or C header files. These can be imported by the DFDL tooling and create your model for you.

You can also review the DFDL working group website and github and see if there is an existing open source model for your data format: http://www.ogf.org/dfdl/
Back to top
View user's profile Send private message
CUrquhartHP
PostPosted: Wed Jul 10, 2013 7:01 am    Post subject: Reply with quote

Newbie

Joined: 10 Jul 2013
Posts: 6

I probably didn't fully explain myself:

We weren't looking to use WebSphere as a message flow solution, merely use the DFDL editor to create a DFDL schema rather than hand write it. As a bonus, it would have been convenient to have the editor analyse the XML file and create the DFDL XSD from this. We would have then used the generated XSD in our application. We have to convert to and from XML / binary.

There seems to be a lack of tools for DFDL.

ICD = interface control document
Back to top
View user's profile Send private message
mqjeff
PostPosted: Wed Jul 10, 2013 7:05 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

You could use Toolkit to analyze the XML document and create an XSD.

You could then tell Toolkit to open that XSD with the DFDL editor.

And then go through and hand-edit each element to ensure that it has the necessary DFDL annotations attached so that it is now a valid DFDL schema, rather than an XML Schema.

This assumes the structure of the XML message is exactly the same as the structure of the binary data that needs to be parsed with DFDL....

Again, the DFDL working group page should have links to non IBM DFDL tools. Whether or not they are better or worse than Toolkit, I wouldn't know. And, again, a DFDL schema is just an XML Schema with additional annotations, so as long as you're willing to hand-create those annotations, you can use any schema tools and a text editor.
Back to top
View user's profile Send private message
CUrquhartHP
PostPosted: Wed Jul 10, 2013 7:39 am    Post subject: Reply with quote

Newbie

Joined: 10 Jul 2013
Posts: 6

Yes we could hand write in XMLSpy / Eclipse but this will be a huge pain, considering the XML has 1500+ attributes.

The XML will be in the same format as the output binary in that the elements' data in the XML will retain the same order.

I'm playing with the DFDL editor and will try what you suggest.


Thanks
Back to top
View user's profile Send private message
kimbert
PostPosted: Wed Jul 10, 2013 7:41 am    Post subject: Reply with quote

Jedi Council

Joined: 29 Jul 2003
Posts: 5542
Location: Southampton

Quote:
We weren't looking to use WebSphere as a message flow solution, merely use the DFDL editor to create a DFDL schema rather than hand write it.
If not WebSphere ( I assume you mean WMB ) then which product *are* you looking to use?
Quote:
As a bonus, it would have been convenient to have the editor analyse the XML file and create the DFDL XSD from this.
So you were hoping to get the logical structure of the XSD created automatically? There are plenty of tools out there that will create you an XSD from an example XML document. But you will need to get the DFDL annotations added to that XSD. That will require knowledge of your data format ( ICD? ). You surely did not expect the WMB toolkit to automatically know about your data format?

It would be useful if you could post an example XML document and the equivalent binary format that you want to produce.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Wed Jul 10, 2013 7:47 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

It looks like even the Broker v8 Development perspective lets you right-click on an xml file and choose "Generate->XML Schema".

Ideally, though, you'd already have an existing XML schema that matched the data... either an internally generated one or from whatever standards group is defining these ICD messages...

And again, if there's already some kind of documentation from that standards body of the binary format ( a c-header file, or etc.) it may be easier to start from that rather than start from a schema. Less information to add in manually afterwards...

But you're the only one that knows if ICD is meaningful outside your organization or not.
Back to top
View user's profile Send private message
CUrquhartHP
PostPosted: Wed Jul 10, 2013 7:49 am    Post subject: Reply with quote

Newbie

Joined: 10 Jul 2013
Posts: 6

Quote:
If not WebSphere ( I assume you mean WMB ) then which product *are* you looking to use?


Yes I meant WMB. I have little knowledge of WebSphere / WMB etc


Quote:
So you were hoping to get the logical structure of the XSD created automatically? There are plenty of tools out there that will create you an XSD from an example XML document. But you will need to get the DFDL annotations added to that XSD. That will require knowledge of your data format ( ICD? ). You surely did not expect the WMB toolkit to automatically know about your data format?


It's the DFDL annotations I was hoping to have automated in some way, even just to reduce the workload slightly. The DFDL open group have said in a document (can't find it now) they don't provide a parser (parsing to binary from some source format) because it's difficult and needs further thought/planning, so it was a long shot!
Back to top
View user's profile Send private message
mqjeff
PostPosted: Wed Jul 10, 2013 7:53 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

Again, broker toolkit new dfdl wizard will let you import physical structure data from things like c header files and cobol copybooks.

XML Schemas don't contain any of this information because they presume an XML physical format...

So you need some other source document that has physical information in it.
Back to top
View user's profile Send private message
smdavies99
PostPosted: Wed Jul 10, 2013 11:21 am    Post subject: Reply with quote

Jedi Council

Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.

In my Organisation we use ICD's all the time

They are what you give a developer so that they can code 'something'. Ideally thay should have all the information needed for that developer to create the end product or in this case, an Interface (a posh name for a message flow )


It seems to me that the OP is missing one key component and that is the DFDL parser that understands all those lovely schema annotations and creates some form of output bitstream. IT is all very well having this nice schema and all the DFSL annotations but unless you can use that to create the output (or read the input) it is surely a waste of time and effort.

If the OP has access to a parser then using the TK might be a good way to get the schema. However there is no telling that the parser he has works in the same way as the IBM Message Broker version and that it supports the same level of functionality as the IBM one.

Then there is the matter of Licensing. If the OP does not have a valid runtime broker license then perhaps they are using the TK in violation of the IBM T's & C's?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995

Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions.
Back to top
View user's profile Send private message
CUrquhartHP
PostPosted: Thu Jul 11, 2013 12:30 am    Post subject: Reply with quote

Newbie

Joined: 10 Jul 2013
Posts: 6

Quote:
Then there is the matter of Licensing. If the OP does not have a valid runtime broker license then perhaps they are using the TK in violation of the IBM T's & C's?


That is another issue to deal with, but that can wait until I've played/evaluated with the TK a bit more, but I suspect we'll need a licence if using a DFDL schema created by the TK DFDL editor, although unless it offers any significant advantage overhand written (in any XML editor) then it won't be worth it.

Quote:
It seems to me that the OP is missing one key component and that is the DFDL parser that understands all those lovely schema annotations and creates some form of output bitstream


Yes. There is an open source tool (Daffodil) that can transform binary data to XML (parsing) but there aren't tools to unparse (source data to binary) - as I said the people that created Daffodil have delayed it due to complexities. So it seems we're stuck, hence trying to find a tool that helps in anyway.

I started the thread without fully explaining what I required and how it would be used. Apologies for confusion.
Back to top
View user's profile Send private message
shanson
PostPosted: Thu Jul 11, 2013 3:17 am    Post subject: Reply with quote

Partisan

Joined: 17 Oct 2003
Posts: 344
Location: IBM Hursley

WMB has a developer edition which you can download for free. This includes the MB Toolkit, which includes the IBM DFDL editor for creating DFDL schemas.

WMB developer edition also allows you to use IBM DFDL Java parser & serializer independently of the WMB runtime broker. There's a zip that you can take and use, either on the same machine or on a different machine. This gives you the same capability as Daffodil, but you get a serializer.

All that is freely available for developing the DFDL schemas and DFDL application. If you want to use your schemas and application to run a real production workload, then that is when you start paying. There's an edition of WMB called 'Express' which is the cheapest to purchase. A stand-alone DFDL application is charged the same as an instance of WMB runtime broker.

There's more info in the WMB infocenter.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » DFDL from XML
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.