| Author |
Message
|
| scravr |
 Posted: Mon Sep 22, 2008 5:48 am Post subject: Broker 6.1: XSD and mapping |
 |
|
Partisan
Joined: 03 Apr 2003
Posts: 391
Location: NY NY USA 10021
|
HI All,
To reduce number of XSD files and maintain less flows, I have this strange request to combine definition of many message types into one XSD (one message set) and map each incoming message to one output message.
Basically, I need to create one XSD to support definition of all SWIFT msg-types, one XSD to support all FIX msg-types, and one XSD to support internal msg-types.
Then map from/to: SWIFT from/to internal, FIX from/to internal, and internal from/to internal, etc.
How can I create one XSD (one message-set) from all SWIFT messages and map to one of output formats?
How can I create one XSD (one message-set) from all FIX messages and map to one of output formats?
How can I create one XSD (one message-set) from all my other internal messages and map to one of output formats? I have many internal formats (XML, JAVA, COBOL, tag/untagged fields, etc)
I guess I need MXM parser. Should I let it validate input messages or do it myself in compute/java nodes?
Any Ideas?
Thanks,
Moshe |
|
| Back to top |
|
 |
| kimbert |
| Posted: Mon Sep 22, 2008 7:43 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
So you're trying to define a 'canonical format' for all messages in your system.
| Quote: |
| How can I create one XSD (one message-set) from all SWIFT messages and map to one of output formats? |
The question does not make sense. You can have many XSDs ( i.e. many message definition files ) in one message set.
| Quote: |
I have many internal formats (XML, JAVA, COBOL, tag/untagged fields, etc)
I guess I need MXM parser |
Yes, you will need the MRM parser for the non-XML formats. You should use XMLNSC for the XML formats.
| Quote: |
| Should I let it validate input messages or do it myself in compute/java nodes? |
As a general rule, it's a bad idea to write validation code if your data definitions can do it for you. If you can code the validation rules into your canonical format ( so using XML Schema's validation rules ) then you should do that. You may need some code to implement validation rules which the XML schema standard does not support.
I suggest that you start by becoming a real expert in XML Schema.  |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Sep 22, 2008 8:03 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Also, you need to be clear.
XML Schema Documents (XSDs) can only describe XML documents.
You can use Message Sets to describe SWIFT and FIX messages. Those happen to use XML Schema Documents (XSDs) "internally", but that doesn't mean that those XSDs make sense as XSDs. They are .mxsd files and not.xsd files for a reason.... they are "valid" XSDs, but they are not meaningful outside of Broker.
Also, there is a point of diminishing returns in creating large message set projects. You will cause yourself trouble if you attempt to create and deploy a single message set that covers every single message that is at all possible under the SWIFT messsaging standard... You are much better off creating message sets that refer to one or more common message sets and contain sections of the standards in ways that make sense for your projects. |
|
| Back to top |
|
|
| kimbert |
| Posted: Mon Sep 22, 2008 11:24 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| Those happen to use XML Schema Documents (XSDs) "internally", but that doesn't mean that those XSDs make sense as XSDs. They are .mxsd files and not.xsd files for a reason.... they are "valid" XSDs, but they are not meaningful outside of Broker. |
This is nonsense! An mxsd file is, by design, a 100% standard XML schema file.
An mxsd contains schema annotations to describe non-XML formats and other broker extensions to schema. But it is still a perfectly normal XSD. The .mxsd extension is purely to distinguish it from the non-annotated xsds in the workspace. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Sep 22, 2008 11:45 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You can not plug an MXSD that describes an MRM-TDS message into a standard XML parser, and have that engine validate or parse a TDS message.
This is exactly and only what I mean by "they are 'valid' XSDs, but they are not meaningful outside of Broker". |
|
| Back to top |
|
|
| kimbert |
| Posted: Mon Sep 22, 2008 12:13 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
Some TDS mxsd files contain extra groups, and are more complex than the XSD which you would have designed for the equivalent XML structure. I think that was your point, and that much is correct.
This is the sentence that I have a problem with:
| Quote: |
| that doesn't mean that those XSDs make sense as XSDs |
In fact, the vast majority of mxsds are identical in structure to the xsd which was imported.
You could plug an mxsd that describes an MRM-TDS message into a standard XML parser, and have that engine validate an XML message ( one with the same logical structure as the TDS message) . And you could do that without even removing the annotations. They are 100% standard XSDs - but some of them have a lot of funny-looking annotations with internal and undocumented meanings. |
|
| Back to top |
|
|
| scravr |
| Posted: Wed Sep 24, 2008 11:44 am Post subject: Broker 6.1: XSD and mapping |
|
|
Partisan
Joined: 03 Apr 2003
Posts: 391
Location: NY NY USA 10021
|
I run few experiment s:
1. You can load/import and create msg set from xsd developed by xmlspy.
2. You can create new msg set/definition with broker toolkit and then open and validate it with NetBeans (just change the file name from xyz.msxd to xyz.xsd).
So xsd vs. mxsd is not an issue.
The basic question I had/have:
I have many formats (xml, swift, fix, few-diff-internal-formats) that can be on both input and output and need to reformat/mapping between.
Meaning: I can have same data/information comes into flow as xml or swift or fix or internal-format and then need to output sometimes as xml or as swift or as fix or as other-internal-format. (YES, it sounds like reformatter utility)
So what should I do:
A. Create 1 msg set with many msg definitions?
How?
How many custom wire formats I need?
What parameters should I put on inputQ-node?
How do I develop this monster Mapping node?
B. Create many msg sets?
How many custom wire formats I need?
What parameters should I put on inputQ-node?
How do I develop this monster Mapping node?
C. Other ideas?
Thanks,
Moshe |
|
| Back to top |
|
|
| kimbert |
| Posted: Thu Sep 25, 2008 1:39 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| I can have same data/information comes into flow as xml or swift or fix or internal-format and then need to output sometimes as xml or as swift or as fix or as other-internal-format |
I bet the various input formats contain *slightly* different information. A 'canonical format' would hide those differences from the message flow logic, and allow it to deal with one logical structure.
| Quote: |
| So what should I do |
I can only give suggestions, but I don't mind saying what I would do, so that others can shoot my suggestions down
- Define a canonical format which can represent the information from any input source
- For each input source, design a message set with exactly one physical format. Each message set can have many message definitions, of course. And those definitions can be scattered across as many message definition files as you like.
- For each input source, create mappings or ESQL subroutines which map the source message into the canonical format
- For each destination, design a message set with exactly one physical format. Some of these message sets might be the same as the ones which represent the input sources
- For each destination, create mappings or ESQL subroutines to map from the canonical format to the output format.
I'm not sure that I would use the Mapping node for this. SWIFT is a huge message set, and you will probably find that ESQL is an easier option. And the code will be more efficient, as long as you use references wherever possible. |
|
| Back to top |
|
|
| scravr |
| Posted: Thu Sep 25, 2008 6:28 am Post subject: |
|
|
Partisan
Joined: 03 Apr 2003
Posts: 391
Location: NY NY USA 10021
|
I am not sure what do you mean by "canonical format" ?
I run a POC on it:
1. Created msg-set-A with XML dictionary presentation format.
2. Created msg-set-B with MRM dictionary presentation firmat of same structure like 1.
3. Created flow FL1 with InputQ validating msg-set-A as input and mapping it to msg-set-B as output.
4. Created flow FL2 with InputQ validating msg-set-B as input and mapping it to msg-set-B as output (just map it to itself as a test).
Both FL1 and FL2 are working OK.
Now I like to have FL3 that can take as input either msg-set-A or msg-set-B and map them to msg-set-B.
1. How can I use msg-set-A.mxsd, msg-set-B.mxsd, and thier dictionaries in InputQ? What parameters should be on InputQ? Should I validate the input? If I validate msg-set-A, then msg-set-B always be error !!!
2. Can I write 1 (and only 1) mapping to map from msg-set-A or msg-set-B (depend on some type) into msg-set-B?
3. In a mapping (or java/compute nodes) How do I know what msg-type comes in?
Thanks,
Mos |
|
| Back to top |
|
|
| kimbert |
| Posted: Thu Sep 25, 2008 2:47 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| I am not sure what do you mean by "canonical format" ? |
It is a logical structure which can represent any of the inputs to your system. It's a good technique for scenarios like this, because it means that the message flow logic can operate on the canonical format without caring about which type of input it came from. You may claim that there is no message flow logic - the message flow is only doing format conversion. That might be true now, but once you have a broker which can parse and write all the messages in your business...
| Quote: |
| How can I use msg-set-A.mxsd, msg-set-B.mxsd, and thier dictionaries in InputQ? What parameters should be on InputQ? |
So you want messages with different physical formats to arrive on the same queue? Then you need to add an MQRFH2 header to your input messages. Check the docs for details.
| Quote: |
| Should I validate the input? If I validate msg-set-A, then msg-set-B always be error !!! |
It depends what you are 'validating'. If ( as you claim ) the logical structure of A and B is identical, then you will not get a validation error - MRM validation operates on the logical message tree, not the physical bitstream.
On the other hand, if there are small differences between the logical structure of A and B, then you should be thinking about an intermediate logical structure ( canonical format ), and do your validation after you have mapped into it.
| Quote: |
| Can I write 1 (and only 1) mapping to map from msg-set-A or msg-set-B (depend on some type) into msg-set-B? |
If message set A and message set B produce identical trees, it should just work. What would you want it to depend on?
| Quote: |
| In a mapping (or java/compute nodes) How do I know what msg-type comes in? |
Look at InputRoot.Properties.MessageType |
|
| Back to top |
|
|
| scravr |
| Posted: Fri Sep 26, 2008 5:02 am Post subject: |
|
|
Partisan
Joined: 03 Apr 2003
Posts: 391
Location: NY NY USA 10021
|
1. The logical structure of A and B is NOT identical. These are diff formats: swift, fix …..
2. Well, if I have 50 SWIFT, 50 FIX, and 50 internal-formats: do I need to develop 150 flows just to map into canonical-format-A. And do I need 150 Q’s? !!! (Then you are right, mapping from canonical-format-A to output will be simple.)
Is there a way to group all 50 SWIFT into one flow, all 50 FIX to 2nf flow, etc.? I guess it will be very difficult to create and maintain XSD presentations.
3. Also can I add attributes to elements on canonical-format-A so I can figure out what was source msg-type, msg-id, …. ? |
|
| Back to top |
|
|
| fschofer |
| Posted: Mon Sep 29, 2008 12:01 am Post subject: |
|
|
Knight
Joined: 02 Jul 2001
Posts: 524
Location: Mainz, Germany
|
Hi.
| Quote: |
| 1. How can I use msg-set-A.mxsd, msg-set-B.mxsd, and thier dictionaries in InputQ? What parameters should be on InputQ? Should I validate the input? If I validate msg-set-A, then msg-set-B always be error !!! |
You could read in your messages as BLOB, determine if its A or B by using some header or by looking at the message content and then use the CREATE PARSE statement to parse and validate with a matching message set.
Greetings
Frank |
|
| Back to top |
|
|
| scravr |
| Posted: Mon Sep 29, 2008 7:27 am Post subject: |
|
|
Partisan
Joined: 03 Apr 2003
Posts: 391
Location: NY NY USA 10021
|
Yes; BLOB can do it. Its just enable the msg to pass thru input-node and defer the parshing.
Did anyone try mapping elements/fields from one of many input sources into singletarget output?
Lets say I pass the input-node, then I like to map what ever comes in (MRM msg-type1, MRM msg-type2,...) into output MRM msg-type3.
Basically, I like to use mapping node tp map fields from any-input-msg to output-msg.
I am getting this red X error on the mapping node spreadsheet saying: "This map is not callable..." and I canot gener bar file.
I guess mapping dose not support many-to-1 !!! |
|
| Back to top |
|
|
| scravr |
| Posted: Tue Sep 30, 2008 12:17 pm Post subject: |
|
|
Partisan
Joined: 03 Apr 2003
Posts: 391
Location: NY NY USA 10021
|
To work with MRM and diff msg lenght and string fields:
I created a choice complex element on the message definition (message level) and pass to MQ input (broker 6.1) as MRM parser with the msg-set, type, and format parameters.
Then in flow, I will find out the msgtype by length, fields-value....
Looks like MRM cannont parse it:
(0x01000021):MRM = (
UNRESOLVED CHOICE = X'303230303230303034'
)
here is the definition for CWF:
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns="http://www.w3.org/2001/XMLSchema" xmlns:msdata="urn:schemas-microsoft-com:xml-msdata" xmlns:mstns="http://www.w3.org/2001/XMLSchema" attributeFormDefault="unqualified" elementFormDefault="unqualified" id="EnterpriseDDocument">
<xs:element msdata:IsDataSet="true" msdata:Prefix="e" name="MsgLvl">
<xs:annotation>
<xs:appinfo source="WMQI_APPINFO">
<MRMessage messageDefinition="/0/MsgLvl;XSDElementDeclaration$MRObject"/>
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:choice maxOccurs="unbounded">
<xs:element form="unqualified" msdata:Prefix="e" name="HeaderDoc">
<xs:complexType>
<xs:sequence>
<xs:element form="unqualified" minOccurs="0" msdata:Prefix="e" name="MsgID" type="string">
<xs:annotation>
<xs:appinfo source="WMQI_APPINFO">
<cwfInclRep messageSetDefaultRep="DOH_MRM_FRMT">
<cwfSimpleRep encodingNull="NullLogicalValue" mrLengthEncoding="fixedLength" paddingCharacter="NUL" stringJustification="leftJustify" typeName="MRCWFStringRep" width="2"/>
</cwfInclRep>
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element maxOccurs="1" minOccurs="1" msdata:Prefix="e" name="Header">
<xs:complexType>
<xs:sequence>
<xs:element form="unqualified" minOccurs="1" name="HDR1">
<xs:annotation>
<xs:appinfo source="WMQI_APPINFO">
<cwfInclRep messageSetDefaultRep="DOH_MRM_FRMT">
<cwfSimpleRep encodingNull="NullLogicalValue" mrLengthEncoding="fixedLength" paddingCharacter="NUL" stringJustification="leftJustify" typeName="MRCWFStringRep" width="3"/>
</cwfInclRep>
</xs:appinfo>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="string">
<xs:length value="3"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element form="unqualified" minOccurs="1" name="HDR2">
<xs:annotation>
<xs:appinfo source="WMQI_APPINFO">
<cwfInclRep messageSetDefaultRep="DOH_MRM_FRMT">
<cwfSimpleRep encodingNull="NullLogicalValue" mrLengthEncoding="fixedLength" paddingCharacter="NUL" stringJustification="leftJustify" typeName="MRCWFStringRep" width="4"/>
</cwfInclRep>
</xs:appinfo>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="string">
<xs:length value="4"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element form="unqualified" msdata:Prefix="e" name="TrailerDoc">
<xs:complexType>
<xs:sequence>
<xs:element form="unqualified" name="Trailer">
<xs:complexType>
<xs:sequence>
<xs:element form="unqualified" minOccurs="1" name="TRLR1">
<xs:annotation>
<xs:appinfo source="WMQI_APPINFO">
<cwfInclRep messageSetDefaultRep="DOH_MRM_FRMT">
<cwfSimpleRep encodingNull="NullLogicalValue" mrLengthEncoding="fixedLength" paddingCharacter="NUL" stringJustification="leftJustify" typeName="MRCWFStringRep" width="5"/>
</cwfInclRep>
</xs:appinfo>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="string">
<xs:length value="5"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:choice>
</xs:complexType>
</xs:element>
</xs:schema> |
|
| Back to top |
|
|
| kimbert |
| Posted: Tue Sep 30, 2008 1:00 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
If I were in your position, I would want the sender to identify the type of message. That's why I suggested adding an RFH2 header to your messages.
Any reason why you cannot do that?
Implementing this as a choice is not a good design - your message definition will get more and more complex as the number of branches in the choice increases. If the message format is CWF, you should be telling message broker what the message type is, not attempting to deduce it from the incoming message. |
|
| Back to top |
|
|
|
|
