| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Append elements to a XML document |
« View previous topic :: View next topic » |
| Author |
Message
|
| Cogito-Ergo-Sum |
 Posted: Fri Jan 27, 2012 9:34 pm Post subject: Append elements to a XML document |
 |
|
Master
Joined: 07 Feb 2006
Posts: 293
Location: Bengaluru, India
|
How do I append elements to a XML document so that it grows from an empty file ?
Requirement
We have the following requirement. In a message flow, we have to generate an XML document in a file (say, use.xml) somewhat like this.
| Code: |
<?xml version="1.0" encoding="utf-8"?><root><a><b>Data 1</b><c>Data 2</c></a></root> |
For every incoming message to our message flow, we have to keep appending to this XML document the <a> tag as the last child of <root>. Initially, use.xml would be empty. Throughout the day, use.xml would keep growing as <a> tag keeps getting appended. The number of <a> tags inserted is expected to be few hundreds. Then, at the end of the day, this XML document would be deleted and a fresh one is created next day. In a given day, the whole of XML document should be made available (we are thinking, manual FTP - for now) to the downstream application.
Option 1
In the message flow, add a FileRead node for the use.xml file. Domain set to XMLNSC. Then, in a Compute node, check if the file is empty. (Hope this could be done by one of the fields in the LocalEnvironment). If yes, then build the file from <root> tag. If not, create <a> tag as last child of <root>.
Option 2
For every incoming message, insert a record into a DB2 table and then use a DatabaseInput node to react to this insertion. I am hoping to use DB2 XML support to generate the document using all the records inserted 'today'.
Question I have
In both the options I mention above, the premise is, I should already have an instance of the whole document before I append. That is, in the first option, MB would have to parse the whole of the document first. In the second, the query would have to run over all inserted records. So, in both cases, wouldn't the performance be expected to degrade over the time ?
Can you please let me know if there is another option ?
Could there be a way where the DOM structure of the XML document is maintained by the broker so that as soon as the message flow begins, the nodes in the flow could get a handle to it directly ? This will not be easy especially if the XML document has lots of updates happening.
_________________
ALL opinions are welcome.
-----------------------------
Debugging tip:When you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth.
---Sherlock Holmes |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Sat Jan 28, 2012 6:03 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You can't use FileOutput node to append to an existing file that is then available to another application.
FileOutput will only append to a file that it is keeping private.
In fact, the entire design you are discussing is extremely questionable. The downstream application should be modified to accept each individual change, rather than a continually increasing set of records.
Regardless, it's easy to write the code to do this "append" in Broker. You just get a reference to the last child, and then add a new record. |
|
| Back to top |
|
|
| kimbert |
| Posted: Sat Jan 28, 2012 12:49 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| t's easy to write the code to do this "append" in Broker. You just get a reference to the last child, and then add a new record. |
Easy, but horribly inefficient for large data sets. In order to append each new <a> record the message flow will have to
1. Parse the entire XML document to produce a message tree
2. Append one more element 'a' in the message tree
3. Convert the entire message tree into a new, longer XML document
My solution would be
- Maintain a growing file that only contains the <a> records
- For each message, create a new <a> record using ASBITSTREAM and append it to a file. Since this is now a true append ( there is no </root> to worry about ) the BLOB domain can be used. The append can be done using ESQL string manipulation functions because BLOB is a string type in ESQL.
- at the end of the day, before forwarding the file to the receiving application, put the XML declaration and the '<root>' at the beginning and put the '</root>' at the end ( again using BLOB domain ).
Note: this solution is a hack, and I don't like it. I agree completely with mqjeff - the best solution is to make the receiving application read from a queue instead of a file. |
|
| Back to top |
|
|
| adubya |
| Posted: Sat Jan 28, 2012 1:21 pm Post subject: |
|
|
Partisan
Joined: 25 Aug 2011
Posts: 377
Location: GU12, UK
|
| kimbert wrote: |
My solution would be
- Maintain a growing file that only contains the <a> records
- For each message, create a new <a> record using ASBITSTREAM and append it to a file. Since this is now a true append ( there is no </root> to worry about ) the BLOB domain can be used. The append can be done using ESQL string manipulation functions because BLOB is a string type in ESQL.
- at the end of the day, before forwarding the file to the receiving application, put the XML declaration and the '<root>' at the beginning and put the '</root>' at the end ( again using BLOB domain ).
. |
I read the OP's requirement differently in that during the day the file needs to be made available to the downstream app (and therefore would need to be valid XML with <root> tags in place) as it grows. And at the end of the day the file is deleted and an empty one constructed for the cycle to begin again.
It does seem to be a bizarre requirement  |
|
| Back to top |
|
|
| Cogito-Ergo-Sum |
| Posted: Sat Jan 28, 2012 7:22 pm Post subject: |
|
|
Master
Joined: 07 Feb 2006
Posts: 293
Location: Bengaluru, India
|
| adubya wrote: |
I read the OP's requirement differently in that during the day the file needs to be made available to the downstream app (and therefore would need to be valid XML with <root> tags in place) as it grows. And at the end of the day the file is deleted and an empty one constructed for the cycle to begin again.
It does seem to be a bizarre requirement |
Yes. At any point, the complete XML should be available for the downstream application. Whether we send this complete XML, via MQ or manual FTP is not the concern right now.
I am also contemplating to send the 'growing' XML via MQ after every append. That is, first message as <root> being empty. Then, second message with first <a> tag. The third message with one more (total 2) <a> tags and so on .
As for the, 'bizarre requirement' - not my call, not my design. 
| kimbert wrote: |
Easy, but horribly inefficient for large data sets. In order to append each new <a> record the message flow will have to
1. Parse the entire XML document to produce a message tree
2. Append one more element 'a' in the message tree
3. Convert the entire message tree into a new, longer XML document |
That is exactly what I concluded too.
_________________
ALL opinions are welcome.
-----------------------------
Debugging tip:When you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth.
---Sherlock Holmes |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sat Jan 28, 2012 11:42 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| Cogito-Ergo-Sum wrote: |
Yes. At any point, the complete XML should be available for the downstream application. Whether we send this complete XML, via MQ or manual FTP is not the concern right now.
|
I think you are wrong here.
Just think about what is going to happen if the downstream application wants to read the data At exactly the same moment it is being updated.
If you are updating a file then something has to give. The file will be locked. So what is the downstream app going to do then? Is it going to take a break, and have a cup of coffee/tea and come back later....
Sorting out the deliver of this incremental data should be your prime concern.
Frankly, this design leads an awful lot to be desired. I have grave doubts about its reliability in a production envirnment.
As an aside,
some of these 'requirements' we have been getting here for the last few months really show a lack of serious thought about the overall end-to-end operation. I've been working in System Integration for more than 30 years. I've not seen as many frankly crackpot/silly/dumb/crazy 'requirement's in all that time as I have seen here in the past year or so. Roll on retirement.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| adubya |
| Posted: Sun Jan 29, 2012 12:55 am Post subject: |
|
|
Partisan
Joined: 25 Aug 2011
Posts: 377
Location: GU12, UK
|
| Quote: |
| As for the, 'bizarre requirement' - not my call, not my design. |
Understood, but presumably you've raised concerns to those who made the call ? Architects/designers don't always get things right  |
|
| Back to top |
|
|
| kimbert |
| Posted: Mon Jan 30, 2012 2:21 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| I've not seen as many frankly crackpot/silly/dumb/crazy 'requirement's in all that time as I have seen here in the past year or so. Roll on retirement. |
Are you sure that you're not looking at the past with rose-tinted spectacles. Old age can do that, you know 
"Nostalgia's not what it used to be" |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Jan 30, 2012 2:53 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| kimbert wrote: |
| Quote: |
| I've not seen as many frankly crackpot/silly/dumb/crazy 'requirement's in all that time as I have seen here in the past year or so. Roll on retirement. |
Are you sure that you're not looking at the past with rose-tinted spectacles. Old age can do that, you know
"Nostalgia's not what it used to be" |
Ok. I get it with the old age stuff. I'm a 
However in my defence, I could go and dig through all my notebooks dating from say 1981/82 and there is no doubt in my mind about what I said is true. I spent 20 years working for a computer manufacturer in a group where a lot of the jobs we got to do were just that, slightly crazy, not in the pricebook. It was often the case of 'Give it to the guys in *****. They'll know what to do with it.'. In many cases, we did just that, made it work.
In all that time, there was just one case where we took one look at what was proposed and flatly refused to get involved.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Jan 30, 2012 5:14 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| smdavies99 wrote: |
| kimbert wrote: |
| Quote: |
| I've not seen as many frankly crackpot/silly/dumb/crazy 'requirement's in all that time as I have seen here in the past year or so. Roll on retirement. |
Are you sure that you're not looking at the past with rose-tinted spectacles. Old age can do that, you know
"Nostalgia's not what it used to be" |
Ok. I get it with the old age stuff. I'm a |
He's just trying to get used to seeing the grey hair in the mirror himself. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
