| Author |
Message
|
| newtobroker |
 Posted: Fri Feb 12, 2010 11:08 pm Post subject: Sequential processing of XML with a dynamic delete |
 |
|
Novice
Joined: 04 Feb 2010
Posts: 23
|
Hi,
I have strange requirement like this
I'm processing an XML sequentially in esql starting from the node Test which has a namespace. I'm reading each tag and im processing it one by one. Whenever i hit an <exit> tag i go out to a different flow and then when the flow returns back, i start from the position after the exit tag again.
So the problem now is that if my xml is of huge size then when i return next time, i keep reading till i find exit tag but dont process it as they were already processed last time.
As in ESQL we dont have feature of XPATH, i dont have a way to directly point to the <exit> when i enter next time.
Now one solution that has been proposed is to have a <delete> tag in the XML which will be like a logical ending of steps. So whenever i find a <delete> tag in the XML, the esql should delete all the tags before that in the XML, in that way when i <exit> and return later my XML will contain only tags after the <delete> tag.
For e.g.
Actual XML
| Code: |
<A>
<B>
<C>....................
...............
..........
<delete>
<K>
<L>
<M>
<exit>
................
................
<X>
<Y>
<Z>
|
So now when i return after exit it should be like
| Code: |
<K>
<L>
<M>
<exit>
................
................
<X>
<Y>
<Z>
|
How can we write such a procedure in esql that can help me delete all the tags before the <delete> tag? Im using XMLNSC.
Thanks,
C* |
|
| Back to top |
|
 |
| mgk |
| Posted: Sat Feb 13, 2010 4:14 am Post subject: |
|
|
Padawan
Joined: 31 Jul 2003
Posts: 1642
|
| Quote: |
| As in ESQL we dont have feature of XPATH, i dont have a way to directly point to the <exit> when i enter next time. |
Actually you do, in ESQL it is a REFERENCE.
Regards,
_________________
MGK
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. |
|
| Back to top |
|
|
| newtobroker |
| Posted: Sat Feb 13, 2010 10:19 am Post subject: |
|
|
Novice
Joined: 04 Feb 2010
Posts: 23
|
mgk,
But we can use reference provided we are in the same thread. If i terminate my thread then i lose my reference.
Lets say i have 20 tags in XML and i processed till 12th tag and then exit processing after setting an attribute "processed=true" on the 12th tag.
Now next time when i start my processing i shud directly start from 13th, if i had xpath i could have given something like '.../processed=true' to directly locate the 12th element but we cannot do that in esql.
I have to then read tag by tag till i reach 13th and then start processing.
Thanks,
C* |
|
| Back to top |
|
|
| kimbert |
| Posted: Sat Feb 13, 2010 1:21 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
I suspect that your problem is a lot simpler than you are making it sound.
| Quote: |
| Lets say i have 20 tags in XML and i processed till 12th tag... |
- What's special about the 12th tag?
- Why can't you process one tag at a time?
| Quote: |
| and then exit processing after setting an attribute "processed=true" on the 12th tag. |
- That implies that the entire message has been parsed, it is now stored within the flow as a huge message tree, and you are adding an attribute into it. That sounds like a good way to get an out of memory condition.
| Quote: |
| Now next time when i start my processing i shud directly start from 13th |
There is no XML technology ( not even XPath ) that will make that an efficient approach. At best, every 'next time' the XML parser will need to walk to the Nth occurrence of the tag. Or the tag on which you have stuck the 'processed=true' attribute. Either way, that's going to be very expensive in CPU.
| Quote: |
to directly locate the 12th element but we cannot do that in esql.
|
As explained above. 'directly' is the wrong word. It may look simple in the XPath syntax, but its a very inefficient algorithm.
In fact, you can do something quite similar in ESQL by using on-demand parsing and deleting each record after it has been processed. I suggest that you read this: http://www.ibm.com/developerworks/websphere/library/techarticles/0505_storey/0505_storey.html |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Feb 13, 2010 8:36 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
I believe what he is trying to apply to a single message is a restart / recovery logic. SAP does that when loading IDOCs from a file.
So this means that he would try to load a huge xml and abort / retry processing of said message.
AFAIK the broker is not geared for that. The simple way would be to have an outside program loading the xml by atomic messages and keeping track of where to restart...
Hope it helps 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| kimbert |
| Posted: Mon Feb 15, 2010 10:54 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
I believe what he is trying to apply to a single message is a restart / recovery logic. SAP does that when loading IDOCs from a file.
So this means that he would try to load a huge xml and abort / retry processing of said message. |
That's possible - but newtobroker has not said so yet. I suspect that there's a reasonably good broker-based solution, once we understand the requirements. |
|
| Back to top |
|
|
|
|
