| Author |
Message
|
| cwazpitt3 |
 Posted: Fri Mar 02, 2012 4:28 am Post subject: XMLNSC Parser Max Depth |
 |
|
Acolyte
Joined: 31 Aug 2011
Posts: 61
|
| I was wondering if there is a limit to how deep an XML document is allowed to be when using the XMLNSC parser. I have an XML output that is only 200k, however, it is extremely nested and is causing me issues. Basically, whenever I propagate to an output node (have tried HttpReply and FileOutput so far), the flow just hangs until the timout is hit (this is a REST service) and it never sends the output. Even more disturbing, I have no idea where the message goes...and no errors that I can find. Any ideas? |
|
| Back to top |
|
 |
| kimbert |
| Posted: Fri Mar 02, 2012 4:40 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
All software has limits. There are no documented limits to nesting depth in XMLNSC, but there will be a limit somewhere.
200K is not a large document, so it's no the absolute size that is causing the problem. Have you tried doing a binary chop on the nesting depth to see whether that is the issue? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Mar 02, 2012 4:45 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
If you want to isolate the problem to the serialization of the message tree, instead of some other issue with the output nodes you're using, then you should either use ASBITSTREAM in an isolated compute node or a ResetContentDescriptor node to set the domain to BLOB.
Then take a user trace and see where the code is spending all of it's time.
If that doesn't tell you anything useful, take a service trace and then open a PMR. |
|
| Back to top |
|
|
| cwazpitt3 |
| Posted: Fri Mar 02, 2012 4:48 am Post subject: |
|
|
Acolyte
Joined: 31 Aug 2011
Posts: 61
|
| kimbert wrote: |
All software has limits. There are no documented limits to nesting depth in XMLNSC, but there will be a limit somewhere.
200K is not a large document, so it's no the absolute size that is causing the problem. Have you tried doing a binary chop on the nesting depth to see whether that is the issue? |
What do you mean by "binary chop on the nesting depth"? |
|
| Back to top |
|
|
| cwazpitt3 |
| Posted: Fri Mar 02, 2012 4:51 am Post subject: |
|
|
Acolyte
Joined: 31 Aug 2011
Posts: 61
|
| mqjeff wrote: |
If you want to isolate the problem to the serialization of the message tree, instead of some other issue with the output nodes you're using, then you should either use ASBITSTREAM in an isolated compute node or a ResetContentDescriptor node to set the domain to BLOB.
Then take a user trace and see where the code is spending all of it's time.
If that doesn't tell you anything useful, take a service trace and then open a PMR. |
I was able to get a file trace of the message to actually create, it was 66 MB in size and not real helpful as you can imagine. My next try was going to be the RCD node to BLOB to see if it writes out to the file. Since this really is a web service, can I write BLOB to an HttpReply node? That's a pretty dumb question I guess, but I think I lose validation, right? |
|
| Back to top |
|
|
| mqsiuser |
| Posted: Fri Mar 02, 2012 5:03 am Post subject: Re: XMLNSC Parser Max Depth |
|
|
Yatiri
Joined: 15 Apr 2008
Posts: 637
Location: Germany
|
| cwazpitt3 wrote: |
| I was wondering if there is a limit to how deep an XML document is allowed to be when using the XMLNSC parser. |
A long time ago someone (probably) thought there should be a limit on the (function-)call stack (which made me problems when implementing quicksort). The limit is somewhere between 100 and 1000 (on the call stack!). The situation is (much) better on the depth of nested xml-documents. Though... it is astonishing if you need significant depths (more than 100) on standard use. May you share with us if it is more than 100 and if so why ?
If it's less than 100 you probably have not really problems on XMLNSC, but just with validating (probably that can get you into trouble there (with validating against xsd), do you really need to validate during runtime?).
_________________
Just use REFERENCEs
Last edited by mqsiuser on Fri Mar 02, 2012 5:06 am; edited 1 time in total |
|
| Back to top |
|
|
| cwazpitt3 |
| Posted: Fri Mar 02, 2012 5:05 am Post subject: Re: XMLNSC Parser Max Depth |
|
|
Acolyte
Joined: 31 Aug 2011
Posts: 61
|
| mqsiuser wrote: |
| cwazpitt3 wrote: |
| I was wondering if there is a limit to how deep an XML document is allowed to be when using the XMLNSC parser. |
A long time ago someone (probably) thought there should be a limit on the (function-)call stack (which made me problems when implementing quicksort). The limit is somewhere between 100 and 1000 (on the call stack!). The situation is (much) better on the depth of nested xml-documents. Though... its astonishing if you need significant depths (more than 100) on standard use. May you share with us if it is more than 100 and if so why ?
If it's less than 100 you probably have not really problems on XMLNSC, but just with validating (probably that can get you into trouble, do you really need to validate during runtime?). |
I would say it is at least 100, probably more. I don't disagree that the design is flawed, but I just wondered if there was some limit I was up against or something I was doing wrong (aside from design) that is causing the issues. I just like to find out these things because other wise, they bug me  |
|
| Back to top |
|
|
| mqsiuser |
| Posted: Fri Mar 02, 2012 5:10 am Post subject: Re: XMLNSC Parser Max Depth |
|
|
Yatiri
Joined: 15 Apr 2008
Posts: 637
Location: Germany
|
| cwazpitt3 wrote: |
| I just like to find out these things because other wise, they bug me |
o.k, I guess then kimbert has answered your question. Aside from the "memory/resources available" I have the strong feeling that there is no like "parameter" that limits the depth. If anyone finds out different or wants to correct me, pls do so.
I am jumping in here because I have also dealt with this during quicksort implementation and guess likely other people do not come across this that often (>100)... probably you should really check and question what you are doing there
_________________
Just use REFERENCEs |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Mar 02, 2012 5:50 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| cwazpitt3 wrote: |
| mqjeff wrote: |
Then take a user trace and see where the code is spending all of it's time.
If that doesn't tell you anything useful, take a service trace and then open a PMR. |
I was able to get a file trace of the message to actually create, it was 66 MB in size and not real helpful as you can imagine. |
A file trace is not remotely the same thing as a user trace. A file trace shows you the structure of the message. A user trace shows you what the message flow is doing, and where it is spending all of it's time.
| cwazpitt3 wrote: |
| My next try was going to be the RCD node to BLOB to see if it writes out to the file. Since this really is a web service, can I write BLOB to an HttpReply node? That's a pretty dumb question I guess, but I think I lose validation, right? |
You can send any kind of message content to an HTTPReply node. Just like you can send any kind of message content to any other output node.
Message validation is performed when a bitstream is parsed into a logical message tree, and when a logical message tree is serialized into a bitstream. So taking efforts to manually serialize the message tree in a separate step from sending the serialized bitstream to the endpoint does not change the fact that validation has occurred. |
|
| Back to top |
|
|
| mqsiuser |
| Posted: Sun Mar 04, 2012 3:37 am Post subject: |
|
|
Yatiri
Joined: 15 Apr 2008
Posts: 637
Location: Germany
|
| mqjeff wrote: |
| A file trace is not remotely the same thing as a user trace. A file trace shows you the structure of the message. A user trace shows you what the message flow is doing, and where it is spending all of it's time. |
As an extra (what you can do) adding to taking the 2 mentioned traces here: If you are processing very large messages (e.g. quite some MB), any trace will likely become very big. Just consider doing the following: Detach the debugger (ofc) and put in the message (you can do that e.g. on a Broker-runtime on you laptop). If your flow is coded properly processing will usually take at most some seconds. Look at what happens and carefully think about what is going on and change and try out a couple of things with your code / message. If the exec-group has 100% CPU, kill it, then remove the backed out message and proceed with your tests (tipp: you need to change something  ).
In your case you would run some trials with increasing message nesting. You will quickly find out about the things your are investigating. Also: To test exactly what you are investigating (e.g. in your case the nesting-depth of xml-elements) remove anything else that is not of interest. Use MQInput-and MQOutput-Nodes probably (for testing (in your case the xml-message-depth)).
Now I gave you all these cool tipps, what are you going to do now !?
Since you brought it up here, it would be great if you did so and let us know (though I am not really expecting it from you ... and I am also not really expecting a limit on the depth of xml documents)... So your result that I expect should be: No problem with the depth, but something else was causing issue(s) (you could tell us about this "something else" though).
And I must actually add  , that quicksort does "just/only" create highly nested "logical" trees (within memory), which are not parsed in or out. So that doesn't really help here !
| mqjeff wrote: |
| Message validation is performed when a bitstream is parsed into a logical message tree, and when a logical message tree is serialized into a bitstream. So taking efforts to manually serialize the message tree in a separate step from sending the serialized bitstream to the endpoint does not change the fact that validation has occurred. |
Note that there is schemaless (no xsd) "validation" (the parser will complain!) (at least for xml, just "well formedness") and schema validation, which is always against a schema (xsd). I was referring to the xsd-variant. So: I could assume that just/only having (xsd-)schema(s) in your (run-time) flow might cause problems on large messages... remove the schema  ... you can just put xml-input messages into your flow without needing to deploy a (xsd-)schema and/or validate against (the xml schema). Note that just using it (xsd) might already motivate the parser to do some stuff with it (validation).
As a general tipp: Try to remove/reduce your flow until it works then you know (and then you can put the o.k. things back in )
_________________
Just use REFERENCEs |
|
| Back to top |
|
|
| cwazpitt3 |
| Posted: Mon Mar 05, 2012 5:16 am Post subject: |
|
|
Acolyte
Joined: 31 Aug 2011
Posts: 61
|
| I just wanted to provide an update to this issue for anyone who might be following along. There clearly is a limit, like kimbert said, to how deep the XML can go. Any output node I used would just hang when handed this gigantic message. I never spent the time to figure out what that limit was because we found the source of the issue and it had nothing to do with Broker. We are using Oracle's XML features to build the XML string and hand it off to Broker and inside the stored procedure creating the XML was a bug that caused it to grow horizontally instead of vertically. It is now working completely as it should with no issues. Thanks to all who provided insight to help me figure this out! |
|
| Back to top |
|
|
|
|
