| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| How to extract and parse large CDATA from SOAP response ? |
« View previous topic :: View next topic » |
| Author |
Message
|
| alex919 |
 Posted: Tue Jul 17, 2012 2:08 pm Post subject: How to extract and parse large CDATA from SOAP response ? |
 |
|
Newbie
Joined: 17 Jul 2012
Posts: 7
|
Hello all.
I would like to have an advice about the following problem:
- message flow invokes an external third-party web service
- the web service is a typical data retrieval service,
it accepts a few selection parameters,
and returns a response similar to this:
| Code: |
soap-envelope
body
response
some metadata
<result>CDATA</result> |
- the CDATA contains the actual (XML-formatted) list of selected records,
(meaning it is an XML string inside XML),
and it can be very large, near a 100MB and above
- the flow needs to extract the CDATA string, parse it to XML,
then split and process the individual records (e.g. transform with a map)
I would like to know the best way to implement such flow
with minimal memory requirements (response time is much less important).
There is an IBM sample about splitting and parsing large amounts of repeating records,
which works wonders with MQ messages,
but does not help when combined with such a SOAP response message.
I have actually tried several different implementations,
but I was not able to reduce memory usage of the flow to acceptable values.
Anything I tried took memory usage of around 1GB of physical memory for a 75MB of data
(measured with broker 8 on a 64bit linux machine).
And it seems that most of that memory use does not come from parsers at all,
but from the SOAPRequest/HTTPRequest nodes internal implementation.
Is there a way to reduce this large memory usage ?
Note:
I have no control over the third party WS,
and can not change it not to work with CDATA.
Thank you,
Alex. |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Tue Jul 17, 2012 7:59 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Make sure when you parse the CData segment that you specify On Demand parsing...
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| alex919 |
| Posted: Tue Jul 17, 2012 11:18 pm Post subject: |
|
|
Newbie
Joined: 17 Jul 2012
Posts: 7
|
I believe the main focus is about extracting the CDATA.
Ideally I would like to put the CDATA contents into a separate MQ message, which could be then propagated to another flow,
and processed using on-demand parsing and ESQL code similar to the IBM large messaging splitter sample.
The problem is that I could not implement CDATA extraction with low memory usage.
Here is what I tried:
- create new MQ message from CDATA contents using CAST to BLOB in a compute node
- parse to a local variable in a compute node using CREATE FIELD... PARSE
- with the Validate setting turned 'on' or 'off'
- with either gateway mode or a specific WSDL
- with both SOAPRequest and HTTPReuest nodes
- with both 'on-demand' parsing and 'complete' parsing (on SOAP/HTTP Request nodes)
- tried to extract the CDATA with XSL transform
All of the above have approximately the same memory usage,
except the XSL transform, which failed with an OutOfMemory exception ... |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Wed Jul 18, 2012 5:22 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jul 18, 2012 8:17 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| alex919 wrote: |
I believe the main focus is about extracting the CDATA.
Ideally I would like to put the CDATA contents into a separate MQ message, which could be then propagated to another flow,
and processed using on-demand parsing and ESQL code similar to the IBM large messaging splitter sample.
The problem is that I could not implement CDATA extraction with low memory usage.
Here is what I tried:
- create new MQ message from CDATA contents using CAST to BLOB in a compute node
- parse to a local variable in a compute node using CREATE FIELD... PARSE
- with the Validate setting turned 'on' or 'off'
- with either gateway mode or a specific WSDL
- with both SOAPRequest and HTTPReuest nodes
- with both 'on-demand' parsing and 'complete' parsing (on SOAP/HTTP Request nodes)
- tried to extract the CDATA with XSL transform
All of the above have approximately the same memory usage,
except the XSL transform, which failed with an OutOfMemory exception ... |
If you want to do that, extract the CData section as a BLOB and don't parse it. Just attach it to a BLOB message and send it out...
Well there is one more thing you could try:
read the Cdata section in increments and propagate to a FileOutput node.
When done close the file.
Have a FileInput node read the file record by record, or using large message techniques...
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| alex919 |
| Posted: Thu Jul 19, 2012 2:01 am Post subject: |
|
|
Newbie
Joined: 17 Jul 2012
Posts: 7
|
| fjb_saper wrote: |
| If you want to do that, extract the CData section as a BLOB and don't parse it. |
Could you please help me with proper ESQL code for doing this ?
Thank you. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
