| Author |
Message
|
| achocks |
 Posted: Tue Jun 19, 2012 7:58 pm Post subject: ESQL Help - Extracting the Binary Values before logging |
 |
|
Voyager
Joined: 28 Nov 2011
Posts: 82
|
WMB experts..Need your help!
I am trying to route a 15MB message through the WMB 7.0. It is a very simple XML
The problem I am getting in to is, when trying to log in to the database. Since the XML message is having encoded Base64 Binary data , the size is huge. My requirement is also not to save the Binary Data. So I decided to remove the binary data alone from the XML message and then log it to the database.
The parser I am using is XMLNSC.
Here is the skeleton message. Note the <Attachment Data> tags. The input XML message has several <Attachment Data>tags like this. My goal is to remove all the <attachment Data> node before logging.
Any help would be greatly appreciated.
-------------------------------------------------------------
| Code: |
<?xml version="1.0"?>
<soap:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Header>... </soap:Header>
<soap:Body>
<sn:submitNewApplication xmlns:sn="http://company.com/xyzoperation">
<txl:TXLife xmlns:txl="http://ACORD.org/Standards/Life/2" xmlns:ext="http://company.com/ACORD/ext">
<txl:TXLifeRequest>
<txl:TransRefGUID>12232323</txl:TransRefGUID>
<txl:TransType tc="103">New Business </txl:TransType>
<txl:OLifE>
<txl:Holding id="Policy_11">
<txl:Attachment id="NBAP">
<txl:AttachmentBasicType tc="2">Image</txl:AttachmentBasicType>
<txl:AttachmentData>HUGE BINARY ATTACHMENT DATA(1MB)</txl:AttachmentData>
<txl:AttachmentType tc="5">Form</txl:AttachmentType>
<txl:MimeTypeTC tc="11">image/tiff</txl:MimeTypeTC>
<txl:TransferEncodingTypeTC tc="4">Base64</txl:TransferEncodingTypeTC>
<txl:ImageType tc="3">TIFF</txl:ImageType>
<txl:ImageSubmissionType tc="5">Document can be recovered from data present</txl:ImageSubmissionType>
</txl:Attachment>
<txl:Attachment id="INFO">
<txl:AttachmentBasicType tc="2">Image</txl:AttachmentBasicType>
<txl:AttachmentData>HUGE ATTACHMENT DATA BINARY(1MB)</txl:AttachmentData>
<txl:AttachmentType tc="5">Form</txl:AttachmentType>
<txl:MimeTypeTC tc="11">image/tiff</txl:MimeTypeTC>
<txl:TransferEncodingTypeTC tc="4">Base64</txl:TransferEncodingTypeTC>
<txl:ImageType tc="3">TIFF</txl:ImageType>
<txl:ImageSubmissionType tc="5">Document can be recovered from data present</txl:ImageSubmissionType>
</txl:Attachment>
</txl:Holding>
<txl:Holding id="Banking">
<txl:HoldingTypeCode tc="7">Banking</txl:HoldingTypeCode>
<txl:Attachment id="BANK">
<txl:AttachmentBasicType tc="2">Image</txl:AttachmentBasicType>
<txl:AttachmentData>HUGE BINARY ATTACHMENT DATA (1MB)</txl:AttachmentData>
<txl:AttachmentType tc="5">Form</txl:AttachmentType>
<txl:MimeTypeTC tc="11">image/tiff</txl:MimeTypeTC>
<txl:TransferEncodingTypeTC tc="4">Base64</txl:TransferEncodingTypeTC>
<txl:ImageType tc="3">TIFF</txl:ImageType>
<txl:ImageSubmissionType tc="5">Document can be recovered from data present</txl:ImageSubmissionType>
</txl:Attachment>
</txl:Holding>
</txl:OLifE>
</txl:TXLifeRequest>
</txl:TXLife>
</sn:submitNewApplication>
</soap:Body>
</soap:Envelope> |
|
|
| Back to top |
|
 |
| marko.pitkanen |
| Posted: Tue Jun 19, 2012 10:20 pm Post subject: |
|
|
Chevalier
Joined: 23 Jul 2008
Posts: 440
Location: Jamsa, Finland
|
Hi achocks,
What tricks have you already tried? Have you already tried walking through your message tree and deleting all elements where fieldname is "AttachmentData", before storing the XMLNSC to database?
--
Marko |
|
| Back to top |
|
|
| kash3338 |
| Posted: Tue Jun 19, 2012 10:50 pm Post subject: |
|
|
Shaman
Joined: 08 Feb 2009
Posts: 709
Location: Chennai, India
|
| SET AttachmentData to NULL |
|
| Back to top |
|
|
| Esa |
| Posted: Tue Jun 19, 2012 11:13 pm Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| kash3338 wrote: |
| SET AttachmentData to NULL |
Sorry, kash3338, but no. This is about large messages. From performance point of view it would be better to use DELETE FIELD. DELETE FIELD will release the memory for reuse immediately. If you set the value to NULL, the memory will be released only when the flow instance terminates.
So, even if setting to NULL works, it is certainly less elegant and not the best practice.
Or, it depends on the requirements. If you need to have the best broker overall performance or the highest throughput for a lot of messages, you should use DELETE FIELD. But if you need to have the shortest possible processing time for individual not so frequently occurring messages, you might get better results byt setting the fields to NULL. Allocating and releasing large memory blocks is relatively expensive and takes some time. So if the memory of elements that are set to NULL is released after the MQ operation is committed, the output message might be available a little earlier. That the memory may be released only after the MQ operation has been committed is my personall guess that needs to be confirmed by somebody who knows better. |
|
| Back to top |
|
|
| Esa |
| Posted: Tue Jun 19, 2012 11:47 pm Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
Here I go again. Killing time while waiting for a meeting to start...
Although, in the OP's case where you will have to copy the input message to OutputRoot anyway, you get the best performance if you do not SET InputRoot to OutputRoot and then remove the attachments but write your own copy procedure that traverses the input message and just skips the attachments while building the output message.
And even better if you follow the principles of Message Broker Large Messaging pattern when parsing the input message. But, of course,only if you really are going to trash the binary attachments and not to do anything with them elsewhere in the flow. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Wed Jun 20, 2012 2:06 am Post subject: Re: ESQL Help - Extracting the Binary Values before logging |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| achocks wrote: |
| Here is the skeleton message. |
Please use [code] tags when posting messages, to make them more readable.
| kash3338 wrote: |
| SET AttachmentData to NULL |
... inside a FOR statement, or some other type of LOOP.
| Esa wrote: |
| DELETE FIELD will release the memory for reuse immediately. |
Ok, that's what the documentation says. But it's a bit odd that assigning NULL to a field wouldn't also release memory for immediate reuse. A common message found in most any usertrace would at least seem to imply otherwise:
| Code: |
| BIP2567: Node <insert_2>: Assigning NULL to <insert_1>, thus deleting it. |
It's especially odd, since that's what the DETACH statement is for: to remove a sub-tree, without actually deleting it. (Do detached sub-trees get deleted when the transaction terminates, or do they just leak memory?)
Generally speaking, the cost for allocating and releasing memory depends on the number of elements to allocate/release, not their size. So, it's not that expensive to release memory for a single element, particularly as this usually involves just marking it as reusable, rather than actually de-allocating it. Releasing memory for a sub-tree with a large number of elements could get expensive, but no more expensive than creating the sub-tree in the first place. Does assigning NULL to a sub-tree also require disconnecting any reference variables pointing into the sub-tree? If so, then there is no compelling reason to delay releasing the memory for it. |
|
| Back to top |
|
|
| Esa |
| Posted: Wed Jun 20, 2012 2:50 am Post subject: Re: ESQL Help - Extracting the Binary Values before logging |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| rekarm01 wrote: |
It's especially odd, since that's what the DETACH statement is for: to remove a sub-tree, without actually deleting it. (Do detached sub-trees get deleted when the transaction terminates, or do they just leak memory?)
|
Yes, the memory allocated by a detached subtree is released when the flow instance terminates.
| The InfoCenter wrote: |
DELETE statement
The DELETE statement detaches and destroys a portion of a message tree, allowing its memory to be reused. This statement is particularly useful when handling very large messages. |
One could draw the conclusion that setting a subtree to NULL perhaps just detaches it, leaving the memory to be released when the flow instance terminates.
| rekarm01 wrote: |
| Generally speaking, the cost for allocating and releasing memory depends on the number of elements to allocate/release, not their size. So, it's not that expensive to release memory for a single element, particularly as this usually involves just marking it as reusable, rather than actually de-allocating it. Releasing memory for a sub-tree with a large number of elements could get expensive, but no more expensive than creating the sub-tree in the first place. |
You have a point here. What makes allocating memory for a large message tree expensive is not only the large amount of memory to be allocated but the large amount of allocation operations to be done - one for each node in the tree.
But I have made experiments with BLOB messages of size 50 - 80 MB and I would say that allocating such an amount of memory even for a single node - the BLOB element - takes a lot of time. But I may have misinterpreted my results.
| rekarm01 wrote: |
| Does assigning NULL to a sub-tree also require disconnecting any reference variables pointing into the sub-tree? If so, then there is no compelling reason to delay releasing the memory for it. |
No, it does not, because the tree is still there, you just do not have a reference to it any more.
On the other hand a reference pointing to an element of a DELETEd subtree should not be possible to use for accessing the element any more. But will it point to the root element of the tree from where it was detached or to an arbitrary memory location - I cannot say yet. The documentation on the other hand says that a reference cannot be null.
These two cases should be easy enough to test. Who will be the first one to report the results? Not me, unfortunately, at least not today. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jun 20, 2012 5:45 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Quote: |
| The documentation on the other hand says that a reference cannot be null |
What does the documentation say about a reference pointing to a null element? The reference per se is not null, the element it is pointing to is...
I guess looking at the previous post that this is where the subtle distinction between a null element and a detached element appears. You do keep a valid reference to a detached element, (have never tried accessing its children though) but cannot move a reference to the children of a null'd field... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| kash3338 |
| Posted: Wed Jun 20, 2012 6:17 am Post subject: Re: ESQL Help - Extracting the Binary Values before logging |
|
|
Shaman
Joined: 08 Feb 2009
Posts: 709
Location: Chennai, India
|
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jun 20, 2012 6:37 am Post subject: Re: ESQL Help - Extracting the Binary Values before logging |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
AHH... but that was not assigning a null value to the field...
This is how you assign a null value to a field:
| Code: |
| SET OutputRoot.XMLNSC.Msg.Data.Name VALUE = NULL; |
I guess you knew that... just clarifying for newbies...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| kash3338 |
| Posted: Wed Jun 20, 2012 6:48 am Post subject: Re: ESQL Help - Extracting the Binary Values before logging |
|
|
Shaman
Joined: 08 Feb 2009
Posts: 709
Location: Chennai, India
|
| fjb_saper wrote: |
AHH... but that was not assigning a null value to the field...
This is how you assign a null value to a field:
| Code: |
| SET OutputRoot.XMLNSC.Msg.Data.Name VALUE = NULL; |
I guess you knew that... just clarifying for newbies... |
For the OP's query, we dont have to assign a NULL value, we need to set NULL to the field. That's what i mentioned above.
The above quote was for the discussion on the memory allocation when we set to NULL and when we Delete. |
|
| Back to top |
|
|
| Esa |
| Posted: Thu Jun 21, 2012 1:53 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
First I want to apologize achocs for hijacking his/her thread. I hope you have got enough pointers to go on with your flow. Copying the message and setting all attachments (or their values) to null should work. What we are discussing here is about millisecond level differences in performace.
| fjb_saper wrote: |
| Quote: |
| The documentation on the other hand says that a reference cannot be null |
What does the documentation say about a reference pointing to a null element? The reference per se is not null, the element it is pointing to is...
I guess looking at the previous post that this is where the subtle distinction between a null element and a detached element appears. You do keep a valid reference to a detached element, (have never tried accessing its children though) but cannot move a reference to the children of a null'd field... |
Yes, you cannot move a reference to a child of a null'd field. If you want to test what happens, you must move the reference before you null the parent.
I made a test (Message Broker V8 on 64-bin Win7, but it should not make any difference).
The test message:
| Code: |
<Message>
<Order>
<Part>
<Number>123</Number>
<Code>P123</Code>
<Name>Part P123</Name>
</Part>
<Customer>
<Name>John</Name>
</Customer>
</Order>
</Message> |
The ESQL:
| Code: |
CREATE COMPUTE MODULE nulltest_Compute
CREATE FUNCTION Main() RETURNS BOOLEAN
BEGIN
SET OutputRoot = InputRoot;
DECLARE partRef REFERENCE TO OutputRoot.XMLNSC."Message".Order.Part;
SET OutputRoot.XMLNSC.Message.Order = NULL;
PROPAGATE DELETE NONE;
CREATE LASTCHILD OF OutputRoot.XMLNSC.Message.Order FROM partRef;
RETURN TRUE;
END;
END MODULE; |
Output message 1:
Output message 2:
| Code: |
<Message>
<Order>
<Part>
<Number>123</Number>
<Code>P123</Code>
<Name>Part P123</Name>
</Part>
</Order>
</Message> |
When I changed the ESQL to PROPAGATE with DELETE DEFAULT, I got an exception from this line:
| Code: |
| CREATE LASTCHILD OF OutputRoot.XMLNSC.Message.Order FROM partRef; |
The exception:
| Code: |
RecoverableException
File:CHARACTER:F:\build\S000_P\src\DataFlowEngine\ImbParser.cpp
Line:INTEGER:2386
Function:CHARACTER:ImbSyntaxElement::copy
Type:CHARACTER:
Name:CHARACTER:
Label:CHARACTER:
Catalog:CHARACTER:BIPmsgs
Severity:INTEGER:2
Number:INTEGER:2335
Text:CHARACTER:User attempted a recursive copy
Insert
Type:INTEGER:15
Text:CHARACTER:baedbd0
Insert
Type:INTEGER:5
Text:CHARACTER:(0x00000000:0x00000000):
Insert
Type:INTEGER:15
Text:CHARACTER:b997c70
Insert
Type:INTEGER:5
Text:CHARACTER:(0x01000000:Name):Root |
Propagating with the default setting releases the XMLNSC parser and thus all the memory that has been allocated on its behalf. Now the subtree really has been deleted and the reference partRef points to nowhere. In the debugger it has no content. But when you try to access it in the ESQL code, it will be set to Root, and you get an exception of trying to issue a recursive copy. |
|
| Back to top |
|
|
| achocks |
| Posted: Thu Jun 21, 2012 11:10 am Post subject: ESQL Help - Extracting the Binary Values before logging |
|
|
Voyager
Joined: 28 Nov 2011
Posts: 82
|
All,
Thanks to every one for the wonderful insights on the DELETE vs NULL and the impact on memory. I feel like I am learning a lot from you guys.
I was trying to find the number of occurances of the 'Attachment' in the message and then planning to loop through and DELETE it before storing it to the database.
I am not sure what I am missing.
Here is the ESQL statement I am using to find the number of attachment field.
| Code: |
DECLARE CountF1 INT CARDINALITY(OutputRoot.XMLNSC.Envelope.Body.submitNewApplication.TXLife.TXLifeRequest.OLifE.Holding.Attachment[]);
|
But the above is always returning zero, even though there is two attachment.
Can some one tell me what I am doing wrong?
Thanks in advance! |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Jun 21, 2012 11:55 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
what is it returning if you use this
| Code: |
| CARDINALITY(myRef.Holding.Attachment[>]); |
?
You do not want to know how many fields below Holding = Holding.*[]
You want to know how many fields below Holding with name of Attachment...
That would be the number of the last attachment field under holding...
from memory Holding.Attachment[>] (meaning look it up and verify...)
But anyways I'd say this information is irrelevant.
Go to the first field by ref and then move the next ref to the next sibling with same name type. Delete the first ref, set and jump next...
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| kimbert |
| Posted: Thu Jun 21, 2012 2:21 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
There is more than one way to find all the elements with name 'Attachment' in the message tree. Iterating over the entire tree using loops is not necessarily the easiest way.
You could use a SELECT statement ( I think ).
You could use a JavaCompute node, and get the matching elements using the XPath expression '$InputRoot//Attachment'.
Once you have that list of matching nodes, you can write a single loop to delete them or set them to NULL or detach them. |
|
| Back to top |
|
|
|
|
