| Author |
Message
|
| bremen |
 Posted: Thu Dec 28, 2006 3:44 pm Post subject: Large XML Message handling technique(URGENT!!!!!!!!!!!!) |
 |
|
Novice
Joined: 28 Dec 2006
Posts: 16
Location: Hindustan
|
Hi all,
I am using broker V5.0 and File extender(Fixpack3).
I am processing F2M(FileInput node-->Computenode-->MQ output node) by taking 10MB i/p file and propagating MQ messages as o/p(~10MB size) using large message technique provided in IBM site .The flow is using around 200MB memory.Can any one help me to reduce the Memory usage(expecting 50MB).
My Code is here:
CREATE COMPUTE MODULE FLOW_F2Q_Compute
CREATE FUNCTION Main() RETURNS BOOLEAN
BEGIN
CALL CopyMessageHeaders();
CALL CopyEntireMessage();
RETURN FALSE;
END;
CREATE PROCEDURE CopyMessageHeaders() BEGIN
DECLARE I INTEGER 1;
DECLARE J INTEGER CARDINALITY(InputRoot.*[]);
WHILE I < J DO
SET OutputRoot.*[I] = InputRoot.*[I];
SET I = I + 1;
END WHILE;
END;
CREATE PROCEDURE CopyEntireMessage() BEGIN
CREATE LASTCHILD OF Environment.Variables DOMAIN('XML') PARSE(InputRoot."BLOB"."BLOB" OPTIONS FolderBitStream);
DECLARE InMessageCopy REFERENCE TO Environment.Variables.OuterMessage;
DECLARE InputMessage REFERENCE TO InMessageCopy;
MOVE InputMessage FIRSTCHILD NAME 'OuterMessage';
-- no of Inner messages
DECLARE Inner_No INTEGER 0;
SET Environment.Variables.TotalInnerMsgs = CARDINALITY(InputMessage.InnerMessage[]);
-- Move onto the first Inner Message
MOVE InputMessage FIRSTCHILD NAME 'InnerMessage';
WHILE LASTMOVE(InputMessage)
DO
CASE FIELDNAME(InputMessage)
WHEN 'InnerMessage' THEN
IF Inner_No>=1 THEN DELETE PREVIOUSSIBLING OF InputMessage;END IF;
SET Inner_No = Inner_No + 1;
CREATE NEXTSIBLING OF OutputRoot.Properties domain 'MQMD';
SET OutputRoot.Properties.ReplyIdentifier = MQCI_NONE;
IF Inner_No = 1 THEN
SET OutputRoot.MQMD.GroupId = InputRoot.MQMD.MsgId;
SET OutputRoot.MQMD.MsgSeqNumber = 1;
SET OutputRoot.MQMD.MsgFlags = MQMF_MSG_IN_GROUP;
ELSEIF
Inner_No= Environment.Variables.TotalInnerMsgs THEN
SET OutputRoot.MQMD.GroupId = InputRoot.MQMD.MsgId;
SET OutputRoot.MQMD.MsgSeqNumber = Environment.Variables.TotalInnerMsgs;
SET OutputRoot.MQMD.MsgFlags = MQMF_LAST_MSG_IN_GROUP;
ELSE
SET OutputRoot.MQMD.GroupId = InputRoot.MQMD.MsgId;
SET OutputRoot.MQMD.MsgSeqNumber = Inner_No;
SET OutputRoot.MQMD.MsgFlags = MQMF_MSG_IN_GROUP;
END IF;
SET OutputRoot.MQMD.Version = MQMD_CURRENT_VERSION;
SET OutputRoot.MQMD.StrucID = MQMD_STRUC_ID;
SET OutputRoot.MQMD.Expiry = -1;
SET OutputRoot.MQMD.Format = 'MQSTR';
SET OutputRoot.MQMD.CodedCharSetId = 500;
SET OutputRoot.MQMD.Encoding=785;
SET OutputRoot.Properties.MessageSet = 'C4PR10O002001';
SET OutputRoot.Properties.MessageDomain = 'MRM';
SET OutputRoot.Properties.MessageFormat = 'CWF1';
SET OutputRoot.Properties.MessageType = 'msg_MbfeMsg';
SET OutputRoot.MRM.MBFETEST.Data1 = InputMessage.MessageBody[1];
SET OutputRoot.MRM.MBFETEST.Data2 = InputMessage.MessageBody[2];
PROPAGATE;
END CASE;
MOVE InputMessage NEXTSIBLING;
END WHILE;
END MODULE;
i/p file structure (<Msg><MBFETest><Data1>hi</Data1><Data2>howru</Data2></MBFETest><MBFETest><Data1>hey</Data1><Data2>ho</Data2></MBFETest><MBFETest><Data1>hi</Data1><Data2></Data2></MBFETest></Msg>)  |
|
| Back to top |
|
 |
| pathipati |
| Posted: Thu Dec 28, 2006 7:10 pm Post subject: |
|
|
Master
Joined: 03 Mar 2006
Posts: 296
|
| Are you sure that 200MB is taken by Broker runtime and not by Toolkit? I don't see any suspicious code in your ESQL. |
|
| Back to top |
|
|
| bremen |
| Posted: Thu Dec 28, 2006 7:52 pm Post subject: Large XML Message handling technique(URGENT!!!!!!!!!!!!) |
|
|
Novice
Joined: 28 Dec 2006
Posts: 16
Location: Hindustan
|
Patipati,
I am talking about Broker runtime memory only.Its taking ~200 MB for 10MB i/p file and ~10MB total o/p msgs.
Thanks,
Rajesh. |
|
| Back to top |
|
|
| Marek |
| Posted: Fri Dec 29, 2006 3:57 am Post subject: |
|
|
Apprentice
Joined: 30 Jun 2004
Posts: 32
Location: Edinburgh
|
I've also had to deal with large messages (10-20MB XML).
I'm not surprised that the example you sent is using 200MB of Broker memory as this equates to approx. 50MB for the Execution Group and approx. 15 x 10MB for the message itself. NB. I don't know why it's 15 times the message size but it's always therabouts for all message sizes.
You're method for dealing with memory usage i.e. deleting the data as you go is excellent and there is nothing wrong with the code.
However, if you wish to reduce memory usage and increase throughput from a design perspective I would recommend that you split the flow into 2 flows. The first should segment the single message by propagating batches of e.g. 250 records in a single message and the second flow should then process and propagate the messages as single records. An MQ group concept can be used to keep them as one unit of work if required. Then consider allowing this flow to run in an Execution Group by itself.
Other methods, but not applicable in this particular example (as you alter the message) are to process messages in Database nodes (don't copy trees from input to output) and to process as a BLOB (where the data does not need referenced) as there is no parser overhead.
Hope this helps. You are not alone (!) |
|
| Back to top |
|
|
| bremen |
| Posted: Fri Dec 29, 2006 1:37 pm Post subject: |
|
|
Novice
Joined: 28 Dec 2006
Posts: 16
Location: Hindustan
|
Marek,
our script is monitoring on execution group not on broker. i think 2 flows is better design perspective.
One more question ??
for example the i/p file data is
<Msg><MBFETest><Data1>hi</Data1><Data2>howru</Data2></MBFETest><MBFETest><Data1>hey</Data1><Data2>ho</Data2></MBFETest><MBFETest><Data1>hi</Data1><Data2></Data2></MBFETest></Msg>
Is it possible to process the file record by record.If you consider <MBFETest>---------</MBFETest> as one record can we use "custom end of record" as </MBFETest> |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri Dec 29, 2006 5:33 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
You should re-read the documentation for the MBFE nodes.
I'm pretty sure it can read your files "record for record", and avoid you having to do anything fancy with your flows involving groups or etc.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Marek |
| Posted: Sat Dec 30, 2006 7:11 am Post subject: |
|
|
Apprentice
Joined: 30 Jun 2004
Posts: 32
Location: Edinburgh
|
For clarification, my post also refers to memory usage per Execution group ( ... on the Broker).
The input message is XML with repeating records labelled as <MBFETest>. Therefore, I would have an XML MQInput/Compute/MQOutput node configuration. I don't understand why you would process an XML message in a File Input node? I'd like to understand the reason. Please clarify.
Assuming the message is already split into batches/segments of records the code in the Compute Node would now look something similar to your original posting:
| Code: |
CREATE LASTCHILD OF Environment.Variables DOMAIN 'XML' NAME 'InputMessage';
SET Environment.Variables.InputMessage = InputBody;
DECLARE MyPointer REFERENCE TO Environment.Variables.InputMessage.Msg.MBFETest[1];
-- Loop through the message contents
WHILE LASTMOVE(MyPointer) = TRUE
DO
-- ***PROCESS DATA HERE***
DECLARE LastProcessedRow REFERENCE TO MyPointer;
MOVE MyPointer TO Environment.Variables.InputMessage.Msg.MBFETest[2];
DELETE FIELD LastProcessedRow;
PROPAGATE;
END WHILE;
RETURN FALSE; |
|
|
| Back to top |
|
|
| bremen |
| Posted: Sat Dec 30, 2006 10:18 am Post subject: |
|
|
Novice
Joined: 28 Dec 2006
Posts: 16
Location: Hindustan
|
Marek,
As per the requirements the i/p file(data in file is in XML format) is coming from external system. Thats why we r using File i/pnode-->compute-->MQ o/p.Probably we dont need to do Segmentation because record is having size less than 1MB. |
|
| Back to top |
|
|
| mvarghese |
| Posted: Sat Dec 30, 2006 8:39 pm Post subject: |
|
|
Centurion
Joined: 27 Sep 2006
Posts: 141
|
Since code cannot do any thing here and also message size is high..
rest ways i seen is use 64 bit Execution group and use Trusted Broker principle..or even try to use 2 execution grp....to balance ur load...
I am not sure this will help..u can have a try..
_________________
Jain Varghese |
|
| Back to top |
|
|
| pathipati |
| Posted: Sat Dec 30, 2006 9:37 pm Post subject: |
|
|
Master
Joined: 03 Mar 2006
Posts: 296
|
| Quote: |
| use 64 bit Execution group |
64 bit Execution group is not supported by all OS.
| Quote: |
| or even try to use 2 execution grp |
2nd execution group may take another 200 MB. |
|
| Back to top |
|
|
| JosephGramig |
| Posted: Sun Dec 31, 2006 6:20 am Post subject: |
|
|
Grand Master
Joined: 09 Feb 2006
Posts: 1244
Location: Gold Coast of Florida, USA
|
Trusted Brokers are not supported on all platforms either.
Sounds like you are breaking the file into smaller messages.
Sounds like jefflowrey is indicating to you that the MBFE node will read smaller amounts of the file for you and reduce the memory usage.
It is always a good idea to read the documentation, again, after you have worked with a component. You may find that it was updated or that you can understand more of it with your new insight.
_________________
Joseph
Administrator - IBM WebSphere MQ (WMQ) V6.0, IBM WebSphere Message Broker (WMB) V6.1 & V6.0
Solution Designer - WMQ V6.0
Solution Developer - WMB V6.1 & V6.0, WMQ V5.3 |
|
| Back to top |
|
|
| AndyT |
| Posted: Mon Jan 15, 2007 6:42 pm Post subject: |
|
|
Novice
Joined: 05 Nov 2003
Posts: 17
Location: Work
|
I think that using the file extender to break the XML data into smaller chunks will cause additional problems. I admit that I have not used the option of splitting based on custom end of record characters but as the data is XML wouldn't the chunks now be invalid XML and thus require more complex coding to process?
I suspect that alot of the memory consumption is associated with the MBFE reading the large file into its working directory and then feeding it to the broker. Could you write a custom java node (or stand alone java appln) to process the large file using a SAX parser and split it into smaller pieces that way? |
|
| Back to top |
|
|
|
|
