| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Execution group shutdown if output message is too large |
« View previous topic :: View next topic » |
| Author |
Message
|
| ksilpras |
 Posted: Thu Sep 07, 2006 8:23 pm Post subject: Execution group shutdown if output message is too large |
 |
|
Newbie
Joined: 29 Jul 2006
Posts: 9
|
Hi All,
I am a newbie here. I have been using WebSphere Message Broker for a while. Currently I am working on EAI project and facing a problem with my message flow when its output message during creating within a Compute node is getting bigger.
The size of the input message is approximately 30 MB (XML) and the output message is to reconstruct of the input message to another format (XML). So I expect that the output message should have the same size as the input one. During message reconstruction within a Compute node in my message flow, instead of the output message at the output queue. The message is rolled back to the dead letter queue. Once I check the Abend file specified in the Event viewer. I found that it said "Failed to allocated memory".
It seems that the cost of buffering the output message before putting it into the output queue is so big that the execution group can not handle such big message and shut itself down.
Also I have followed the guideline in ESQL reference below for handling large XML message. However this is a guideline to handle large Input XML message not the large output XML message. Any idea?
| Quote: |
Manipulating a large message tree can demand a lot of storage. If you design a message flow that handles large messages made up of repeating structures, you can code ESQL statements that help to reduce the storage load on the broker. These statements support both random and sequential access to the message, but assume that you do not need access to the whole message at one time. These ESQL statements cause the broker to perform limited parsing of the message, and to keep only that part of the message tree that reflects a single record in storage at a time. If your processing requires you to retain information from record to record (for example, to calculate a total price from a repeating structure of items in an order), you can either declare, initialize, and maintain ESQL variables, or you can save values in another part of the message tree, for example LocalEnvironment. This technique reduces the memory used by the broker to that needed to hold the full input and output bit streams, plus that needed for just one record’s trees, and provides memory savings when even a small number of repeats is encountered in the message. The broker uses partial parsing and the ability to parse specified parts of the message tree to and from the corresponding part of the bit stream. To use these techniques in your Compute node, apply these general techniques:
1 Copy the body of the input message as a bit stream to a special folder in the output message. This creates a modifiable copy of the input message that is not parsed and that therefore uses a minimum amount of memory.
2 Avoid any inspection of the input message. This avoids the need to parse the message.
3 Use a loop and a reference variable to step through the message one record at a time. For each record:
– Use normal transforms to build a corresponding output subtree in a second special folder.
– Use the ASBITSTREAM function to generate a bit stream for the output subtree that is stored in a BitStream element placed in the position in the tree that corresponds to its required position in the final bit stream.
– Use the DELETE statement to delete both the current input and output record message trees when you have completed their manipulation.
– When you have completed the processing of all records, detach the special folders so that they do not appear in the output bit stream.
|
Many thanks in advance for any comment.
Ken |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Fri Sep 08, 2006 2:24 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
You need to increase the EG's heap size, maybe.
There are instructions for doing this... somewhere.... maybe in the Info Center?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| cottonmouth |
| Posted: Fri Sep 08, 2006 3:10 am Post subject: |
|
|
Newbie
Joined: 07 Sep 2006
Posts: 5
|
I withdraw my last comment.. Should have tread the full story and not jumped to conclusions too fast - me bad.. 
However the broker needs up to 20 (hearsay) times the memory of the incoming message to be able to parse it fully. To avoid this use the Environment.var at all times to store the payload, this limits the allocation to 2-3 times the initial message size..
Once you've put the incoming payload in the environment.var only pass headers laong among compute nodes before reassembling all pieces at the end.
_________________
Ours is not to reason why,
ours is just to do or die. |
|
| Back to top |
|
|
| ksilpras |
| Posted: Sun Sep 10, 2006 6:56 pm Post subject: |
|
|
Newbie
Joined: 29 Jul 2006
Posts: 9
|
Hi cottonmouth, jefflowrey
Thank you for your comments. Increasing heap memory is another way to protect the execution group to shut down itself. However, i also have tried to keep the payload in the Environment.var. For some reason, It seems like the Environment doesn't preserve the structure of the original XML message, it then converts those XML attributes into elements of the variable. May be another solution is to do create a variable under the Environment.var with XML domain.
For example,
CREATE LASTCHILD OF Environment.var DOMAIN('XML') NAME 'PayLoad';
Thank you so much for sharing
Ken |
|
| Back to top |
|
|
| kimbert |
| Posted: Mon Sep 11, 2006 12:19 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| It seems like the Environment doesn't preserve the structure of the original XML message, it then converts those XML attributes into elements of the variable |
As you say, Environment can preserve your attributes, but you have to create the environment tree using the DOMAIN clause on the CREATE LASTCHILD statement.
See here for example: http://www.mqseries.net/phpBB2/viewtopic.php?t=31250 |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
