| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| DataFlowEngine grows costantly in memory allocated |
« View previous topic :: View next topic » |
| Author |
Message
|
| fcasali |
 Posted: Fri Jul 09, 2004 6:28 am Post subject: DataFlowEngine grows costantly in memory allocated |
 |
|
Novice
Joined: 09 Jul 2004
Posts: 12
Location: Torino, Italy
|
Hi to all.
We have deployed a WMQI 2.1 (CSD05) Broker Domain in a production environment with a quite high message rate.
Now we have a problem with execution groups (the DataFlowEngine. processes) which grow costantly in memory allocated (from 60Mb up to 900Mb and higher).
It seems that flows in those execution groups don't release correctly the memory allocated.
Because we use only standard nodes and there is no ESQL command for allocate/free memory, the question is:
- there is some known bug/fix about this problem?
- there is some ESQL best practice that can help in mantaining constant the memory usage ?
_________________
Regards,
Fabrizio Casali
IT Specialist Advisory |
|
| Back to top |
|
 |
| Michael Dag |
| Posted: Fri Jul 09, 2004 9:53 am Post subject: |
|
|
Jedi Knight
Joined: 13 Jun 2002
Posts: 2602
Location: The Netherlands (Amsterdam)
|
when not used the memory should release automagically.
If you don't see a 'release' of memory, post your code here to see if there is something strange in it.
or open a problem ticket with IBM.
_________________
Michael
MQSystems Facebook page |
|
| Back to top |
|
|
| fcasali |
| Posted: Mon Jul 12, 2004 12:16 am Post subject: |
|
|
Novice
Joined: 09 Jul 2004
Posts: 12
Location: Torino, Italy
|
Thank you for the reply.
Unfortunatly I cannot post to you the ESQL because it's a bulk of code spreaded out into flows of 15/20 nodes.
However the general schema we have follow is to calculate all the needed values in some internal nodes and save them into the Environment. Then, at the very last Compute node, we assemble the output message reading its fields from the Enviroment.
I have the suspect that the problem arises from the management of the memory bound to the Environment structure.
In short:
- what's the destiny of the elements detached from the structure? Are they deleted at the end of the message processing ?
- what about all the other values inserted in the Environment ? Are they cleared for each message or simply detached and made unreachable ?
_________________
Regards,
Fabrizio Casali
IT Specialist Advisory |
|
| Back to top |
|
|
| fcasali |
| Posted: Fri Jul 16, 2004 12:41 am Post subject: |
|
|
Novice
Joined: 09 Jul 2004
Posts: 12
Location: Torino, Italy
|
OK, we have solved the problem with the help of a guy from the ITS Techincal Support of IBM Italy.
Memory leak is due to a problem related to the APAR IY45242 ("Message throughput degrades following exception in MQOutput node") fixed with the FIXPACK 6 (CSD06). In short: when an exception raises in a MQOutput node with the fail terminal not wired, memory allocated to the message and LocalEnvironment is never reclaimed.
However I think that this is not the only problem related to the memory usage. I have seen the RAM allocated to the DataFlowEngine growing very slowly (yes, only a few Kbytes but ... why?) even connecting the fail terminals of the MQOutput nodes. I suppose that this behaviour is due to the memory allocation strategy of the Broker. This is an excerpt from the "Release Notes" of the CSD05 (I don't know if this information is present in the very last version of the ESQL Reference manual):
...
===============================
New ESQL DELETE FIELD statement
===============================
DELETE FIELD
>--DELETE--+--FIELD------------------+--FieldReference--;-->>
+--FIRSTCHILD------+--OF--+
+--LASTCHILD-------+
+--PREVIOUSIBLING--+
+--NEXTSIBLING-----+
The DELETE FIELD statement detaches and destroys a portion of a message tree allowing its memory to be reused. This statement is particularly useful when handling very large messages.
...
Handling large messages
Messages entering and leaving a broker are in a bitstream form (that is, a
string of bytes). However, internally they are handled in a tree form in
which each field is represented by a separate object (syntax element) and
the objects linked together by pointers. The broker does this to give the
transform writer rapid, random access to the message's fields. Without the
tree form, you have sequential access only to fields, which makes the writing of transforms very difficult.
However, the convenience of the tree form has a disadvantage. A tree
representation of a message is usually bigger and sometimes much bigger than its bitstream equivalent. The causes of this expansion include:
The presence of the pointers that link the objects together
Translation of character data into unicode (often a X2 expansion)
The presence of field names (which might have been implicit in the bitstream form)
The presence of control data associated with the broker's operation
For large input or output messages, this expansion causes the large memory usage by the broker and it might even reach the host system's virtual memory limit.
...
===============================
So my suggestion is the following:
when you SET or CREATE a FIELD in the Environment/LocalEnvironment remember to DELETE it (not only DETACH it ...) before the end of the flow.
This helps the Broker to better manage its memory.
I tried this approach and I had good response: at last memory is stable!
_________________
Regards,
Fabrizio Casali
IT Specialist Advisory |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
