| Author |
Message
|
| rharikumar |
 Posted: Thu Jun 26, 2014 3:46 am Post subject: Memory Spikes - WMB |
 |
|
Novice
Joined: 14 Oct 2013
Posts: 16
|
Hello All
I have a WMB flow , which takes in a flat file as input , parses it using a DFDL parser , splits each record in the flat file as a separate xml and the drops it to a queue .
In a case , where the flat file holds more than , say 7000 records , memory consumption spikes and as its a fact that consumed memory cannot be release automatically , i end up restarting the execution group .
DFDL parser and the code are all cost effective . However a solution to this issue is difficult to approach or achieve .
Any guidance is much appreciated .
During the memory spike , pileups are observed in SYSTEM.CLUSTER.TRANSMIT |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Thu Jun 26, 2014 4:01 am Post subject: Re: Memory Spikes - WMB |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| rharikumar wrote: |
i end up restarting the execution group .
|
Why do you feel the need to do that? Internally Broker will release the RAM so that when another flow requires it, it is available and the broker does not have to grab it from a Operating System. Why do you feel the need to return it to the OS? Are your systems that short of RAM?
| Quote: |
During the memory spike , pileups are observed in SYSTEM.CLUSTER.TRANSMIT |
What do you observe in the messages on that Queue? What do you mean by pileup? To me a pile up is something that happens frequently on busy roads when many vehicles crash. Do these message clear themselves?
What really happens.
7K records in a file is not that large. There are posts here talking about more than a million records in a file. Do you read all the file at once and the parse it? Or do you read/parse record by record? The memory consumption pattern may well be different between them.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| rharikumar |
| Posted: Thu Jun 26, 2014 5:09 am Post subject: |
|
|
Novice
Joined: 14 Oct 2013
Posts: 16
|
1 . The entire file (all the records in it ) is first parsed with the DFDL message model and then is processed by esql to split each one as XML .
2. Message pileups in that queue mentioned which clears when the execution group is restarted . (Uncommitted message count )
3.The memory consumption is very specific to this execution group which holds this component . (dataflowengine memory consumption)
Heap size of the eg is 1 GB and so when this threshold is reached it is borrowing memory from OS , threatening overall service . |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Jun 26, 2014 7:25 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
What is the transactional setting on the MQOutput node? Are you sure your setting is the right one? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| kimbert |
| Posted: Thu Jun 26, 2014 9:04 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
What is the 'Records and Elements' setting on the FileInput node?
And what domain are you using for the FileInput node ( DFDL or BLOB? )
Does the problem still happen if you remove the MQOutput node?
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| Esa |
| Posted: Sat Jun 28, 2014 2:59 am Post subject: Re: Memory Spikes - WMB |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| rharikumar wrote: |
I have a WMB flow , which takes in a flat file as input , parses it using a DFDL parser , splits each record in the flat file as a separate xml and the drops it to a queue .
In a case , where the flat file holds more than , say 7000 records , memory consumption spikes and as its a fact that consumed memory cannot be release automatically , i end up restarting the execution group . |
7000 records is not much. There is something wrong with your design. Have you taken a look at the 'Large messaging' sample. Adapting it to DFDL is not difficult.
An even easier solution is to follow kimbert's advice above and let the File Input node do the splitting for you.
| rharikumar wrote: |
DFDL parser and the code are all cost effective |
Many companies buy cost effective code nowadays. Unfortunately cost effective is seldom the same as effective.
Dare to try to prove your claim by posting a sample?
| rharikumar wrote: |
During the memory spike , pileups are observed in SYSTEM.CLUSTER.TRANSMIT |
This is natural because you are putting 7000 messages within one transaction. You could avoid this by setting the transaction mode of the MQ Output node to 'No'. But doing so you would also lose the contact admin of the flow, which means that if the flow crashes (or you restart the eg), some of the records will be sent twice.
From that point of view message pileup is not dangerous, it's your friend. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sat Jun 28, 2014 3:34 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| rharikumar wrote: |
2. Message pileups in that queue mentioned which clears when the execution group is restarted . (Uncommitted message count )
|
Why do you think that having some UNCOMMITED messages is a 'Pileup'?
They are waiting for either the transaction to end. If the transaction is long running then perhaps....
1) You might like to look at the Queue Manager Property for the Max number of uncommitted messages
OR
2) re-code your flow so that it commits the messages in appropriate sized batches.
As I see it, pretty well everything is running as expected. There is something wrong in your flow that it does not commit the messages wither singly or in batches.
You really need to give proper consideration to transactionality and the implications of not committing thousands of messages. What would happen if you wrote every message with 'Transactionality = No'? Would the world come to a shuddering halt?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Esa |
| Posted: Sat Jun 28, 2014 7:23 am Post subject: Re: Memory Spikes - WMB |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| Esa wrote: |
This is natural because you are putting 7000 messages within one transaction. You could avoid this by setting the transaction mode of the MQ Output node to 'No'. But doing so you would also lose the contact admin of the flow, which means that if the flow crashes (or you restart the eg), some of the records will be sent twice.
|
... But doing so you would also lose the t-r-a-n-s-a-c-t-i-o-n-a-l-i-t-y of the flow...
Just found a new word that gets automatically replaced with 'contact admin'  |
|
| Back to top |
|
|
| rharikumar |
| Posted: Wed Jul 02, 2014 1:47 am Post subject: |
|
|
Novice
Joined: 14 Oct 2013
Posts: 16
|
Hello All
Thanks a lot for your help .
Update : With a input file of 60k records , i faced a similiar issue yesterday and the execution group housing this application started consuming 90% memory and i restarted the execution group .
@smdavies99
1. Max number of uncommitted messages in QM is set to 20k . I will take your input and test the scenario by changing it to the max limit .
2. I am not sure how i can split the messages and commit in bacthes . I will work on this .
@kimbert
1. We use MQInput Node only . Sorry if i have provided a confusion here .
@fjb_saper
The transaction mode of the mqoutputnode is set to automatic .
MQInputNode's property is set to 'YES' |
|
| Back to top |
|
|
| Esa |
| Posted: Wed Jul 02, 2014 2:09 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| rharikumar wrote: |
@kimbert
1. We use MQInput Node only . Sorry if i have provided a confusion here .
|
Are you using BLOB or DFDL parser in MQ Input node, then? |
|
| Back to top |
|
|
| rharikumar |
| Posted: Wed Jul 02, 2014 6:36 am Post subject: |
|
|
Novice
Joined: 14 Oct 2013
Posts: 16
|
| Esa wrote: |
| rharikumar wrote: |
@kimbert
1. We use MQInput Node only . Sorry if i have provided a confusion here .
|
Are you using BLOB or DFDL parser in MQ Input node, then? |
Im using DFDL parser . |
|
| Back to top |
|
|
| Esa |
| Posted: Wed Jul 02, 2014 6:53 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| And you copy InputRoot.DFDL to a temporary location to create a mutable copy of it as shown in the 'Large Messaging' sample? |
|
| Back to top |
|
|
|
|
