| Author |
Message
|
| anild |
 Posted: Mon Apr 27, 2009 10:42 pm Post subject: Dataflow Engine using High memory |
 |
|
Novice
Joined: 19 Sep 2007
Posts: 13
|
| Retrieving Data from database and constructing xml messages 100 at time assigning them to Environment Variable. Memory is getting accumulated even after clearing environment variables by setting it null. resulting in high memory usage and execution group hug. we are processing close to 900000 records. |
|
| Back to top |
|
 |
| elvis_gn |
| Posted: Mon Apr 27, 2009 11:18 pm Post subject: |
|
|
Padawan
Joined: 08 Oct 2004
Posts: 1905
Location: Dubai
|
Hi anild,
What else can you expect....your design is flawed...Why are you processing 900000 records ? Are you migrating the data using WMB ?
Regards. |
|
| Back to top |
|
|
| anild |
| Posted: Mon Apr 27, 2009 11:42 pm Post subject: |
|
|
Novice
Joined: 19 Sep 2007
Posts: 13
|
We are receive Max 900000 records in day end we need to process to another system. this is the requirement.
| elvis_gn wrote: |
Hi anild,
What else can you expect....your design is flawed...Why are you processing 900000 records ? Are you migrating the data using WMB ?
Regards. |
|
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Apr 27, 2009 11:59 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| anild wrote: |
| We are receive Max 900000 records in day end we need to process to another system. this is the requirement. |
And you're meeting this requirement with a flawed design. There are any number of better ways of doing this.
As to why the memory is not being recovered, there are a number of potential explainations of this. You don't mention platform or WMB version, so maybe applying some maintenance will help.
But a better design will help more.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| anild |
| Posted: Tue Apr 28, 2009 12:23 am Post subject: |
|
|
Novice
Joined: 19 Sep 2007
Posts: 13
|
We are using Message Broker Version 6.1
| Vitor wrote: |
| anild wrote: |
| We are receive Max 900000 records in day end we need to process to another system. this is the requirement. |
And you're meeting this requirement with a flawed design. There are any number of better ways of doing this.
As to why the memory is not being recovered, there are a number of potential explainations of this. You don't mention platform or WMB version, so maybe applying some maintenance will help.
But a better design will help more. |
|
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Apr 28, 2009 12:33 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| anild wrote: |
We are using Message Broker Version 6.1
|
Well if you're on the latest maintenance of this latest version, then it's certainly redesign time.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| anild |
| Posted: Tue Apr 28, 2009 1:06 am Post subject: |
|
|
Novice
Joined: 19 Sep 2007
Posts: 13
|
Could you please suggest me how we can proceed.
| Vitor wrote: |
| anild wrote: |
We are using Message Broker Version 6.1
|
Well if you're on the latest maintenance of this latest version, then it's certainly redesign time. |
|
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Apr 28, 2009 1:25 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| anild wrote: |
Could you please suggest me how we can proceed.
|
You need to be thinking far more about this data and how it needs to be processed. I suspect (but only suspect) that there was an existing batch process that has been moved/migrated into WMB without serious thought. What are these 90,000 records? Why xml documents of 100 records each? How do they relate? What is their content?
Any more than that & I'll need an address to send an invoice! 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
