| Author |
Message
|
| angka |
 Posted: Wed Jul 27, 2011 8:56 pm Post subject: How to calculate how much memory a WTX map will consume? |
 |
|
Chevalier
Joined: 20 Sep 2005
Posts: 406
|
Hi,
I am using wtx integration server 8.3.0.4 on WMB 7.0.0.2 running on windows 2008 R2.
I encountered some issue when passing in big messages to the map. I am running 10 instances of the flows. each message passed in is 10mb.. when i did and load test, all 10 instances is activated and the memory of the dataflowengine get beyond 1gb and then the dataflowengine restarted.
The input to the map is 10mb (piped delimiter) and output 20mb (fixed length). I did some calculation, 10mb + 20mb + 10mb(overhead as workspace (mem)) = 40mb per instance. therefore, 10 instances = 400mb.
however, somehow when i monitored the dataflowengine, it is using more then 1gb of mem when processing the messages. it will restart when it hits around 1.2gb.
Anyone can help??
Thanks |
|
| Back to top |
|
 |
| j.f.sorge |
| Posted: Wed Jul 27, 2011 9:35 pm Post subject: |
|
|
Master
Joined: 27 Feb 2008
Posts: 218
|
If broker creates abend files I would raise a PMR.
_________________
IBM Certified Solution Designer - WebSphere MQ V6.0
IBM Certified Solution Developer - WebSphere Message Broker V6.0
IBM Certified Solution Developer - WebSphere Message Broker V6.1
IBM Certified Solution Developer - WebSphere Message Broker V7.0 |
|
| Back to top |
|
|
| kimbert |
| Posted: Thu Jul 28, 2011 3:59 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| If your map is a simple one then you could replace the WTX node with a FileInput node + MRM TDS parser + Mapping node. Then you could pass one record at a time into the message flow, and memory usage/scalability would not be an issue any more. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Jul 28, 2011 4:26 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
Kimbert is right. WTX is a legacy product and wont be around much longer (in its current incantation) anyway. If you are writing new code, take the route he suggested.
Also, don't begin your performance tests with ten instances. Start small and work upwards.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jul 28, 2011 4:39 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| lancelotlinc wrote: |
| WTX is a legacy product and wont be around much longer (in its current incantation) anyway. |
Is this actual news with a link or in the same vein as "everyone will be on Power7 in 5 years"?
If the first case it would be interesting. If the second case (that everyone will migrate off WTX in a short period of time) it's something that could easily happen.
Just trying to distinguish fact from opinon for the benefit of future readers.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Jul 28, 2011 4:59 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| Vitor wrote: |
| lancelotlinc wrote: |
| WTX is a legacy product and wont be around much longer (in its current incantation) anyway. |
Is this actual news with a link or in the same vein as "everyone will be on Power7 in 5 years"?
If the first case it would be interesting. If the second case (that everyone will migrate off WTX in a short period of time) it's something that could easily happen.
Just trying to distinguish fact from opinon for the benefit of future readers. |
Power7 ....!! 
Karl, the Product Manager for IBM B2B Translation and Transformation solutions offers that his three products are not scheduled to be sunset.
The smoke of the ugly rumor keeps rising by many different customers who see the same performance issues as our OP for the current implementation of WTX. Even when a client of mine, an infamous credit card network (which is supposedly one of the core use cases for WTX), discovered performance implications with using WTX maps, through an official PMR, the response was, abandon WTX and use MRM instead.
Whether or not the product is withdrawn from the market, the defacto withdrawal happens whenever a customer tries to use WTX and figures out its faster with standard Broker components, and easier to maintain. Another gripe is that its not possible to track all the WTX source code in a source management system since some of the WTX source code is not in text format and therefore makes it difficult to merge WTX source code changes from hotfix back into the mainstream.
So the name of WTX may keep marching on, but the product needs a rewrite to address crucial shortfalls, or it will, by the lack of users, be virtually sunset.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jul 28, 2011 5:13 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
I thank you for the context and clarification.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| angka |
| Posted: Thu Jul 28, 2011 6:33 pm Post subject: |
|
|
Chevalier
Joined: 20 Sep 2005
Posts: 406
|
Hi,
The transformation is simple, just transforming from Piped delimited to fixed length. However, the customer bought tx already so we can't use MRM parser.
And we started multiple instances were because we need to meet sla.
Thanks. |
|
| Back to top |
|
|
| kimbert |
| Posted: Fri Jul 29, 2011 2:00 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
In that case, change the map so that it operates on a single record ( single line ) at a time, instead of operating on the entire file. Then use FileInput with the 'Delimited' option to split up the file.
It probably won't perform as well as the native parser, but at least you won't blow up memory, and you should be able to run as many instances of the flow as you need to meet the SLA. |
|
| Back to top |
|
|
| joebuckeye |
| Posted: Fri Jul 29, 2011 5:33 am Post subject: |
|
|
Partisan
Joined: 24 Aug 2007
Posts: 364
Location: Columbus, OH
|
| angka wrote: |
| And we started multiple instances were because we need to meet sla. Thanks. |
Why do you think increasing instances will automatically increase the performance of your flow?
If there are certain resources being used by the map those may not be sharable and could cause the multiple instances to have to wait until the shared resource has been released.
And your memory calculation is quite flawed as you are only considering the size of your message. The DataFlowEngine itself requires some of that memory for itself and any other flows to work.
See how much memory is being used by your flows before you do any processing to get a baseline for how much memory you have to work with. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Fri Jul 29, 2011 5:36 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| angka wrote: |
Hi,
The transformation is simple, just transforming from Piped delimited to fixed length. However, the customer bought tx already so we can't use MRM parser.
And we started multiple instances were because we need to meet sla.
Thanks. |
Hahahahahahahahaha. I bought a Boeing 747 already, does that mean I cannot drive my car?
Seriously, choose the best tool for the job.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Jul 29, 2011 5:43 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| lancelotlinc wrote: |
| Seriously, choose the best tool for the job. |
Whilst I do in the most part agree here, I fear the OP is in the unenviable position of the client having bought WTX (possibly because there was an article in a glossy management magazine) and they're grimly determined to get value for their investment by using it.
Browbeating some poor guy is always a more desireable management option than sitting round a table saying "wow, that decision sucked".
Sad yet true.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
