| Author |
Message
|
| ajayrana |
 Posted: Wed May 21, 2014 7:48 am Post subject: Performance Issue |
 |
|
Apprentice
Joined: 08 May 2014
Posts: 35
|
Hi All,
Facing issue with uploading a TXT delimited file(70 fields) with 10000 records and file size (14 mb).
Now the file ran for 2 hours and in the end i had to abort the operation.
If I process the same file with 500 records the files is getting processed in less than a minute.
My message flow is something like below with record detection as whole file:
File --> Mapping --(sorts the file)--> Mapping -->Insert/Update
Is there nay parameter or value which needs to be changed to process such big file ??
Thanks & Regards
Ajay Rana |
|
| Back to top |
|
 |
| Vitor |
| Posted: Wed May 21, 2014 8:47 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
14Mb is not a big file.
Which version of WMB/IIB?
Why are you sorting the file? Not only is it not an ideal use of the software's time (a number of dedicated sorting utilities are available) but it implies an affinity between the records that,during processing, could be causing the slowdown.
Where is the time going? What is the flow spending 2 hours doing? How have you determined that it's a problem with the flow and not (for example) the database getting slower and slower as you try to insert or update 10000 record in a single UoW?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed May 21, 2014 8:51 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
And from your previous post here if you're still using [index] anywhere (like to sort the file) then that's where to start looking for the problem.
That construction will behave exactly as you describe with even fairly modest values of index. For index = 10000 then performance will suck as you describe.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| ajayrana |
| Posted: Wed May 21, 2014 9:49 am Post subject: |
|
|
Apprentice
Joined: 08 May 2014
Posts: 35
|
Hi Vitor,
Thanks for your reply !!
I will try to answer your questions ...
Why are you sorting the file?
Sorting is required because this is transaction file and reversal of transactions are also present .. to process the transaction i should have the actual transaction first. The file has no order for that .. reversals are coming before the actual transaction for many transactions
We are using version 9 of IIB
Where is the time going? What is the flow spending 2 hours doing? How have you determined that it's a problem with the flow and not (for example) the database getting slower and slower as you try to insert or update 10000 record in a single UoW?
I was only monitoring the cpu utilization and it was well above 80%. It will help if you could tell me a way to monitor the system so that i m able to answer your questions better.
PS: I tried sorting the file without any DB insert/update and it happened in less than 30 seconds.
Thanks
Ajay Rana |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed May 21, 2014 10:07 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ajayrana wrote: |
| Sorting is required because this is transaction file and reversal of transactions are also present .. to process the transaction i should have the actual transaction first. The file has no order for that .. reversals are coming before the actual transaction for many transactions |
So there is affinity in the file; this is a poor design. I would also have thought that using an external sort utility that than custom written mapping would be more efficient and cheaper in terms of Total Cost Of Ownership.
| ajayrana wrote: |
| I was only monitoring the cpu utilization and it was well above 80%. |
And where is the CPU when you're not running this flow? 79%? Where is the database you're using running? Same box? Different box?
| ajayrana wrote: |
| It will help if you could tell me a way to monitor the system so that i m able to answer your questions better. |
IIB contains a range of tools designed to produce resource and flow statistics down to the node level. They're all documented.
| ajayrana wrote: |
| PS: I tried sorting the file without any DB insert/update and it happened in less than 30 seconds. |
Well that does rather point the finger at your database and/or your database process doesn't it?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| ajayrana |
| Posted: Thu May 22, 2014 7:35 am Post subject: |
|
|
Apprentice
Joined: 08 May 2014
Posts: 35
|
Hi Vitor,
We tried increasing the heap size in our setup and that worked for us.
Actually nobody changed the default value of 256 MB in initial setup. We have changed it to 4GB now considering this will be our pre-production setup.
Thanks for all the help !!
Thanks & Regards
Ajay Rana |
|
| Back to top |
|
|
|
|
