| Author |
Message
|
| yaakovd |
 Posted: Fri Jun 27, 2008 10:55 am Post subject: MB optimization |
 |
|
Partisan
Joined: 20 Jan 2003
Posts: 319
Location: Israel
|
Hi ALL
I have message flow that transforms binary file to ASCII comma delimited. MB 6.1 on WIN XP, 2GB RAM.
8 MB contains fix length bloks of data. Each block has header, data records, trailer and filler.
1. read file using file input node
2. read bloks of data and removes fillers
3. parse record and decode fields.
4. writing output file.
I use on-demand parsing because records are of different types.
Most of fields require custom decoding (call ESQL functions).
Step 3 takes aproximatelly 4 hours for 8 MB file (44000 records).
In real system it will run on strong AIX machine, but 4 hours...
Any idea of optimization or MB configuration?
_________________
Best regards.
Yaakov
SWG, IBM Commerce, Israel |
|
| Back to top |
|
 |
| yaakovd |
| Posted: Sun Jun 29, 2008 1:01 am Post subject: |
|
|
Partisan
Joined: 20 Jan 2003
Posts: 319
Location: Israel
|
I see no one helps... 
I'm running files with different number of records:
240 - les then 1 sec - 24000 records/per sec
2400 - 7 sec - 20500 records/per sec
8000 - 2.85 min - 2800 records/per sec
16000 - 11.85 min - 1350 records/per sec
40 000 - 4-6 hours
60 000 record - low memory... It is just 8 MB file... 
_________________
Best regards.
Yaakov
SWG, IBM Commerce, Israel |
|
| Back to top |
|
|
| sami.stormrage |
| Posted: Sun Jun 29, 2008 1:19 am Post subject: |
|
|
Disciple
Joined: 25 Jun 2008
Posts: 186
Location: Bangalore/Singapore
|
MQ and MB are meant for smaller messages. If you have huge messages to transfer, try breaking them up using throtelling utility (write ur own custom code).. these things are seldom useful when u r transferring a huge database from one place to another.
_________________
*forgetting everything  * |
|
| Back to top |
|
|
| yaakovd |
| Posted: Sun Jun 29, 2008 8:15 pm Post subject: |
|
|
Partisan
Joined: 20 Jan 2003
Posts: 319
Location: Israel
|
 MB and MQ supports 100 MB files.
My system is not only transfer, and I don't have problem with transfer and simple transformation on 20 MB files.
But in complicated conversion I have resources issue with 1-2 MB file.
_________________
Best regards.
Yaakov
SWG, IBM Commerce, Israel |
|
| Back to top |
|
|
| sebastianhirt |
| Posted: Mon Jun 30, 2008 1:41 am Post subject: |
|
|
Yatiri
Joined: 07 Jun 2004
Posts: 620
Location: Germany
|
| Have you checked the performance of your ESQL code? In many cases there is some room for improvement. (e.g. proper usage of references, usage of SELECT rather then several Set commands...) |
|
| Back to top |
|
|
| golam1983 |
| Posted: Mon Jun 30, 2008 3:57 am Post subject: |
|
|
Apprentice
Joined: 27 Jun 2008
Posts: 35
|
I had faced such a scenario when i was making a no of database calls from my flow...for this we can merge all queries into single query and pass it to oracle....
Also as sebastianhirt mentioned try using the reference variable...it has a lot of impact for large file... |
|
| Back to top |
|
|
|
|
