| Author |
Message
|
| ravilegolas |
 Posted: Thu Nov 22, 2012 5:34 am Post subject: Reg: EDIFACT messages |
 |
|
Novice
Joined: 04 Sep 2012
Posts: 15
|
Hi experts,
Presently I am working on EDIFACT messages of size ranging from 2 MB to 10 MB. My flow parses the XML message, and extracts information from the tree and inserts the record in the table based on business logic. insertion of records in the table takes a long time. I have done the coding in esql. My code has multiple while loops which cannot be avoided. I would like to know your views as to whether WMB is the right tool for this case. Is there any other programming language which will perform this operation faster. |
|
| Back to top |
|
 |
| Esa |
| Posted: Thu Nov 22, 2012 5:43 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| Review the Large Message Processing sample. It should give you ideas on how you can make the flow faster. At least the preprocessor flow that you obviously have for transforming EDIFACT into XML, and probably the database update flow, too. |
|
| Back to top |
|
|
| kimbert |
| Posted: Thu Nov 22, 2012 9:24 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5543
Location: Southampton
|
| Quote: |
| I would like to know your views as to whether WMB is the right tool for this case. |
It may well be the correct tool. But I would not be prepared to advise you based on the very limited information that you have given us. |
|
| Back to top |
|
|
| ravilegolas |
| Posted: Thu Nov 22, 2012 1:57 pm Post subject: |
|
|
Novice
Joined: 04 Sep 2012
Posts: 15
|
Hi Kimbert,
My current project is about parsing EDIFACT message, applying business logic and storing the value in DB for data warehousing. The problem we are facing is that, even though correct output is getting generated, it is taking lot of time to process. We have multiple while loops in our business logic. To reduce the memory consumption and also the number of while loops, I have used Delete field in my code. We receive around 500 messages is a day, with each message generating around 10000 records which needs to be inserted in the table.
My first question is
1. Whether using delete field in the code is a good practice. Does it improve the performance.
2. Any tips on how to improve the performance of the code at flow level and broker level.
3. A general question, ie. whether WMB can be used for such a scenario. |
|
| Back to top |
|
|
| visasimbu |
| Posted: Thu Nov 22, 2012 10:17 pm Post subject: |
|
|
Disciple
Joined: 06 Nov 2009
Posts: 171
|
Try the below options.
1) Use the JCN and wrap the insert statement with the prepared statement.
2) Copy the Inputroot to the outputroot and access the Input value from the outputroot for better performance. After the manipulation delete the Input value from the outputroot. |
|
| Back to top |
|
|
| kash3338 |
| Posted: Thu Nov 22, 2012 10:26 pm Post subject: |
|
|
Shaman
Joined: 08 Feb 2009
Posts: 709
Location: Chennai, India
|
| ravilegolas wrote: |
1. Whether using delete field in the code is a good practice. Does it improve the performance.
|
Your problem seems to with the time taken to insert/update the records in DB and not memory. using DELETE is always a better option when you no more need the contents in the tree.
| ravilegolas wrote: |
2. Any tips on how to improve the performance of the code at flow level and broker level.
|
There is documentation provided for performance tuning at code/flow level when using DB. Refer,
http://www.ibm.com/developerworks/websphere/library/techarticles/0809_kudikala/0809_kudikala.html
| ravilegolas wrote: |
3. A general question, ie. whether WMB can be used for such a scenario. |
As pointed earlier, WMB can be used for this. No matter what ever tool you use, if your looping logic is complicate and time consuming, the performance will be same across tools.
Better you can try to opyimize your coding logic to improve the performance. |
|
| Back to top |
|
|
| ravilegolas |
| Posted: Fri Nov 23, 2012 2:47 am Post subject: |
|
|
Novice
Joined: 04 Sep 2012
Posts: 15
|
Hi Kash,
In my flow, I have developed a separate flow for data insertion. The insertion is very fast. But more time is taken in populating records for insertion because of multiple while loops and large tree structure. There was a debate in my office whether esql as well as WMB is the right tool for such purpose. Some were suggesting to use Java for this kind of manipulations.But as you said, even I feel that what ever language you use for coding, if the logic involves while loops with big tree structure, then the time taken by both will be same. |
|
| Back to top |
|
|
| ravilegolas |
| Posted: Fri Nov 23, 2012 2:49 am Post subject: |
|
|
Novice
Joined: 04 Sep 2012
Posts: 15
|
| BTW kash3338 and visasimbu... thanks a lot for your thoughts.. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Nov 23, 2012 3:34 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20767
Location: LI,NY
|
make sure that in your while loops you are not using indices but references and move ref NEXTSIBLING. Also make sure you keep tree walking to a minimum (2 levels). Otherwise use references.
ESQL and Java will show you about the same performance, provided each is written according to the performance principles for the broker.
Java forces you to use a reference when walking the tree... but will allow you to write very inefficient XPath statements...
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| ravilegolas |
| Posted: Mon Nov 26, 2012 7:59 am Post subject: |
|
|
Novice
Joined: 04 Sep 2012
Posts: 15
|
Hi,
Finally I found out, why there was a makor performance issue when processing EDIFACT message. My code refers to a look up table having around 800 rows. I stored the lookup table values in Environment variable and used select statement to populate values. My code was selecting the values from look up table many times while processing. Because of this reason, the performance of the code was very slow. Any idea on how to select values from the look up table without the performance getting affected. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Mon Nov 26, 2012 8:00 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
|
| Back to top |
|
|
| kimbert |
| Posted: Tue Nov 27, 2012 2:26 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5543
Location: Southampton
|
| Quote: |
| My code was selecting the values from look up table many times while processing. Because of this reason, the performance of the code was very slow |
One solution would be to load the lookup table into a Java ArrayList and perform a binary search on the list. Other solutions may perform even better, but without knowing more about the key values ( please don't post them here!) it's hard to be specific. |
|
| Back to top |
|
|
| mqsiuser |
| Posted: Tue Nov 27, 2012 2:30 am Post subject: |
|
|
Yatiri
Joined: 15 Apr 2008
Posts: 637
Location: Germany
|
| ravilegolas wrote: |
| My code refers to a look up table having around 800 rows. I stored the lookup table values in Environment variable and used select statement to populate values. My code was selecting the values from look up table many times while processing. Because of this reason, the performance of the code was very slow. Any idea on how to select values from the look up table without the performance getting affected. |
SHARED ROW
make sure your ESQL loads if COUNT(rows) = 0
_________________
Just use REFERENCEs |
|
| Back to top |
|
|
| kimbert |
| Posted: Tue Nov 27, 2012 4:39 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5543
Location: Southampton
|
| Why would querying a large lookup table stored in a SHARED ROW be faster than querying a large lookup table stored in the Environment tree? Or am I missing the point? |
|
| Back to top |
|
|
| mqsiuser |
| Posted: Tue Nov 27, 2012 5:06 am Post subject: |
|
|
Yatiri
Joined: 15 Apr 2008
Posts: 637
Location: Germany
|
I thought like:
ENVIRONMENT: Lookup table needs to be loaded for each flow invocation (on each message).
SHARED ROW: Lookup table needs to be loaded only on the first (and after each restart of the flow / execgroup (and any logic you come up with that deals with reloading (when necessary) / invalidating.
Depends on you lookup-data... if its quite stable you can go with a SHARED ROW (and cache it). If it changes a lot you need to reload it more often.
_________________
Just use REFERENCEs |
|
| Back to top |
|
|
|
|
