ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » Reg: EDIFACT messages

Post new topic  Reply to topic Goto page 1, 2  Next
 Reg: EDIFACT messages « View previous topic :: View next topic » 
Author Message
ravilegolas
PostPosted: Thu Nov 22, 2012 5:34 am    Post subject: Reg: EDIFACT messages Reply with quote

Novice

Joined: 04 Sep 2012
Posts: 15

Hi experts,
Presently I am working on EDIFACT messages of size ranging from 2 MB to 10 MB. My flow parses the XML message, and extracts information from the tree and inserts the record in the table based on business logic. insertion of records in the table takes a long time. I have done the coding in esql. My code has multiple while loops which cannot be avoided. I would like to know your views as to whether WMB is the right tool for this case. Is there any other programming language which will perform this operation faster.
Back to top
View user's profile Send private message
Esa
PostPosted: Thu Nov 22, 2012 5:43 am    Post subject: Reply with quote

Grand Master

Joined: 22 May 2008
Posts: 1387
Location: Finland

Review the Large Message Processing sample. It should give you ideas on how you can make the flow faster. At least the preprocessor flow that you obviously have for transforming EDIFACT into XML, and probably the database update flow, too.
Back to top
View user's profile Send private message
kimbert
PostPosted: Thu Nov 22, 2012 9:24 am    Post subject: Reply with quote

Jedi Council

Joined: 29 Jul 2003
Posts: 5542
Location: Southampton

Quote:
I would like to know your views as to whether WMB is the right tool for this case.
It may well be the correct tool. But I would not be prepared to advise you based on the very limited information that you have given us.
Back to top
View user's profile Send private message
ravilegolas
PostPosted: Thu Nov 22, 2012 1:57 pm    Post subject: Reply with quote

Novice

Joined: 04 Sep 2012
Posts: 15

Hi Kimbert,
My current project is about parsing EDIFACT message, applying business logic and storing the value in DB for data warehousing. The problem we are facing is that, even though correct output is getting generated, it is taking lot of time to process. We have multiple while loops in our business logic. To reduce the memory consumption and also the number of while loops, I have used Delete field in my code. We receive around 500 messages is a day, with each message generating around 10000 records which needs to be inserted in the table.
My first question is
1. Whether using delete field in the code is a good practice. Does it improve the performance.
2. Any tips on how to improve the performance of the code at flow level and broker level.
3. A general question, ie. whether WMB can be used for such a scenario.
Back to top
View user's profile Send private message
visasimbu
PostPosted: Thu Nov 22, 2012 10:17 pm    Post subject: Reply with quote

Disciple

Joined: 06 Nov 2009
Posts: 171

Try the below options.

1) Use the JCN and wrap the insert statement with the prepared statement.
2) Copy the Inputroot to the outputroot and access the Input value from the outputroot for better performance. After the manipulation delete the Input value from the outputroot.
Back to top
View user's profile Send private message Send e-mail
kash3338
PostPosted: Thu Nov 22, 2012 10:26 pm    Post subject: Reply with quote

Shaman

Joined: 08 Feb 2009
Posts: 709
Location: Chennai, India

ravilegolas wrote:

1. Whether using delete field in the code is a good practice. Does it improve the performance.


Your problem seems to with the time taken to insert/update the records in DB and not memory. using DELETE is always a better option when you no more need the contents in the tree.

ravilegolas wrote:

2. Any tips on how to improve the performance of the code at flow level and broker level.


There is documentation provided for performance tuning at code/flow level when using DB. Refer,
http://www.ibm.com/developerworks/websphere/library/techarticles/0809_kudikala/0809_kudikala.html

ravilegolas wrote:

3. A general question, ie. whether WMB can be used for such a scenario.


As pointed earlier, WMB can be used for this. No matter what ever tool you use, if your looping logic is complicate and time consuming, the performance will be same across tools.

Better you can try to opyimize your coding logic to improve the performance.
Back to top
View user's profile Send private message Send e-mail
ravilegolas
PostPosted: Fri Nov 23, 2012 2:47 am    Post subject: Reply with quote

Novice

Joined: 04 Sep 2012
Posts: 15

Hi Kash,
In my flow, I have developed a separate flow for data insertion. The insertion is very fast. But more time is taken in populating records for insertion because of multiple while loops and large tree structure. There was a debate in my office whether esql as well as WMB is the right tool for such purpose. Some were suggesting to use Java for this kind of manipulations.But as you said, even I feel that what ever language you use for coding, if the logic involves while loops with big tree structure, then the time taken by both will be same.
Back to top
View user's profile Send private message
ravilegolas
PostPosted: Fri Nov 23, 2012 2:49 am    Post subject: Reply with quote

Novice

Joined: 04 Sep 2012
Posts: 15

BTW kash3338 and visasimbu... thanks a lot for your thoughts..
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Fri Nov 23, 2012 3:34 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY

make sure that in your while loops you are not using indices but references and move ref NEXTSIBLING. Also make sure you keep tree walking to a minimum (2 levels). Otherwise use references.

ESQL and Java will show you about the same performance, provided each is written according to the performance principles for the broker.

Java forces you to use a reference when walking the tree... but will allow you to write very inefficient XPath statements...

Have fun
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
ravilegolas
PostPosted: Mon Nov 26, 2012 7:59 am    Post subject: Reply with quote

Novice

Joined: 04 Sep 2012
Posts: 15

Hi,
Finally I found out, why there was a makor performance issue when processing EDIFACT message. My code refers to a look up table having around 800 rows. I stored the lookup table values in Environment variable and used select statement to populate values. My code was selecting the values from look up table many times while processing. Because of this reason, the performance of the code was very slow. Any idea on how to select values from the look up table without the performance getting affected.
Back to top
View user's profile Send private message
lancelotlinc
PostPosted: Mon Nov 26, 2012 8:00 am    Post subject: Reply with quote

Jedi Knight

Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA

Yes, store your table of 800 rows in a cache, like solidDb or eXtremeScale.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER
Back to top
View user's profile Send private message Send e-mail
kimbert
PostPosted: Tue Nov 27, 2012 2:26 am    Post subject: Reply with quote

Jedi Council

Joined: 29 Jul 2003
Posts: 5542
Location: Southampton

Quote:
My code was selecting the values from look up table many times while processing. Because of this reason, the performance of the code was very slow
One solution would be to load the lookup table into a Java ArrayList and perform a binary search on the list. Other solutions may perform even better, but without knowing more about the key values ( please don't post them here!) it's hard to be specific.
Back to top
View user's profile Send private message
mqsiuser
PostPosted: Tue Nov 27, 2012 2:30 am    Post subject: Reply with quote

Yatiri

Joined: 15 Apr 2008
Posts: 637
Location: Germany

ravilegolas wrote:
My code refers to a look up table having around 800 rows. I stored the lookup table values in Environment variable and used select statement to populate values. My code was selecting the values from look up table many times while processing. Because of this reason, the performance of the code was very slow. Any idea on how to select values from the look up table without the performance getting affected.


SHARED ROW

make sure your ESQL loads if COUNT(rows) = 0
_________________
Just use REFERENCEs
Back to top
View user's profile Send private message
kimbert
PostPosted: Tue Nov 27, 2012 4:39 am    Post subject: Reply with quote

Jedi Council

Joined: 29 Jul 2003
Posts: 5542
Location: Southampton

Why would querying a large lookup table stored in a SHARED ROW be faster than querying a large lookup table stored in the Environment tree? Or am I missing the point?
Back to top
View user's profile Send private message
mqsiuser
PostPosted: Tue Nov 27, 2012 5:06 am    Post subject: Reply with quote

Yatiri

Joined: 15 Apr 2008
Posts: 637
Location: Germany

I thought like:

ENVIRONMENT: Lookup table needs to be loaded for each flow invocation (on each message).

SHARED ROW: Lookup table needs to be loaded only on the first (and after each restart of the flow / execgroup (and any logic you come up with that deals with reloading (when necessary) / invalidating.

Depends on you lookup-data... if its quite stable you can go with a SHARED ROW (and cache it). If it changes a lot you need to reload it more often.
_________________
Just use REFERENCEs
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » Reg: EDIFACT messages
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.