| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Performance problem in parsing TDS with choice |
« View previous topic :: View next topic » |
| Author |
Message
|
| yaakovd_new |
 Posted: Sun Mar 09, 2008 8:54 pm Post subject: Performance problem in parsing TDS with choice |
 |
|
Apprentice
Joined: 03 Feb 2004
Posts: 25
Location: Israel
|
I have tagged delimited Message set, where each line-record is tagged and delimited with <LF>.
Line-record contains tagged delimited fields in format:
[<SOH>][tag][tag-data separator <STX>][data][delimiter <ETX>].
Im each line all element are optional and can be in any order.
I defined the message model as following:
| Code: |
cdrFile of type CDRFile
-----> cdr 1-unlimited of type CDR
-------> CGRgroup 1-unlimited (global group)
|-------> ref to field 1
|-------> ref to field 2
|-------> ref to field 3
|-------> .......
|-------> ref to field n
All field defined as global elements of type String with associated tag.
CDRFile type - sequence, All element delimited, delimiter <LF>, group terminator <LF>
CDR type - Tagged delimited, group terminator <EOT>, delimiter <ETX>
CGRgroup global group - choice, group indicator <SOH>, tag-data separator <STX>.
|
It is working fine, bat my understanding that "choice" has significant impact on performance.
I tried to use other types - unordered set etc., but it is not compatible with tagged delimited and all elements delimited formats.
Any idea how to improve performance?
Thanks
_________________
_________________
Best regards.
Yaakov Dolgov
E-Business Solutions Consultant
IBM Global Services, Israel
preiger.yaakov@il.ibm.com |
|
| Back to top |
|
 |
| kimbert |
| Posted: Mon Mar 10, 2008 12:36 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| my understanding that "choice" has significant impact on performance |
Not particularly. Obviously, the parser has to work harder when parsing Tagged fields than it does when parsing fixed-length records. Apart from that, there are no 'extra' costs associated with using choice.
Maybe you were thinking of Use Data Pattern - that can be quite expensive in CPU.
| Quote: |
| I tried to use other types - unordered set etc |
UnorderedSet would give almsot exactly the same performance as the equivalent choice.
| Quote: |
| but it is not compatible with tagged delimited |
There is nothing to stop you using Tagged Delimited with UnorderedSet. You probably know that anyway, but I'll correct the mistake in case someone else reads that and believes it
| Quote: |
| Any idea how to improve performance? |
If you're not on v6, upgrade straight away and get a big drop in TDS parser CPU usage. If you're already on v6, model it the straightforward way ( repeating Tagged Delimited choice ). You will not improve performance using an alternative model. |
|
| Back to top |
|
|
| yaakovd_new |
| Posted: Mon Mar 10, 2008 3:07 am Post subject: |
|
|
Apprentice
Joined: 03 Feb 2004
Posts: 25
Location: Israel
|
Thanks for reply!
I'm already with MB 6.1...
I'm really happy with my model... but when I parsed 600 KB file on my laptop (2GB RAM, but with a lot of soft working) it took 10 seconds... Hope it will be a liitle faster on 4-way 16 GB UNIX machine 
_________________
_________________
Best regards.
Yaakov Dolgov
E-Business Solutions Consultant
IBM Global Services, Israel
preiger.yaakov@il.ibm.com |
|
| Back to top |
|
|
| kimbert |
| Posted: Mon Mar 10, 2008 3:52 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
I guess your choice group has a large number of members.
Performance will depend on
- how many branches there are in the choice
- how many incorrect guesses the parser makes before it resolves each instance of the choice.
So if possible, order the members of the choice in order of frequency. |
|
| Back to top |
|
|
| yaakovd_new |
| Posted: Mon Mar 10, 2008 5:01 am Post subject: |
|
|
Apprentice
Joined: 03 Feb 2004
Posts: 25
Location: Israel
|
Yep... I know...
I have 75 fields in choice but totally unordered...
Actually there are several fixed formats that I want to parce by single parser. In worse case, I will separate them.
thanks
_________________
_________________
Best regards.
Yaakov Dolgov
E-Business Solutions Consultant
IBM Global Services, Israel
preiger.yaakov@il.ibm.com |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
