| Author |
Message
|
| nick405 |
 Posted: Tue May 20, 2014 12:21 pm Post subject: Latency while processing 160 MB file through fileinput node |
 |
|
Newbie
Joined: 19 May 2014
Posts: 4
|
Hey friends,
I was able to process and split the 160 MB XML file using file input node and file output node. Just want to brief what ever I did in this scenario.
1. The large 160 MB XML file has nearly 100000 <FinancialIn> tags. Based on it I am splitting into smaller files of 1000 <FinancialIn> tags in each file. I was able to do it but it took me 3 days to see my 100 files in the output directory. I am looking out for a faster approach to resolve this. First of all parsing the message is taking take on the file input node.
2. Variables are set in the session that started the broker.
3. Have increased the heap size of the Execution group as well
Please can you bring down some inputs to actually process this file fastly |
|
| Back to top |
|
 |
| paustin_ours |
| Posted: Tue May 20, 2014 12:44 pm Post subject: |
|
|
Yatiri
Joined: 19 May 2004
Posts: 667
Location: columbus,oh
|
| If that is all that you are doing. It would be way faster to do it in Java or PERL. |
|
| Back to top |
|
|
| nick405 |
| Posted: Tue May 20, 2014 1:06 pm Post subject: |
|
|
Newbie
Joined: 19 May 2014
Posts: 4
|
Thanks for the Reply.
Well, This will not be it in the functionality of the Application. There is additional Processing to be done to these smaller chunks.
So, In WMB does it take that long to process a 160 MB file ?
I am on a zlinux, v8.0.0.3
I have increased the Memory of the EG to 1GB and even then it is the same.
Please let me know if I miss any other information here
Any Help is Appreciated.javascript:emoticon(' ')
Thank You |
|
| Back to top |
|
|
| paustin_ours |
| Posted: Tue May 20, 2014 1:13 pm Post subject: |
|
|
Yatiri
Joined: 19 May 2004
Posts: 667
Location: columbus,oh
|
| Quote: |
There is additional Processing to be done to these smaller chunks.
|
That is after the split right? You can do the split in Java/PERL and then process smaller files in broker.
That said. 160MB file is not a huge. I have worked with files in GB.
what setting you have in the fileinput node and fileoutput node? |
|
| Back to top |
|
|
| nick405 |
| Posted: Tue May 20, 2014 2:20 pm Post subject: |
|
|
Newbie
Joined: 19 May 2014
Posts: 4
|
I have been testing with "Whole File"
I tried with XMLNSC and BLOB + RCD (parsing the XML).
Either way it is the same. Its moving to mqsitransitin. But I dont see any thing in the output directory even after couple of hours!!
Thank you. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue May 20, 2014 8:09 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
you're killing yourself in processing it the wrong way...
Check out the big message sample.
Use references and not index.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| kimbert |
| Posted: Tue May 20, 2014 9:49 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| I have been testing with "Whole File" |
That is your problem. If you use Parsed Record Sequence then you should be able to process the file using a lot less memory. It might even get processed faster.
As fjb_saper says, check out the Large Messaging sample.
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
|
|
