| Author |
Message
|
| ajayrana |
 Posted: Mon May 26, 2014 9:31 am Post subject: File processing taking up all memory |
 |
|
Apprentice
Joined: 08 May 2014
Posts: 35
|
Hi All,
I am trying to read a fixed length file with some 140 fields. The file size is 189 MB and it has 160000 records to process with single record length 1483 characters.
The record detection property is set to 'Parsed Record Sequence'.
My message flow representation :
FileReader - > Mapping Transformation (which inserts into MSSQL database)
The transaction mode on the file reader is set to No and Transaction property at mapping is set to automatic.
Now every time i try to process the file it keeps on executing for more than an hour (with taking up all the memory in the system) and then when there is no memory left the file processing fails.
Is there a way i can define the database commit size in my bar file ??
Thanks & Regards
Ajay Rana
Last edited by ajayrana on Tue May 27, 2014 10:19 pm; edited 1 time in total |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Mon May 26, 2014 9:41 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Set the mapping node transaction mode to NO 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| kimbert |
| Posted: Mon May 26, 2014 1:49 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
it has 160000 records to process with single record length 1483 characters.
The record detection property is set to 'Parsed Record Sequence'. |
If every record is 1483 characters then you may as well use 'Fixed Length' rather than 'Parsed record sequence'. Or maybe you have a good reason for not doing that?
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon May 26, 2014 4:31 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
ajayrana, please stop putting "URGENT" in all your subject lines. Its presumptuous to think you'll get a faster or better response from people doing this on their own time on a free website by placing that word in all caps in the subject line. In fact, it very likely has the exact opposite effect you intend.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| ajayrana |
| Posted: Tue May 27, 2014 10:26 pm Post subject: |
|
|
Apprentice
Joined: 08 May 2014
Posts: 35
|
Hi Kimbert,
The file includes header and trailer which are not of the same length(forgot to mention that).
We have resolved the issue by using Delimited option in records and elements.
This required us to move all the three sections(header/body/trailer) of the DFDL under a choice element.
This worked well for us. Thanks for the direction.
Hi fjb_saper,
I tried doing this before trying the above approach but the drop down to select the transaction mode only shows automatic in Mapping transformation properties.
Hi PeterPotKay,
I did not realise this would have such impact on the users of the community.
My apologies !! I have removed URGENT from all my posts.
Regards
Ajay Rana |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed May 28, 2014 12:31 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| ajayrana wrote: |
Hi fjb_saper,
I tried doing this before trying the above approach but the drop down to select the transaction mode only shows automatic in Mapping transformation properties.
Regards
Ajay Rana |
So may be you should do the DB insert in a node that gives more control over the transactionality...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
