| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| FileInput NodeV7 -Delimited mode-Mem footprint/ performance |
« View previous topic :: View next topic » |
| Author |
Message
|
| jbanoop |
 Posted: Tue May 18, 2010 12:47 pm Post subject: FileInput NodeV7 -Delimited mode-Mem footprint/ performance |
 |
|
Chevalier
Joined: 17 Sep 2005
Posts: 401
Location: SC
|
All,
Our Environment is : MB V 7.0, AIX.
We have a requirement in which we have approx 100,000 records (6-8 MB file) in a file in which each record needs to be parsed (MRM) , transformed and written out to an output file.
(FIN -> RCD -> Compute ->FON)
We are able to successfully do this using the FileInput node with its Record detection set to 'delimited' and with the delimiter as the 'DOS or UNIX Line End'.
My question was about the memory footprint and any other performance considrations for this strategy. From my understanding, since the File Nodes read and write (appends) record by record, this approach should not be memory intensive but would like to confirm it here on the forum.
Also, since this involves multiple File I/O operations, could this slow down the overall process or consume more CPU thn desirable ?
Looking forward to your insights on this ...
Regards, |
|
| Back to top |
|
 |
| kimbert |
| Posted: Tue May 18, 2010 1:47 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
You have set up your flow in the way that the product development team intended, so that's a good start 
The FileInput node does not read the entire file into memory - it streams it in smaller buffers. Since you are using a streaming input node, the MRM parser will respect that, and will not request more data than it needs. You are using Parsed Record Sequence, so that sounds like a good, memory-efficient setup.
I'll let others comment on the possible file I/O slowdown. For my money, the file I/O problems would need to be quite bad before I would sacrifice the efficient use of memory. |
|
| Back to top |
|
|
| jbanoop |
| Posted: Wed May 19, 2010 4:50 am Post subject: |
|
|
Chevalier
Joined: 17 Sep 2005
Posts: 401
Location: SC
|
Kimbert,
Thanks for your insight, needless to say, very helpful as always . |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
