Posted: Tue Sep 12, 2017 1:48 am Post subject: Reading Multiple Input Files via IIB
Newbie
Joined: 12 Sep 2017 Posts: 5
Hi
I have a requirement wherein I will receive two different CSV files with different layouts via File Input (FTP). There is one Main CSV File and another is the cross-reference CSV file. So, basically I need to read the Main CSV file and create an XML file but 4 of the tags in the XML will have to be created by referring to the cross-reference CSV file.
I had few questions:
1) Are these kinds of requirements commonplace for IIB? I thought that in most cases IIB receives a single file as input and transforms it.
2) Can I use the Collector Node to make sure we receive the two files within a specified time limit and use two DFDL's to parse these two files?
3) In order to read the cross-reference CSV file, Can I use Global Cache to store the data in memory and use it in the JavaCompute node with the Main file to create the XML file?
Posted: Tue Sep 12, 2017 6:14 am Post subject: Re: Reading Multiple Input Files via IIB
Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
vishysblue wrote:
1) Are these kinds of requirements commonplace for IIB? I thought that in most cases IIB receives a single file as input and transforms it.
It's an unusual (and somewhat daft) design for any product or application. Far more complexity than needed and loads of delicious places it can go wrong.
vishysblue wrote:
2) Can I use the Collector Node to make sure we receive the two files within a specified time limit and use two DFDL's to parse these two files?
A better question is what do you plan to do if they don't arrive in the time limit? What does the design say? Press on with the file you've got? Pretend nothing happened and ignore the file you received? What do you do when the other file eventually turns up?
See above re: loads of places this can go wrong
vishysblue wrote:
3) In order to read the cross-reference CSV file, Can I use Global Cache to store the data in memory and use it in the JavaCompute node with the Main file to create the XML file?
If you can read the cross reference file first. What will you do if the main file arrives first?
See above re: loads of places this can go wrong
vishysblue wrote:
Your advice is greatly appreciated.
Determine how often these cross reference values actually change. If they're sending you the same file over and over, store it in a database and refer to that each time the main file turns up. Build a trigger mechanism so that when the cross reference changes, they don't send a main file until you acknowledge processing of the new cross reference data. _________________ Honesty is the best policy.
Insanity is the best defence.
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum