MQSeries.net :: View topic - 10 - 20 Mb CSV file to database table

MQSeries.net

Tech Exchange

Education

Certifications

Library

Info Center

SupportPacs

FAQÂ Â

Usergroups

RSS Feed - WebSphere MQ Support

RSS Feed - Message Broker Support

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » 10 - 20 Mb CSV file to database table

10 - 20 Mb CSV file to database table

« View previous topic :: View next topic »

Author

Message

venky

Posted: Thu Nov 10, 2016 4:31 pm Post subject: 10 - 20 Mb CSV file to database table

Master

Joined: 08 Jul 2003
Posts: 205

Hi - we have a requirement to get (a 10 mb file from a FTP location) and perform transformation on some of the fields and write to a target DB table. This happens once a day. While this looks like a classic ETL scenario, our ETL team is pushing this to the IIB platform, actually we have around 25 of these requirements.

Questions:
1. Is this OK to implement this scenario on the IIB platform ?
2. Any performance implications given there will be around 25 of these (10-20 MB files) ?
3. Also when we write this to the DB table, is there a way to write all records in one shot ? vs. writing one record at a time to the target table.?

Thanks,

timber

Posted: Fri Nov 11, 2016 1:57 pm Post subject:

Grand Master

Joined: 25 Aug 2015
Posts: 1280

1. You can do almost anything with IIB if you try hard enough.
2. Possibly. Especially if you do not take care to design your message flow carefully. There are lots of posts about large file handling in this forum. The IIB Knowledge Center has a topic about it too.
3. Probably. I recommend some research on that. The Knowledge Center is a good place to start.

mqjeff

Posted: Wed Nov 16, 2016 8:33 am Post subject:

Grand Master

Joined: 25 Jun 2008
Posts: 17447

"once a day" is problematic to solve in IIB directly
"read a file" is problematic in any programming language in any situation - how do you know the file is ready?
Bulk insert of records depends entirely on the DB and how you build your message flow.
ESQL Insert doesn't let you do bulk insert.
Dealing with all records from each file will require a large amount of memory.
See timber's notes about large file processing
go back and revisit the whole process of accessing the data in these files.
aim to turn the data in the files into messages

_________________
chmod -R ugo-wx /

Display posts from previous:

Page 1 of 1

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » 10 - 20 Mb CSV file to database table

Jump to:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

Protected by Anti-Spam ACP