| Author |
Message
|
| krypton |
 Posted: Tue Feb 28, 2017 9:50 am Post subject: Reading application Transactional DB from IIB/Message Broker |
 |
|
Disciple
Joined: 14 Mar 2010
Posts: 186
|
Hello folks,
we have been tasked to read the application Transactional DB and create batch files from the data and transfer it to other applications using IIB.
I am not in favor of reading the application DB, and I have asked the application team to handle it at their end to generate Batch files.
Am I right to believe that IIB should not play with other application databases. any thoughts?
_________________
Dreams are not something which you watch when you are asleep,it is something which doesn't let you sleep. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Tue Feb 28, 2017 10:04 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Batch files is the worst part of this task.
Reading an application DB is a somewhat normal part of transformation logic.
Batch files are just a bad idea when you have lots of better transports - MQ, MFT, HTTP, REST, etc... that can take single records and process them individually, rather than having to process huge chunks of data that might not be complete...
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Feb 28, 2017 10:19 am Post subject: Re: Reading application Transactional DB from IIB/Message Br |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| krypton wrote: |
| Am I right to believe that IIB should not play with other application databases. any thoughts? |
IIB is a perfectly reasonable component to use as part of an "application", in the same way an application could contain both Java under WAS on Linux and something like Angular on Windows to provide a front end. Indeed, it's a plausible design decision to use IIB to link such components.
I agree with my most worthy associate about batch files, and would also wonder if IIB is the best choice of ETL tool if all you want is a straightforward dump of db data into batch files.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| krypton |
| Posted: Wed Mar 01, 2017 7:03 am Post subject: |
|
|
Disciple
Joined: 14 Mar 2010
Posts: 186
|
| Quote: |
Reading an application DB is a somewhat normal part of transformation logic.
|
Thank you for reply, I forgot to mention that the Data we have to read from DB is of size around 500 MB. We are worrried about the overhead /slow processing time for such huge amount of data.
Is there a better way to handle this?
we have to take this huge dump of data and create XML files and need to make sure none of the XML file size reaches more than 100 MB.
_________________
Dreams are not something which you watch when you are asleep,it is something which doesn't let you sleep. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Mar 01, 2017 7:11 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Can you read the database row by row, instead of as a block?
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| krypton |
| Posted: Wed Mar 01, 2017 9:25 am Post subject: |
|
|
Disciple
Joined: 14 Mar 2010
Posts: 186
|
"Can you read the database row by row, instead of as a block?"
I can try that, now how do we control the XML file size using IIB? (which in our case is limited to 100 MB), we have to create XML files from the database but also have to ensure that each file size should not be more than 100 MB.
Does Writing XML data in the file would be more time consuming than writing normal text data?
_________________
Dreams are not something which you watch when you are asleep,it is something which doesn't let you sleep. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Mar 01, 2017 10:01 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| krypton wrote: |
"Can you read the database row by row, instead of as a block?"
I can try that, now how do we control the XML file size using IIB? (which in our case is limited to 100 MB), we have to create XML files from the database but also have to ensure that each file size should not be more than 100 MB.
Does Writing XML data in the file would be more time consuming than writing normal text data? |
If you can use normal text, then why do you need XML?
Processing individual records in IIB instead of big chunks of data is much more efficient, and uses less memory.
For the rest of your questions, think of it as a normal programming problem - how do you create a batch who's size is not larger than A.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
|
|
